Log File Analyser
See which search and AI crawlers hit your site, the URLs they crawl most, and the status codes they get - from your access log.
How to use it
- Export a slice of your server access log (Apache or Nginx, common or combined format).
- Paste or upload it here. Large files are read locally and never uploaded.
- Read the crawler breakdown: which bots hit your site, the most-crawled URLs, and the status codes they got.
- Look for waste - bots hammering low-value URLs, or crawlers hitting 404s and redirects.
What it does, and what it does not
It does: parse standard access-log lines, identify hits from search and AI crawlers by user-agent (Googlebot, Bingbot, GPTBot, ClaudeBot and more), and summarise hits per bot, the top crawled URLs, and the status codes served.
It does not: verify that a bot is genuine (user-agents can be faked), or replace a full log analysis platform for millions of lines. It is a fast, private read on a representative sample - and your log never leaves your machine.
Reference
Your server access log is the only record of what search engines actually did on your site, as opposed to what you assume they did. Every time Googlebot or an AI crawler requests a URL, your server writes a line. Reading those lines - log file analysis - is the most direct way to understand crawl behaviour: which pages get crawled and how often, which get ignored, and where crawlers waste effort on URLs that do not matter.
This tool reads the common and combined log formats that Apache and Nginx produce by default. Each line carries the requesting IP, a timestamp, the request method and path, the status code returned, and the user-agent. The tool matches the user-agent against the known search and AI crawlers, then aggregates: total hits per bot, the URLs those bots requested most, and the spread of status codes. That last one is where problems hide - a pile of 404s in crawler traffic means bots are spending your crawl budget on dead URLs, and a wave of 301s means they are hitting redirects that should have been updated at the source.
What to look for. If Googlebot is spending most of its requests on faceted-search URLs, filter parameters or old paginated pages, that is crawl budget you would rather it spent on your money pages - a signal to tighten robots.txt or internal linking. If important pages barely appear, they may be too deep or poorly linked. On the AI side, the log is where you see whether GPTBot, ClaudeBot and the rest are actually crawling you, which informs whether to allow or block them. One caveat worth keeping: user-agents can be spoofed, so a line claiming to be Googlebot is not proof it was - genuine verification means checking the IP with a reverse DNS lookup, which is beyond a browser tool. For spotting patterns in a sample, though, this gives you the picture fast, and because it runs locally, you can analyse a client's log without it ever leaving your computer.
Questions
What log format does this accept?
The common and combined access-log formats that Apache and Nginx use by default - the lines with an IP, a bracketed date, the request in quotes, a status code and a user-agent.
Is my log file uploaded?
No. Whether you paste or upload, the log is read and analysed entirely in your browser. It never leaves your machine, which is why you can safely analyse a client's log.
Can it confirm a bot is really Googlebot?
No. User-agents can be faked. Confirming a genuine Googlebot means a reverse DNS lookup on the IP, which a browser cannot do. For spotting crawl patterns in a sample, the user-agent is enough.
What should I look for?
Crawlers wasting budget on low-value URLs, filter or parameter pages, or hitting 404s and redirects. Also whether AI crawlers like GPTBot are accessing your site.
How much can I paste?
A representative sample works best. Very large files are truncated for speed. For millions of lines, use a dedicated log analysis platform.