Server logs are the only record of what actually happened when a crawler hit your site. Every other signal, a crawl simulation from an SEO tool, a citation you spotted manually in ChatGPT, an assumption about what “AI crawlers probably do,” is a model of behavior. The log line is the behavior. Google’s own crawl budget documentation frames this directly: crawling capacity and demand are real, finite resources shaped by your server’s serving capacity and the perceived value of your content, and the only way to see how that budget is actually being spent is to look at what got requested.1
That discipline, log file analysis, is decades old in technical SEO and consistently underused, because pulling and parsing raw access logs is more work than running a crawler simulation. It has become more valuable, not less, now that GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and half a dozen other AI crawlers are hitting the same servers Googlebot always has. Most B2B teams have never looked at what any of them are actually doing on their site. This post covers the framework we run to find out, and the one verification step almost everyone skips.
Why a user-agent string alone doesn’t tell you who really crawled you
Every major AI crawler declares a specific user-agent string: GPTBot identifies itself as such in its request headers, and so do ClaudeBot and PerplexityBot. The problem is that a user-agent string is just text a request sends about itself; nothing stops a scraper, a competitor, or a data broker from setting that same string and pretending to be GPTBot. That is not a hypothetical risk. Cloudflare reported in August 2025 that Perplexity’s crawler, after being explicitly blocked by robots.txt on customer sites, switched to undeclared user agents impersonating a Chrome browser and rotated IP addresses across different network providers to keep crawling anyway, at a volume Cloudflare described as millions of additional daily requests. Cloudflare responded by revoking Perplexity’s verified-bot status.2 Whatever you conclude about that specific dispute, it settles the underlying question: you cannot take a user-agent string at face value when you’re deciding what a bot is actually allowed to do with your content.
The fix is IP verification, and every major AI crawler operator that wants to be trusted publishes what you need to do it. OpenAI documents its crawlers and their published IP ranges directly.3 Anthropic’s help center documents ClaudeBot, Claude-User, and Claude-SearchBot as three distinct crawlers with different purposes, all of which are expected to honor robots.txt.4 Perplexity publishes its own verification endpoint for the same purpose.5 A log audit that skips this step and just string-matches on user agent is counting some unknown mix of real crawler traffic and anyone who copied that string into a scraper.
The AI Crawler Log Audit: a 4-step framework
This is the sequence we run when a client wants to know what AI crawlers are actually doing on their site, rather than guessing from a robots.txt file and hoping.
Step 1: isolate and verify
Export raw access logs directly from your server or CDN, not a sampled or aggregated analytics report, since bot traffic is exactly the kind of low-frequency, high-signal activity that sampling throws away. Filter to the known AI bot user-agent strings, then run the verification pass before doing any other analysis: tag each line as verified or unverified against the operator’s published IP list, and do all subsequent counting on the verified set only. Skipping this step is the single most common mistake we see in a first-pass log audit, and it is also the step the Perplexity stealth-crawling incident above shows is not optional.
Step 2: segment by bot and section
Once you trust the data, break it down two ways: by which bot (GPTBot vs. ClaudeBot vs. PerplexityBot vs. Google-Extended, each counted separately, not lumped into one “AI crawler” bucket) and by which part of your site each one is actually spending time on. A B2B SaaS site often finds that AI crawlers hit documentation and comparison pages heavily and barely touch the pages marketing spent the most budget on. That gap is only visible once you segment.
Step 3: correlate with outcomes
Crawl activity by itself tells you a page was read, not that it did anything for you. Cross-reference which pages get the most verified AI crawler visits against which pages you’re actually seeing cited in ChatGPT, Perplexity, or Google AI Overviews. A page that gets crawled constantly but never cited is a different problem, usually a content quality or extractability issue, than a page that’s never crawled at all, which is usually an access or discoverability issue.
Step 4: set policy and cadence
Use what step 3 showed to write explicit, per-bot robots.txt rules rather than one blanket allow or disallow rule for every AI crawler, since a company willing to let ClaudeBot train on its public docs might still want to block a bot that’s clearly not honoring its own stated behavior. We cover the specific per-bot rule syntax and common mistakes in our robots.txt guide for B2B SaaS. Then put the audit itself on a recurring calendar. Bot behavior is not static: operators add new crawlers, change IP ranges, and, as the Perplexity case shows, occasionally change how honestly they identify themselves.
Major AI crawlers compared
Getting started without a dedicated log tool
Dedicated platforms like Screaming Frog Log File Analyzer, Botify, and JetOctopus automate verification and segmentation and are worth it once you’re doing this on a recurring basis across a large site. For a first pass, or a one-off audit, a scripted approach against raw Apache or Nginx logs, or your CDN’s raw log export, is enough to answer the questions that matter: which bots are hitting you, whether they’re really who they claim to be, and what they’re actually reading. If you want a broader read on your technical SEO foundation before or alongside a crawler audit, including crawlability issues that affect both traditional search and AI systems, our technical SEO team runs this as part of a full audit. For teams that want the citation side of this mapped at the same time as the crawl side, a GEO audit covers both in one pass.
Frequently asked questions
What is log file analysis in SEO?
Log file analysis is the practice of examining your web server’s raw access logs to see, request by request, exactly which pages a crawler visited, when, and what response code it got. Unlike a crawl simulation from a third-party tool, server logs are ground truth: they show what Googlebot, Bingbot, GPTBot, ClaudeBot, and other bots actually did on your site, not a model of what they might do.
How do you use log file analysis for SEO and AI visibility?
Pull raw server logs (not a sampled analytics export), filter requests to known bot user agents, verify each hit against the bot’s officially published IP range, then segment the verified traffic by bot and by site section to see what each system is actually reading. That data tells you where crawl budget is wasted, which pages AI crawlers ignore, and where your robots.txt or CDN rules need to change.
What is GPTBot and how do I verify it?
GPTBot is OpenAI’s web crawler, used to collect training data for its models. Its user-agent string can be spoofed by any scraper, so OpenAI publishes an official list of the IP ranges it crawls from at a dedicated JSON endpoint, and a log line should only be counted as genuine GPTBot traffic if its source IP appears in that published range.3
Does PerplexityBot respect robots.txt?
Perplexity’s declared crawler says it does, but Cloudflare reported in August 2025 that Perplexity was also using undeclared, spoofed user agents and rotating IP addresses to continue crawling sites that had explicitly blocked it, and delisted Perplexity’s crawler as verified as a result.2 That history is exactly why IP verification, not user-agent string matching alone, matters when you audit AI crawler behavior in your logs.
What log file analysis tools are used for SEO?
Dedicated tools like Screaming Frog Log File Analyzer, Botify, and JetOctopus are built for this and handle bot verification and segmentation automatically. For a lighter first pass, or for a one-off AI-crawler audit, a scripted approach against raw Apache or Nginx access logs, or your CDN’s raw log export, works fine and keeps you in control of exactly which bots and fields you’re checking.
How often should I audit AI crawler traffic in my logs?
Monthly is a reasonable default for most B2B sites, with an extra pull after any major site change: a migration, a new CDN or WAF rule, a robots.txt update, or a redesign that changes URL structure. AI crawler behavior itself also changes as vendors update their bots, so a log audit that was accurate six months ago may not reflect current crawling patterns.
Sources
1. Google Search Central, “Crawl Budget Management For Large Sites,” Google for Developers. ↩
2. Search Engine Journal, “Cloudflare Delists And Blocks Perplexity From Crawling Websites,” August 2025. ↩
3. OpenAI, “Overview of OpenAI Crawlers,” OpenAI Developer Platform. ↩
4. Anthropic, “Does Anthropic Crawl Data From the Web, and How Can Site Owners Block the Crawler?,” Anthropic Help Center. ↩
5. Perplexity, “Perplexity Crawlers,” Perplexity Documentation. ↩
Share this article
Ryan Brooks leads technical SEO at MV3 Marketing, specializing in schema architecture, entity graphs, crawlability, and the structural signals that determine whether AI answer engines cite a page.
Ready to audit your organic growth opportunity?
$2,500 flat. 5 business days. Six deliverables tied to pipeline , not rankings. No retainer required.
Get the Organic Growth Audit →