Most B2B SaaS and e-commerce teams find out they have a crawl budget problem the same way: organic traffic is flat, new pages take weeks to get indexed, and Google Search Console’s Crawl Stats report shows a total request count that looks fine in aggregate but tells you nothing about what actually got crawled. The Crawl Stats report is a summary. Your server’s raw access logs are the record. Every request Googlebot makes lands there, with the exact URL, timestamp, and response code, and that is the only place you can actually see whether your crawl budget is going toward the pages that drive revenue or toward filtered category pages, session parameters, and redirect chains nobody meant to keep.
Google’s own documentation defines crawl budget as the combination of two separate constraints: a crawl capacity limit, which is how many connections your server can hold without degrading for real users, and crawl demand, which is how much Google actually wants to crawl your site based on perceived value, popularity, and how often content changes.1 Most teams focus entirely on the first number and ignore the second, which is backwards for the sites where crawl budget actually matters: large B2B SaaS platforms with faceted documentation and pricing permutations, and e-commerce catalogs with filter, sort, and search-driven URLs that can multiply a few thousand real pages into millions of crawlable ones.
Page count is the wrong thing to worry about
The most common misconception about crawl budget is that it is primarily a function of site size, and Google’s own team has been walking that back in public. In a May 2025 episode of Google’s Search Off the Record podcast, Search Relations lead Gary Illyes said directly that backend cost, not URL count, is usually the bigger constraint: “If you are making expensive database calls, that’s going to cost the server a lot,” and that it is often “not crawling that is eating up the resources, it’s indexing and potentially serving or what you are doing with the data when you are processing that data.”2 A site with 500,000 dynamically generated pages behind slow database queries can create more crawl problems than a site with 2 million pages served from a static cache, a distinction that matters directly for SaaS platforms with database-backed pricing calculators, docs search, or per-account URL structures.
This is also happening against a backdrop of more total crawler traffic than most teams account for. Cloudflare’s 2025 Radar year-in-review found that Googlebot alone generated roughly 4.5% of all HTML requests across its network, while every other AI crawler combined, GPTBot, ClaudeBot, PerplexityBot, and the rest, added another 4.2%.3 Your server is fielding meaningfully more automated traffic than it was two years ago, which makes it more expensive, not less, to let crawl budget go to waste on URLs that were never worth crawling in the first place.
The Crawl Efficiency Ratio (CER) Method: a 5-step framework
This is the sequence we run when a B2B SaaS or e-commerce client wants to know whether their crawl budget is actually going where it should, instead of guessing from an aggregate Search Console number.
Step 1: pull and verify
Export raw access logs from your server or CDN for a minimum two-to-four week window; a shorter sample will not capture your full crawl pattern, especially for sections Google visits weekly rather than daily. Filter to the Googlebot user-agent, then verify each hit against Google’s published IP ranges before counting it, since user-agent strings are trivial to spoof and an unverified count will overstate real Googlebot activity with scraper traffic pretending to be it.
Step 2: classify every hit
Once the verified set is trustworthy, tag every single request into one of four buckets: indexable canonical pages returning a 200, redirects (3xx), duplicate or non-canonical URLs (parameters, facets, session IDs, paginated or sorted variants of a canonical page), and errors or blocked-but-crawled URLs (4xx, 5xx, soft 404s, or pages disallowed in robots.txt that Google is still hitting from old links). Nothing should be left in an “other” bucket; if a URL pattern does not fit, that usually means you have found a URL structure problem you did not know existed.
Step 3: calculate the CER
Divide indexable canonical 200 hits by total verified hits to get your Crawl Efficiency Ratio. This single number is more actionable than a raw crawl count because it is comparable over time and across sections regardless of how much your total site size changes. A documentation subdomain with a 70% CER is healthy; a product catalog with a 30% CER, where 70% of Google’s attention on that section is going to filter and sort permutations instead of actual product pages, is a real, fixable problem.
Step 4: weight by cost, not just count
This is the step most crawl budget guides skip, and it is the one Google’s own team has recently pushed practitioners toward. Two waste categories with identical hit counts are not equally worth fixing if one is a static, cached parameter URL and the other triggers a live database query, a personalization lookup, or a search index call on every request.2 Pull rough backend cost from your application logs or ask engineering which URL patterns hit the database directly, and prioritize fixes there first, even if the raw crawl-hit count is lower than a cheaper, higher-volume waste category.
Step 5: fix, re-baseline, set cadence
Implement the highest-weighted fix, whether that is a canonical tag, a robots.txt disallow, a parameter-handling rule, or a redirect cleanup, then re-pull logs two to four weeks later and recalculate the CER to confirm it actually moved. Set a recurring cadence from there: Google’s own crawl budget guidance is explicitly aimed at sites with a million-plus pages updating weekly or 10,000-plus pages updating daily, and those sites should treat this as a monthly discipline.1 Smaller B2B SaaS sites can run it quarterly, with an extra pull after any migration or major URL structure change; we cover migration-specific crawl risk in our XML sitemap guide for large B2B SaaS sites.
Common crawl budget waste categories compared
What a CER report actually looks like
A before-and-after CER breakdown is the clearest way to show a stakeholder what a crawl budget fix actually accomplished. The mockup below illustrates the format we build for clients; the numbers are invented for illustration and not drawn from any real account.
Why this matters more on e-commerce catalogs
Faceted navigation is the single biggest crawl budget risk on e-commerce sites because filter and sort combinations multiply fast: a category with 5 filters and 4 values each can generate thousands of crawlable URL combinations from a page count in the dozens. Botify’s own analysis of a large classified marketplace found non-canonical URLs made up 97% of one million crawled pages, while the indexable URL count for that same section was closer to 25,000, and even then, Google was only reaching a little over half of those indexable pages in a given month.4 Botify’s broader cross-industry data puts the average unoptimized site at only about 40% of its strategic URLs crawled monthly, meaning 60% of the pages a business actually wants ranked are not getting regular attention from Google at all.4 That gap is exactly what the CER Method is built to surface and close.
Getting started without enterprise tooling
Dedicated platforms like Screaming Frog Log File Analyser, Botify, JetOctopus, and OnCrawl automate verification and classification and are worth the cost once you are running this on a recurring basis across a large site. For a first pass, a scripted approach against raw Apache, Nginx, or CDN log exports is enough to calculate an initial CER and find your biggest waste category. If you want AI crawler behavior audited alongside traditional Googlebot crawl budget, since the two increasingly hit the same URLs for different reasons, our AI Crawler Log File Analysis framework covers the verification and segmentation steps for GPTBot, ClaudeBot, and PerplexityBot specifically. For a broader technical foundation review, our technical SEO team runs crawl budget and log file audits as part of a full site assessment, and if you want a fast read on where your site currently stands, a GEO audit covers crawlability alongside AI citation visibility in one pass.
Frequently asked questions
What is crawl budget optimization?
Crawl budget optimization is the practice of managing which URLs on your site Googlebot spends its limited crawl capacity on, so that indexable, canonical, revenue-relevant pages get crawled and refreshed instead of duplicate, parameterized, redirected, or error-state URLs. Google defines crawl budget as the combination of crawl capacity limit, how much crawling your server can handle, and crawl demand, how much Google actually wants to crawl your site.
What is log file analysis in SEO?
Log file analysis is the practice of examining your web server’s raw access logs, not a sampled export or a third-party crawl simulation, to see exactly which URLs Googlebot and other crawlers requested, how often, and what status code each request returned. It is the only method that shows actual crawler behavior rather than a model of it, which is why it is the standard method for diagnosing crawl budget problems on large sites.
How do you optimize crawl budget for an e-commerce site?
Start by pulling verified Googlebot hits from raw server logs and classifying them by URL type, since e-commerce sites typically lose the most crawl budget to faceted navigation, filter and sort parameters, and internal search result pages that generate near-infinite duplicate URL combinations. Consolidate those patterns with canonical tags or parameter handling, block the ones with no search value in robots.txt, fix redirect chains from old product and category URLs, and re-check your log data after the change to confirm the ratio of crawls going to indexable pages actually improved.
What is the difference between crawl rate limit and crawl demand?
Crawl rate limit, which Google also calls crawl capacity limit, is the maximum number of simultaneous connections and the delay between fetches that Googlebot will use without degrading your server’s performance for real visitors. Crawl demand is separate: it is how much Google actually wants to crawl your site, based on the perceived value, popularity, and staleness of your content. Your actual crawl budget is effectively the lower of the two.
What tools are used for SEO log file analysis?
Dedicated platforms like Screaming Frog Log File Analyser, Botify, JetOctopus, and OnCrawl are built specifically for this and handle bot verification and URL classification automatically at scale. For a smaller site or a first-pass audit, a scripted approach against raw Apache, Nginx, or CDN access logs works fine and keeps you in control of exactly which fields and bots you are checking.
How often should a B2B SaaS site audit crawl budget and log files?
Google’s own guidance is aimed at sites with a million or more pages updating weekly, or sites of 10,000-plus pages that change daily; those sites should treat log file analysis as a monthly discipline. Smaller B2B SaaS sites under that threshold can run it quarterly, with an extra pull after any site migration, major redesign, or new faceted or parameter-driven feature that could change how many URLs the site is generating.
Sources
1. Google Search Central, “Crawl Budget Management For Large Sites,” Google for Developers. ↩
2. Search Engine Journal, “Google: Database Speed Beats Page Count For Crawl Budget,” May 2025. ↩
3. Cloudflare, “The 2025 Cloudflare Radar Year in Review,” Cloudflare Blog. ↩
4. Botify, “All About Crawl Budget Optimization,” Botify Blog. ↩
Share this article
Ryan Brooks leads technical SEO at MV3 Marketing, specializing in schema architecture, entity graphs, crawlability, and the structural signals that determine whether AI answer engines cite a page.
Ready to audit your organic growth opportunity?
$2,500 flat. 5 business days. Six deliverables tied to pipeline , not rankings. No retainer required.
Get the Organic Growth Audit →