Quick answer: robots.txt is a plain text file at the root of your domain (yourdomain.com/robots.txt) that tells crawlers which parts of your site they may request. It does not remove pages from Google’s index, does not password-protect anything, and a single misplaced rule can quietly block Googlebot from your CSS and JavaScript, tank your rendering, and cost you rankings for months before anyone notices.
For a B2B SaaS site, robots.txt carries more weight than most teams assume. You have marketing pages, a docs subdomain, an app subdomain, staging-style paths that are actually live, and now a growing list of AI crawlers deciding whether to train on or cite your content. Get the file wrong and you can accidentally block the exact bots you want indexing your pricing page. Get it right and it becomes a small, boring, load-bearing piece of your SEO infrastructure.
What Is robots.txt, Exactly?
robots.txt implements the Robots Exclusion Protocol, a voluntary standard that well-behaved crawlers check before requesting URLs on your site. It lives at exactly one location: the root of the domain, over HTTPS if your site is HTTPS. A file at yourdomain.com/blog/robots.txt does nothing. Each rule set targets a user-agent (a specific bot, or the wildcard * for all of them) and lists Disallow paths that bot should skip and, optionally, Allow paths that override a broader disallow.
What robots.txt does not do matters just as much. It does not de-index a page that is already indexed. It does not enforce access control, since it is a request, not a lock, and malicious bots ignore it entirely. And it does not stop a page from appearing in search results if other sites link to it, since Google can still index the URL from the link context even without crawling the page itself. For true removal, you want a noindex meta tag or HTTP header, not a Disallow rule, and the two should never be combined on the same URL since a blocked page can’t be crawled to see the noindex tag in the first place.
Google’s own reference: for the canonical technical spec, see Google Search Central’s Introduction to robots.txt and How to write and submit a robots.txt file.
Three Mistakes That Quietly Break B2B SaaS Sites
1. Blocking the CSS and JS Googlebot needs to render your app
The most common failure on SaaS sites is a leftover Disallow: /wp-content/, Disallow: /assets/, or Disallow: /_next/ rule that was meant to hide something else, but also blocks the stylesheets and scripts your marketing pages depend on. Googlebot renders pages with a headless Chromium instance before indexing them. If it can’t fetch your CSS and JS, it sees a broken, unstyled, sometimes blank layout, which can suppress how much of your content gets indexed and how the page is understood for ranking. This is the single most searched, lowest-competition problem in this space for a reason: it’s easy to cause and easy to miss, because the page looks completely normal in a browser.
2. Blocking paths that look like staging but are actually live
SaaS marketing sites accumulate paths like /beta/, /preview/, /v2/, or /app-demo/ that started as internal or staging routes and quietly became real, linked, indexable pages. A rule written months ago to keep a staging folder out of search can end up blocking a live product page nobody remembers is covered by that Disallow. Audit your robots.txt against your current live sitemap, not against what the site looked like when the rule was written.
3. Conflicting or redundant rules across user-agent blocks
Once a robots.txt file has more than two or three user-agent blocks (which is normal once you’re managing AI crawlers separately from search bots), it’s easy to write a specific rule for one bot that contradicts the wildcard block below it, or to disallow a path for * and then forget to add an explicit Allow for the one bot you actually wanted to permit. Crawlers apply the most specific matching rule, not the first one they encounter, so overlapping rules don’t fail loudly. They just quietly do the wrong thing.
Traditional Search Bots vs. AI Crawlers: What Belongs in a 2026 robots.txt
A modern robots.txt for a B2B SaaS site has to account for two different populations of bots with two different purposes: search indexing bots that decide whether you rank, and AI crawlers that decide whether a model can cite, summarize, or train on your content. They are not the same decision, and treating them identically is a common gap.
None of these AI crawlers are required to respect robots.txt the way Googlebot does, since the exclusion protocol is voluntary, but the major labs have publicly documented that their crawlers do check it, which makes it the primary lever most sites actually have. If you want AI answer engines citing your content, the practical move for most B2B SaaS marketing sites is to allow the crawlers tied to answer surfaces (PerplexityBot) and make a deliberate, documented choice about the ones tied to model training (GPTBot, Google-Extended, ClaudeBot), rather than leaving the default silently open or silently blocked.
Good vs. Bad: What This Looks Like in Practice
Common mistake
User-agent: * Disallow: /wp-content/ Disallow: /assets/ Disallow: /beta/
Blocks every bot, including Googlebot, from rendering assets, and silently blocks a live product path under /beta/.
Corrected version
User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php User-agent: PerplexityBot Allow: / Sitemap: https://yourdomain.com/sitemap.xml
Only blocks the admin backend, leaves rendering assets and live marketing paths open, and points crawlers to the sitemap.
How to Audit Your robots.txt
A robots.txt audit takes fifteen minutes and should happen every time you launch a new section, migrate a subdomain, or notice an unexplained drop in indexed pages.
- Read the live file directly. Load yourdomain.com/robots.txt in a browser. Confirm it’s serving a 200 status, not a redirect or a 404, and that it’s plain text, not HTML.
- Check the robots.txt report in Google Search Console. This is the current version of what used to be called the robots.txt Tester, and it shows the exact file Google last fetched, flags syntax errors, and lets you test whether a specific URL is blocked. See Google’s guidance at Create and submit a robots.txt file.
- Spot-check the pages you most need indexed. Pricing, product, docs landing pages, and your highest-traffic blog posts. Confirm none of them fall under a Disallow rule, intentionally or by accident.
- Confirm CSS and JS aren’t blocked. Use the URL Inspection tool’s rendered screenshot in Search Console. If the rendered version looks broken or unstyled compared to what a real visitor sees, an asset path is likely disallowed.
- Review your AI crawler rules on purpose, not by default. Decide, in writing, which of GPTBot, Google-Extended, PerplexityBot, and ClaudeBot you want crawling your site, and make sure the file reflects that decision rather than whatever was copied from a template two years ago. Our AI crawler log file analysis framework covers how to verify which bots are actually hitting your site, and what they’re reading, before you write that decision into robots.txt.
If you want this folded into a broader technical health check rather than handled as a one-off, it’s one of the checks we cover in our full technical SEO audit checklist for B2B SaaS, alongside crawl budget, indexation, and Core Web Vitals.
Frequently Asked Questions
What is robots.txt?
robots.txt is a plain text file at the root of a domain that tells web crawlers which URLs they’re allowed to request. It’s a voluntary standard, so well-behaved bots like Googlebot check it before crawling, but it isn’t a security control and doesn’t remove pages from search results.
Why is Googlebot blocked by robots.txt on my site?
Usually because a Disallow rule targeting the wildcard user-agent (*) is broader than intended, commonly a folder like /wp-content/, /assets/, or /_next/ that also happens to hold the CSS and JS files needed to render your pages. Check the exact rule in Search Console’s robots.txt report and narrow the Disallow path.
What’s the best robots.txt tester?
Google Search Console’s robots.txt report is the most reliable option since it shows the exact file Google fetched and lets you test specific URLs against it. It replaced the older standalone robots.txt Tester tool but does the same job.
Does robots.txt stop AI models from training on my content?
It can, for crawlers that choose to respect it, like GPTBot, Google-Extended, and ClaudeBot. It’s a request the crawler operator has agreed to honor, not a technical block, so it only works against crawlers that follow the standard.
Should I block AI crawlers entirely?
Not by default. Blocking everything removes you from AI-generated answers and citations along with training data. Most B2B SaaS sites are better served by allowing crawlers tied to live answer surfaces, like PerplexityBot, and making a deliberate call on pure training crawlers based on their own content strategy.
robots.txt is one file, but it sits upstream of everything else your SEO and GEO work depends on: if a crawler can’t reach your pages, none of your content, schema, or link-building work gets seen in the first place. If you want a second set of eyes on your file, your sitemap, and the rest of your crawlability setup, our SEO services start with exactly this kind of technical audit.
Share this article
Ryan Brooks leads technical SEO at MV3 Marketing, specializing in schema architecture, entity graphs, crawlability, and the structural signals that determine whether AI answer engines cite a page.
Ready to audit your organic growth opportunity?
$2,500 flat. 5 business days. Six deliverables tied to pipeline , not rankings. No retainer required.
Get the Organic Growth Audit →