Every AI-crawler checker tool we could find only reads a site’s robots.txt rules and reports whether a bot is allowed or disallowed. None of them analyze what a bot actually does once it starts crawling. This skill answers that from your own real server logs: how much of the site a bot covers, whether it’s hitting broken pages, and whether its interest is growing or declining over time.
How it works
const { parseLog } = require('./parser.js');
const { analyze } = require('./analyzer.js');
const { entries } = parseLog(logFileContents);
const report = analyze(entries, mySiteUrls); // siteUrls is optional
console.log(report.GPTBot);
// { total_hits: 1204, unique_urls_crawled: 340, error_rate_pct: 4.5,
// crawl_trend: { direction: 'rising', pct_change: 38.2 },
// coverage: { coverage_pct: 82.5, never_crawled: [...] }, flags: [] }
What the flags mean
High error rate: over 20% of a bot’s requests returned 4xx/5xx. Usually means the bot is hitting stale or broken URLs. Crawl frequency dropped: average daily hits fell 25%+ between the first and second half of your log window. Low coverage: fewer than 50% of your listed site URLs were crawled, with the exact never-crawled URLs listed out.
The known-bot list is dated on purpose
AI crawler user-agent strings change as vendors add new bots. known-bots.json is kept separate from the analysis logic and dated so you can verify it against each vendor’s current documentation rather than trusting it as permanent.
FAQ
Is this skill free to use?
Yes. Free to download and run yourself. MV3 charges $175/hr only if you want it run against your real server logs on a schedule.
What log format does it need?
Standard nginx/Apache combined log format. Most hosts, including WP Engine, can export this.
Does my log data get sent to MV3?
No. The parser and analyzer run entirely in your own environment with zero outbound network calls.
How is this different from just checking robots.txt?
robots.txt only says what a bot is allowed to do. This analyzes what it actually did: real coverage, real error rates, and a real crawl-frequency trend from your own logs.
How current is the known-bot list?
Verified 2026-08-12 against each vendor's published documentation. New crawlers get added by vendors regularly, so check known-bots.json's date before relying on it for a compliance decision.
Where do I get support if something breaks?
Open a GitHub Issue for bugs or questions. For paid help running this against your real logs, book a scoping call.
