Free Tool
AI Crawler Access Checker
An AI engine can only cite what it can read. Check whether your site actually lets the AI crawlers in — before you wonder why you’re never in the answer.
How it works
Enter your domain and the tool reads your robots.txt, then checks it against the fourteen most important AI crawlers — OpenAI’s GPTBot, Anthropic’s ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended and more. You get an instant map of who’s allowed and who’s blocked. If the bots can’t reach you, no amount of great content will get you cited.
Once they’re allowed, guide them to your best content with the llms.txt Generator.
Frequently asked questions
Should I allow AI crawlers like GPTBot?
If you want AI answer engines to find, cite, and recommend you, yes — blocked crawlers can’t read your content, so they can’t quote you. Some publishers block them to protect content or negotiate licensing, which is a different trade-off.
How do I allow AI crawlers?
Edit your robots.txt to add Allow rules (or remove Disallow rules) for user-agents like GPTBot, ClaudeBot, PerplexityBot and Google-Extended, then publish an llms.txt index to guide them.
We’ll email you the complete breakdown from this tool plus the exact fixes to make — no spam.
More free tools
- The Zero-Click Toolkit — all six tools in one place
- The AI Visibility Toolkit — AI Overviews + ChatGPT
- The Agent-Readiness Toolkit — get ready for AI agents
- The Content & Rewrite Toolkit — write content AI cites
- Answer-First Content Checker — is your draft quotable?
- AI Crawler Access Checker — are AI bots allowed in?
- The AI Crawler Blocking Report (data)
- DigiJaws Data (original research)
- llms.txt Checker — do you have a valid llms.txt?
- AI Overview Citation Checker — are you cited?
- The AI Overview Prevalence Benchmark (data)
- ChatGPT Visibility Report — does ChatGPT name you?
- Zero-Click Signature Analyzer — is AI eating your clicks?
- Zero-Click Revenue Calculator — what the gap costs you
- AI Overviews Traffic Tracker
- Keyword & Rewrite Optimizer
- AI Citability Checker — is your page ready to be cited?
- AI Overview Risk Checker — is a keyword a zero-click trap?
- Share of AI Answers — does AI recommend you?
- Schema Markup Generator — FAQ / Article JSON-LD
- The DigiJaws Library — the zero-click reference series