How to Let AI Crawlers Access Your Site (Step by Step)

If your robots.txt blocks AI crawlers, you are invisible to AI search no matter how good your content is. The fix takes minutes: confirm the major AI bots are allowed, and explicitly welcome them. Here is exactly how.

Steps

1. Open your robots.txt at yoursite.com/robots.txt.
2. Look for any Disallow: / under a wildcard User-agent: *, or rules naming AI bots. That is what blocks them.
3. Add explicit allow rules for the major AI crawlers: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and Google-Extended (controls Gemini and AI Overviews grounding).
4. Keep your Sitemap: line so crawlers can still find your pages.
5. Save, then re-open the file to confirm the changes are live.

Notes

Blocking AI crawlers does not protect your content from competitors — it just removes you from the answers buyers see. Unless you have a specific reason to opt out, allow them. Google-Extended is separate from regular Googlebot; blocking it can remove you from Google AI Overviews while leaving classic search intact.

Check it automatically

The free SEO Agent tells you instantly whether AI crawlers can reach your site, as part of your AI-readiness score. Back to the Get Found by AI playbook.