The AI Crawler Blocking Report

DIGIJAWS DATA · JULY 2026

The AI Crawler Blocking Report

How often do the web’s biggest sites block AI crawlers from reading their content? We checked the live robots.txt of 44 major sites across six industries. The short answer: 43% block at least one AI crawler — but it’s wildly uneven.

43%
of major sites block at least one AI crawler
4.5
AI bots blocked on the average site (of 14 tracked)
100%
of news sites block at least one AI crawler
0%
of marketing, SaaS & big-tech sites block them

Most-blocked AI crawlers

Share of the 44 sampled sites that block each bot in robots.txt. Anthropic’s ClaudeBot and ByteDance’s Bytespider are the most-blocked.

ClaudeBot · Anthropic40.9%
Bytespider · ByteDance40.9%
CCBot · Common Crawl38.6%
PerplexityBot · Perplexity36.4%
Applebot-Extended · Apple36.4%
meta-externalagent · Meta36.4%
anthropic-ai · Anthropic34.1%
Amazonbot · Amazon34.1%
GPTBot · OpenAI29.5%
cohere-ai · Cohere29.5%
Google-Extended · Google25.0%
Perplexity-User · Perplexity25.0%
ChatGPT-User · OpenAI20.5%
OAI-SearchBot · OpenAI18.2%

Blocking by industry

Share of sites in each category that block at least one AI crawler, and the average number of bots blocked.

News & Publishers
100%
block an AI crawler · avg 10.5 of 14 bots blocked
Reference & UGC
71%
block an AI crawler · avg 8.4 bots blocked
Ecommerce & Retail
29%
block an AI crawler · avg 2.9 bots blocked
SaaS & Software
14%
block an AI crawler · avg 0.1 bots blocked
Marketing & SEO
0%
block an AI crawler · doors wide open
Big Tech
0%
block an AI crawler · doors wide open
Research report graphic for “The AI Crawler Blocking Report” with headline metric tiles and a trend chart

What the data says

The AI-crawler debate is really a publisher debate. Every news site we checked blocks at least one AI crawler, and the average news site blocks more than ten of the fourteen bots we track — a direct response to AI answer engines summarizing their journalism without sending clicks back. Reference and user-generated-content sites (Reddit, Quora, Wikipedia and peers) are close behind at 71%.

Everyone else is leaving the door open. Not a single marketing, SEO, SaaS, or big-tech site in our sample blocks AI crawlers — these companies want to be trained on and cited. That split matters: if you sell to businesses, your competitors are almost certainly allowing AI crawlers, so blocking them mostly removes you from the answers your buyers now ask AI engines for.

Among individual bots, Anthropic’s ClaudeBot and ByteDance’s Bytespider are the most-blocked (40.9% each), while OpenAI’s search-facing bots (OAI-SearchBot, ChatGPT-User) are the least-blocked — sites appear more willing to be cited in live answers than to be used for model training.

Is your site blocking the crawlers that feed AI answers?

Check your own robots.txt against all 14 AI crawlers in seconds — free, no signup.

Run the AI Crawler Access Checker →

Methodology

In July 2026 we programmatically fetched the live robots.txt of 44 well-known sites spanning six categories (news & publishers, reference & UGC, ecommerce, SaaS, marketing/SEO, and big tech) using the DigiJaws AI Crawler Access Checker. For each site we recorded whether the file explicitly disallows each of 14 AI crawlers — GPTBot, OAI-SearchBot, ChatGPT-User (OpenAI); ClaudeBot, anthropic-ai (Anthropic); PerplexityBot, Perplexity-User (Perplexity); Google-Extended (Google); Applebot-Extended (Apple); Amazonbot (Amazon); meta-externalagent (Meta); Bytespider (ByteDance); CCBot (Common Crawl); and cohere-ai (Cohere). A bot is counted as “blocked” when robots.txt disallows the site root for that user-agent. This is a directional sample of major sites, not a census; we refresh it periodically. DigiJaws, July 2026.

Illustration: an AI crawler scanning a site blueprint with a sitemap, structured data and a plain-text file, passing access checks
From diagnosis to done
The free check finds the gaps. The platform closes them.

DigiJaws monitors your AI visibility continuously, alerts you when a crawler rule or signal drops, and generates the publish-ready fixes — across your whole site, on autopilot.

See the platform →View pricing

Frequently asked questions

Which AI crawlers should I allow?

Separate them by what they do. Crawlers that feed answer engines and cite sources, such as OAI-SearchBot, ChatGPT-User and PerplexityBot, are how you get named in answers, so blocking them removes you from that surface. Tokens that govern training corpora, such as Google-Extended and Applebot-Extended, are a different decision with no direct visibility cost. Most businesses are best served allowing the first group and deciding the second on principle rather than on traffic.

Does blocking an AI crawler remove content already collected?

No. A robots.txt rule governs future requests only. Anything already fetched remains in whatever system it reached, and copies of your material on sites you do not control are entirely unaffected. Blocking is a forward-looking control, so it is worth deciding early rather than treating it as something you can undo later.

Is blocking AI crawlers bad for SEO?

Not for classic search rankings, provided you keep Googlebot and Bingbot allowed. Those are separate user agents from the AI-specific ones, and blocking Google-Extended does not affect how the same pages rank in Google search. The risk is narrower than people assume: what you lose is presence in AI-generated answers, not position in the results page.

How do I check which crawlers my site currently blocks?

Read your robots.txt and look for the specific user-agent tokens rather than assuming a wildcard covers them, because many AI crawlers only honour rules written against their own name. Then confirm against your server logs, since a rule and actual behaviour are different things, and a CDN or firewall can block a crawler that robots.txt appears to allow.

More free tools

Run your free scan →

Free tools →My toolkit →Locations →Comparisons →
Verified Agent-Ready by DigiJaws
See who AI recommends instead of you — free AEO audit, 90-day plan you keepGet my free audit