Most of the argument about AI crawlers is about whether you should let them in. Publishers sue, Cloudflare ships blocking by default, and the comment sections fill up with people who feel strongly either way.
Almost nobody has measured the boring question underneath it: how many websites are already blocking AI crawlers, and do their owners know?
So we measured it. Between 27 and 30 August 2026 we ran a live crawlability check on 314 websites, fetching each one as 38 different AI crawlers in turn and comparing what came back against what the site's own robots.txt claims. 43% refused at least one AI crawler. Only 23 sites out of 314 had written that rule down anywhere.
The rest of this article is what we found, where the honest caveats are, and the one result that surprised us: the single biggest author of AI crawler rules on these websites was not the person who owns them.
How many websites block AI crawlers?
Every square is one of the 314 websites we checked
179 sites
Fully reachable
Every one of the 38 crawler tokens we tested got through.
112 sites
Blocked, but not in robots.txt
The server turns a crawler away. robots.txt says nothing about it.
23 sites
Blocked in robots.txt
Visible, auditable, reversible in a text editor. 18 of the 23 were written by Cloudflare, not the owner.
135 of the 314 sites refuse at least one crawler. That much matches the intuition anyone who works on this expects.
The split underneath it does not. A written robots.txt rule is visible, auditable and takes ten seconds to reverse. A server refusal is none of those things. The crawler asks for the homepage, the server answers 403 Forbidden, and nothing anywhere in the site's own configuration records that it happened. The owner can read their robots.txt all day and see a welcome mat.
Most affected sites are not blocking wholesale. 80 of the 135 refuse only one or two crawlers, 34 refuse between three and nine, 20 refuse between ten and twenty-nine, and exactly one site refuses all 38. The average of 4.4 refused crawlers per affected site is dragged upward by that tail, so the median of 2 is the more honest number.
Most of it is deliberate, and that is the part people get wrong
The finding it would be easy to oversell is that 43% of sites are accidentally shutting themselves out of AI search. They are not, and the data says so plainly.
Of the 135 sites that refuse a crawler, 102 refuse only training scrapers: Bytespider, Amazonbot, CCBot, Meta-ExternalAgent and their relatives. Those crawlers take content to train models and return nothing. Refusing them is a legitimate, common and increasingly default choice, and a site that does only that has lost nothing in AI search.
The number that should worry you is smaller and sharper. 33 sites refuse at least one crawler that fetches pages in order to answer a question somebody is asking right now: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Claude-User, DuckAssistBot and the rest of that group. Those are the requests that end in a citation with a link.
Of those 33 sites, 2 wrote the rule in robots.txt. The other 31 did not. On 20 of them there is no AI crawler rule anywhere in robots.txt at all, so there is nothing to find even if you go looking.
That is the study in one line. Blocking scrapers is a decision. Blocking the assistants is, in almost every case here, a side effect of blocking the scrapers with an instrument too blunt to tell them apart.
Two ways to shut a crawler out, and only one leaves a note
A robots.txt rule is a published policy. Somebody typed it, or a plugin typed it for them, and any crawler that respects the convention will honour it. You can read it, audit it and delete it.
A server refusal is a fact about the connection. A firewall rule, a bot filter, a security plugin or a CDN setting recognises the user agent and closes the door. There is no note. There is frequently no logging the owner will ever look at. And these rules tend to predate the question entirely: a "block bad bots" toggle switched on in 2021 does not know that ChatGPT-User would exist in 2026.
126 of the 314 sites have at least one server-level refusal. 54 of those sites are refusing from behind Cloudflare and 72 from something at the origin, which turns out to matter a great deal later on.
Cloudflare wrote most of the rules, not the owners
Here is the result we did not expect. Of the 23 sites carrying an AI crawler rule in robots.txt, 18 did not write it. Cloudflare did.
Those 18 files contain a block like this, delivered by Cloudflare's managed robots.txt feature and preceded by several paragraphs of Content Signals Policy boilerplate about Article 4 of the EU copyright directive:
# BEGIN Cloudflare Managed content
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
# END Cloudflare Managed Content
On several of these sites the very next thing in the file is the owner's own SEO plugin, cheerfully allowing everything:
# START YOAST BLOCK
# ---------------------------
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap_index.xml
# ---------------------------
# END YOAST BLOCK
Two configurations, two different intents, one file. The owner's tool says come in. Their CDN, above it, says no to eight named crawlers. Only 5 of the 314 site owners in this study wrote an AI crawler rule themselves.
This is also where the widely repeated "22 sites opted out of Google and Apple AI training" framing needs correcting, including in our own earlier draft of this data. 19 sites disallow Google-Extended or Applebot-Extended, the two tokens that are not crawlers at all but consent switches governing whether Google and Apple may use your content in AI answers. 18 of those 19 also disallow GPTBot, CCBot and Bytespider in the same breath. These are not sites that made a considered decision about Google's AI training. They are sites that received a standard list.
A separate 7 sites carry Cloudflare's Content-Signal line with no Disallow rules behind it. That is a stated preference with nothing enforcing it, and no major AI company has committed to reading it. We wrote about what changes in Cloudflare's September 15 defaults when the announcement landed in July. This is what the early version of it looks like in the wild, five weeks before the deadline.
What the refusal actually looks like
Every refusal we recorded, counted only where robots.txt allowed that crawler, so the table is specifically the silent kind. One site can appear in more than one row.
| Response | What it is | Refusals | Sites |
|---|---|---|---|
403 Forbidden | A flat refusal. A firewall rule or security plugin recognised the user agent. | 509 | 113 |
| Cloudflare challenge | A browser check the crawler cannot solve. To a bot, an unsolvable puzzle is a locked door. | 24 | 7 |
406 Not Acceptable | Rejected on sight, usually from a user-agent rule at the web server. | 20 | 5 |
Cloudflare cf_mitigated | A 403 issued by a Cloudflare managed rule rather than a customer rule. | 5 | 1 |
558 refusals in total. 93% of them are a plain 403, which is the signature of a rule keyed to the user agent string and nothing else.
Rate-limit responses are excluded from every figure in this study. Our check fires around 30 requests at a site in a burst and can trip a rate limiter by itself, so a 429 says something about our traffic rather than about the site's policy. Counting them would have added 107 refusals across 39 sites, and inflated the headline. We would rather the methodology survived scrutiny than the number looked bigger.
Which crawlers get turned away
Share of the 314 sites that refuse each crawler
written in robots.txtrefused by the server, nothing written down
The order is the argument. Bytespider and Amazonbot sit at the top because people block them on purpose, and the amber segment shows that a meaningful share of those blocks were written down deliberately.
Everything below GPTBot is almost pure red. PerplexityBot is refused by 8.0% of sites and exactly one of them wrote a rule about it. OAI-SearchBot, the crawler that builds ChatGPT's search index, is refused by 7.0% and again by one written rule. These crawlers are not being blocked. They are being caught.
Does the block work? We checked it against the server logs
A crawlability check tells you what our request received. It does not, by itself, tell you what the real crawler receives. So we tested the check against four weeks of actual server logs from the same sites, covering 1 to 26 August, before the check ran.
Share of sites that logged a real page request from the crawler, 1 to 26 August 2026
Grouped by what our check found on the same sites a few days later.
GPTBot
ClaudeBot
PerplexityBot
OAI-SearchBot
Two things fall out of this, and the second one is more useful than the first.
The check is measuring something real. Sites where we found GPTBot refused at the edge logged zero GPTBot page requests in four weeks. Sites where we found it reachable logged some on 80.4% of them. The same pattern holds for ClaudeBot, 79.9% against 4.2%, and for PerplexityBot, 65.1% against 11.1%.
Where the block lives decides whether it works. An edge block, at Cloudflare, happens before the request ever reaches your site, so the crawler gets nothing and your logs record nothing. An origin block, from a security plugin or a server rule, happens after the request arrives. That is why origin-blocked sites still show high log rates: the request lands, gets recorded, and is then refused. The crawler still leaves empty-handed.
For OAI-SearchBot the contrast is stark. Sites blocking it at the origin logged a page request 84.6% of the time, barely below the 87.3% of fully reachable sites. Sites blocking it at the edge logged one 0.0% of the time. If you are trying to work out whether you have a problem by looking at your own analytics or a crawler-stats plugin, an origin block is close to invisible in exactly the way a real refusal is not.
Robots.txt sits in between, as you would expect from a convention rather than an enforcement mechanism. 33.3% of the sites disallowing GPTBot still logged GPTBot page requests, and 17.6% of those disallowing ClaudeBot logged ClaudeBot. Neither figure includes robots.txt or sitemap fetches, which are not ingested into these logs, so every request counted is a crawler asking for a page it was told not to take.
What we could not show
The result you would most want from this study is that blocked sites earn less AI traffic. We cannot show it, and it would be easy to pretend otherwise.
Across June to August, median AI referral visits per site ran at 12 for the 269 reachable sites, 6 for the 11 sites blocking OpenAI's crawlers at the edge, and 5 for the 17 blocking them at the origin. That points the right way. Then the 17 sites disallowing OpenAI in robots.txt came in at 19, higher than the reachable group, which is the opposite of the story.
With groups of 11 to 17 sites and no control for site size, page count or niche, these differences are not separable from noise. This study establishes that a large minority of sites refuse AI crawlers and that most of them cannot see it from their own configuration. It does not establish what that costs them. For evidence on the other end of the funnel, our 108,000-page study of AI search optimization measures referral traffic properly, with a control group and a matched window.
Methodology
Data source. First-party crawlability check records and bot logs from the LovedByAI platform. Nothing here comes from a third-party estimator.
What the check does. For each website it fetches robots.txt, evaluates it against 38 AI crawler and consent tokens exactly as those crawlers evaluate it, then sends a live request to the homepage using each crawler's own user agent, plus an unbranded control request from the same address.
Sample. One check per website, the most recent completed run between 27 and 30 August 2026, giving 326 checks. 12 are excluded because the unbranded control request was itself refused, so nothing about those sites is AI-specific. The study sample is 314 websites.
Who these sites are. Paying LovedByAI customers, overwhelmingly small and medium business sites on WordPress. The monthly sweep does not cover free-tier sites. Every figure describes these 314 sites and should not be read as a figure for websites in general.
Definition of a robots block. The site's robots.txt disallows that crawler's token from the homepage path.
Definition of a server refusal. robots.txt allows the crawler, and the live request carrying that crawler's user agent returned 403, 406 or an unsolvable Cloudflare challenge, while the unbranded control request from the same address succeeded. Because the control succeeded, the refusal is attributable to the user agent string rather than to our IP address or to the site being down.
429 exclusion. Rate-limit responses are excluded everywhere. The check sends around 30 requests in a burst and can trigger a rate limiter unaided. Including them would have added 107 refusals across 39 sites.
Site categories. Mutually exclusive and ranked by visibility: a site with any AI crawler rule in robots.txt is counted as declared (23), a site with no such rule but at least one server refusal is counted as silent (112), everything else is reachable (179). 14 sites have both a written rule and a server refusal and are counted under the written rule.
Consent tokens. 8 of the 38 tokens are consent switches with no crawler behind them and are therefore evaluated against robots.txt only, never probed: anthropic-ai, Applebot-Extended, Google-Extended, Google-NotebookLM, Grok-User, GrokBot, Grok-bot and xAI-Bot. They can appear as a robots block and never as a server refusal.
Assistant crawlers. The 33-site figure counts a site if any of these were blocked either way: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Perplexity, Claude-SearchBot, Claude-User, Claude-Web, DuckAssistBot, YouBot, MistralAI-User, DeepSeek-User, Google-Agent, Google-GeminiNotebook, Google-NotebookLM, Meta-ExternalFetcher, Grok-User.
Log validation. Bot requests from 1 to 26 August 2026, entirely before the check window, on the assumption that WAF and firewall configuration is stable over weeks. The metric is the share of sites in each group that logged at least one request from that crawler, chosen over request volume because crawl totals in this fleet are dominated by a handful of large sites. Requests for .txt and .xml paths are not ingested, so robots.txt and sitemap fetches cannot inflate the figure.
Privacy. No website is identified, and no site-level data is published. Only counts, crawler tokens and HTTP response codes. The robots.txt excerpts above are the verbatim Cloudflare and Yoast boilerplate that appears identically across many sites, with the domain replaced.
Known limits. Not a random sample of the web. A single homepage fetch per crawler, so a site could refuse a crawler on inner pages and pass here. Configuration is measured at one moment, and a bot-fight mode that samples traffic could refuse our request and admit the next one. The log validation is observational: it shows that the check's verdict predicts what appears in the logs, not that the block caused the absence.
Reproducibility. Every figure comes from one set of SQL queries against the production crawlability and bot-log tables. If you want the query behind a specific number, ask us and we will send it.
What to do about this
1. Read the server's answer, not robots.txt. 112 of these 314 sites have a robots.txt that tells you nothing useful about who can reach them. Fetching your own homepage as GPTBot with curl -A takes a minute and will tell you more than reading your configuration for an hour. Our free AI search checker does it for all 38 tokens at once and names each one that gets refused.
2. Decide which group you actually mean. Blocking Bytespider, Amazonbot and CCBot is a real position and 102 sites here hold it consistently. Blocking OAI-SearchBot and ChatGPT-User is a different decision entirely, and 31 sites appear to have made it by accident. Whatever you choose, write it in robots.txt so the next person can see it, then make the firewall match.
3. Check what your CDN has written on your behalf. 18 of the 23 written rules in this study came from Cloudflare's managed content block, not from the site owner. If you are on Cloudflare, open your own robots.txt in a browser and look for the # BEGIN Cloudflare Managed content marker before September 15 changes the defaults again.
If your site turns out to be reachable and you still are not showing up in AI answers, crawlability was never the bottleneck and the problem is further down the page. The signs a website is invisible to AI search covers what to look at next.
