There's a sentence I keep hearing from site owners, and it's about to land a lot of them in trouble:
"We blocked the AI bots."Which ones, though? "AI bots" isn't a single thing. OpenAI alone runs four crawlers, and they do completely different jobs. One trains models. One builds the search index that decides whether ChatGPT can even find you. One fetches your page live, mid-conversation, the instant a user asks something you could answer. Block all of them with one rule and you haven't protected your content. You've deleted your business from the fastest-growing discovery channel on the web, and you've kept none of the leverage you thought you were keeping.
This matters right now because of Cloudflare's September 15 changes. From that date, Agent and Training bots are blocked by default on ad-monetized pages, both for new domains and for zones that never saved an explicit preference. Search bots stay allowed. If you want AI assistants to keep citing you after September 15, the winning setup is deliberate, and it's more granular than a single toggle.
TL;DR: You can block AI training and keep your AI citations at the same time, because different bots control each one. Allow the search indexers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and the user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User). Block the training crawlers (GPTBot, ClaudeBot, CCBot) if you want to, and it won't cost you a single citation. Then make your Cloudflare settings and your robots.txt agree, and verify it with your logs rather than your assumptions.
One AI company, several bots, very different stakes
Every major AI vendor now publishes separate, well-documented crawlers for each purpose. This is exactly the Search / Agent / Training split that Cloudflare's new controls are built around. Here's the map:
OpenAI (ChatGPT)
From OpenAI's crawler documentation:
| Bot | What it does | Block it and… |
|---|---|---|
GPTBot | Crawls content to train foundation models | Nothing visible happens. OpenAI states it "does not affect ChatGPT search results or citations" |
OAI-SearchBot | Builds the index behind ChatGPT search | Your site drops out of ChatGPT search answers |
ChatGPT-User | Fetches pages live when a user's question calls for them | ChatGPT can't read your page mid-conversation, so no live quote and no link |
OAI-AdsBot | Checks landing pages submitted to ChatGPT ads | Only relevant if you run ChatGPT ads |
That first row is the whole game. OpenAI is explicit that blocking GPTBot costs you zero search visibility. Training and visibility are decoupled on purpose. Most blanket "block AI" configs were written before that split existed, so they treat GPTBot and OAI-SearchBot as the same thing. That's the mistake.
Anthropic (Claude)
Anthropic mirrors the same structure with three crawlers:
| Bot | What it does | Block it and… |
|---|---|---|
ClaudeBot | Collects content for model training | You're left out of future training data; visibility is untouched |
Claude-SearchBot | Indexes the web to improve Claude's search results | In Anthropic's own words, it "reduces your site's visibility and accuracy in user search results" |
Claude-User | Fetches your site when a user asks Claude something | Claude can't retrieve your content in response to user queries |
All three respect robots.txt, and Anthropic publishes verification IPs. If ranking inside Claude's answers matters to you, we've covered the content side in the Claude answers playbook. But none of that helps if Claude-User is eating 403s at your edge.
Perplexity
Perplexity runs two, and notably discloses no training crawler at all:
| Bot | What it does | Block it and… |
|---|---|---|
PerplexityBot | Indexes sites for Perplexity search results, and per Perplexity is "not used to crawl content for AI foundation models" | You may vanish from Perplexity results and citations |
Perplexity-User | Visits pages for user-initiated questions | Users can't pull your content into answers; note it "generally ignores robots.txt" because a human triggered the request |
That last note carries weight. User-initiated fetchers like Perplexity-User (and OpenAI says something similar about ChatGPT-User) may not honor robots.txt at all. The only reliable way to control them is at the network layer, which is precisely why Cloudflare's Agent category, and its new default, has real teeth.
Why blocking "Agent" bots quietly kills your citations
Cloudflare's Agent category is where the September 15 default does the most invisible damage, because Agent bots work for a specific human at a specific moment.
Walk through what actually happens. Someone asks ChatGPT for the best invoicing tool for freelancers. ChatGPT decides it needs current information, checks its search index, and sends ChatGPT-User out to read the top candidate pages right now. If your page returns a 200, your content gets read, quoted, and linked in the answer: a warm referral at the exact moment of decision. If your page returns a 403, ChatGPT doesn't note your absence with an apology. It reads whichever competitor it can reach and cites them instead.
The user never learns you existed. And nothing in your analytics tells you it happened, because you can't measure the traffic you never received. This is the defining trait of AI visibility problems: the failure is silent. A ranking drop shows up in Search Console. A citation drop shows up nowhere.
The allow/block matrix
This is the setup we recommend for businesses that want AI assistants recommending them while still opting out of model training:
| Bot | Vendor | Cloudflare category | Verdict |
|---|---|---|---|
OAI-SearchBot | OpenAI | Search | ✅ Allow |
ChatGPT-User | OpenAI | Agent | ✅ Allow |
GPTBot | OpenAI | Training | 🟡 Your call, blocking costs no visibility |
Claude-SearchBot | Anthropic | Search | ✅ Allow |
Claude-User | Anthropic | Agent | ✅ Allow |
ClaudeBot | Anthropic | Training | 🟡 Your call, blocking costs no visibility |
PerplexityBot | Perplexity | Search | ✅ Allow |
Perplexity-User | Perplexity | Agent | ✅ Allow |
CCBot | Common Crawl | Training | 🟡 Your call (feeds many training datasets) |
Google-Extended | Training control | 🟡 Your call. A robots.txt token, not a crawler, and it doesn't affect Google Search | |
Googlebot, Bingbot, Applebot | Mixed-use | Search + Training | ⚠️ Allow explicitly, because the strictest category rule wins |
Two implementation layers, and both need to agree.
Layer 1: Cloudflare dashboard
Go to Security → Settings → Configure AI bot policies and set:
- Search: Allow. Table stakes for appearing in AI search at all.
- Agent: Allow. This is the one the September 15 default flips against you on ad-monetized pages, and, as covered above, the one robots.txt can't reliably control.
- Training: Block if that's your policy, but first add explicit allow rules for Googlebot, Bingbot, and Applebot so the mixed-use logic can't take out your ordinary SEO by accident.
Skip this screen entirely and Cloudflare decides for you on September 15. Existing zones with an explicit preference keep it; everyone else moves onto the new defaults.
Layer 2: robots.txt
Your robots.txt should tell the same story, both because well-behaved training crawlers honor it and because it's a public, machine-readable statement of your policy:
# --- AI search & user-initiated fetchers: keep citations alive ---
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
# --- Model training: opted out ---
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
(Google-Extended controls whether Google may use your content for Gemini training. It doesn't touch Googlebot or your rankings.) On WordPress, our GPTBot playbook walks through where robots.txt actually lives and how plugins fight over it.
Trust your logs, not the settings screen
Every configuration above can fail silently: a WAF rule someone added back in 2024, a rate limit, a managed challenge no bot will ever solve. The settings screen shows you what you asked for. Only your logs show you what's happening. After you configure, and again after September 15, spend ten minutes confirming it:
- Pull edge logs for the AI user-agents. In Cloudflare, Security → Events lets you filter by user-agent. On the server side, grep your access logs for
OAI-SearchBot,ChatGPT-User,Claude-User, andPerplexityBot. - Read the status codes. A healthy setup is a wall of
200s. Failure looks like a403, meaning a firewall or bot rule is blocking (find which rule fired under Security Events), or a429, meaning your rate limiting is throttling legitimate crawlers. A challenge page served to a bot counts as a block too, even though nothing says the word "block." - Check the direction you can't see. Ask ChatGPT (with search on) and Perplexity the questions your site ought to win, and see whether you're cited. Then confirm the referrals are actually landing by tracking AI traffic in your analytics.
One caveat deserves its own future article. Even when robots.txt allows a bot and Cloudflare allows it, access still isn't guaranteed, because hosting-level firewalls, security plugins, and stacked CDNs all get a veto. If you'd rather have this checked for you, a free scan of your site flags AI-access blockers alongside the rest of your GEO setup.
The window this opens
Here's the strategic read. On September 15, a real slice of the web goes dark to Agent-category bots, and not because those owners weighed the trade-offs. A default changed and they didn't look. AI assistants will keep answering the same questions. They'll just source those answers from the sites that stayed reachable.
Visibility in AI answers is becoming permissioned. For once, the edge doesn't go to whoever has the biggest content budget. It goes to whoever configured three settings correctly during the ten weeks everyone else spent ignoring the email.
