Your shopping cart is empty!
Cloudflare will write your robots.txt, and it cannot tell AI bots apart
Publication date: 25.09.2026
In short
- On 21 August 2026 Cloudflare announced Bot Preference Sync, which writes robots.txt for you from your dashboard settings.
- The policy can only be set for three bot categories — Search, Agent and Training — not for individual crawlers.
- For new customers the feature is on by default from 15 September 2026.
Allowing GPTBot while blocking Bytespider is not possible in Cloudflare's new feature: there are no per-bot settings.
Slobodan Manic writes about it in Search Engine Journal, and his analysis is worth reading for anyone whose site sits behind Cloudflare.
Three categories instead of a list of crawlers
Bot Preference Sync generates robots.txt entries from the settings in the panel. They are placed between the lines # BEGIN Cloudflare Bot Preference Sync and the matching end marker, while the rest of the file stays below unchanged.
The feature is available on all plans. Bots are split into Search, Agent and Training, and each category offers the same three options: block everywhere, block only on pages with ads, or allow.
| Category | Option 1 | Option 2 | Option 3 |
|---|---|---|---|
| Search | block everywhere | block on pages with ads | allow |
| Agent | block everywhere | block on pages with ads | allow |
| Training | block everywhere | block on pages with ads | allow |
Manic shows why that is not enough. His own policy distinguishes between companies: he allows the training crawlers of GPTBot, Claude and Perplexity, and blocks Bytespider and Meta, based on what each side gives back.
Cloudflare's three categories cannot express that decision.
He also points out that a robots.txt rule stops only the crawler that has decided to stop. To one that ignores it, the rule does nothing.
Cloudflare's conditions and the default for new customers
Cloudflare published four disclosure conditions a crawler has to meet to avoid being blocked: respect a "no training" signal, allow opting out of AI summaries, show publishers URL-level metrics, and demonstrate publicly that blocking training does not hurt search rankings.
By Manic's assessment only Microsoft meets them fully, while Google mixes snippets in ordinary search with participation in AI Overviews and therefore falls outside the conditions.
That is the author's assessment: at the time of publication Cloudflare has not stated the procedure by which it will verify compliance with those four points.
Manic gives a separate warning about the default. The wording "starting point will not add any blocks on your behalf" sounds gentle, but as he describes it, new customers get the Training and Agent categories blocked on pages with ads by default.
For a site that earns from impressions that may be exactly what is wanted. For a site that needs visibility in AI answers, the opposite.
What to make of it if your site is behind Cloudflare
First: check your file. Tell the team that part of robots.txt is now written in the Cloudflare panel rather than only in the repository, and compare the current rules with the bot policy before the feature reaches your account.
Second: remember that AI crawlers and search bots are different things. How these rules interact with indexing is covered in our guide to robots.txt and sitemap.xml and in the piece on indexing and robots.txt.
I'd also keep a dated copy of the current robots.txt. When unfamiliar lines appear between the BEGIN and END markers, you will have something to compare against.
It will be interesting to see how many sites find a block between BEGIN and END lines in their robots.txt without understanding where it came from.
