GPTBot: 1,091 crawls per referral visit

1,091 crawls for a single referral visit — that is the ratio Cloudflare measured for OpenAI's bots in August 2025. This arithmetic is what triggered the wave of GPTBot blocks: the crawler OpenAI shipped on 7 August 2023 to collect data for training its foundation models.

It is identified by the GPTBot token in robots.txt, and the user agent has already reached version 1.4 — which is exactly why rigid regex firewall rules break after every update. OpenAI publishes its address ranges in gptbot.json: 21 IPv4 prefixes, mostly Azure, and no IPv6 entries at all. The full breakdown is over at PPC Land.

What the numbers say:

• 35.7% of the top 1,000 sites were blocking GPTBot back in August 2024 — up from roughly 5% at launch;
• 11.7% of all AI crawling on Cloudflare's network (July 2025), versus 4.7% a year earlier;
• 70.6% of news sites that blocked the bot still show up as citations in AI answers (BuzzStream, March 2026);
• 15% of AI page fetchers in Europe reached disallowed URLs (TollBit, August 2026).

The cost side keeps being revised too: Rutgers and Wharton researchers first put blocking losses at 23.1% of monthly visits, then cut the estimate in April 2026 to about 7% of weekly traffic within six weeks.

The main trap is confusing four different bots. GPTBot collects training data, OAI-SearchBot indexes for ChatGPT search, ChatGPT-User fetches pages on a user's request, OAI-AdsBot checks submitted landing pages. Block them all in one line and you lose ChatGPT visibility while your text still reaches training sets through Common Crawl archives. So the decision stopped being binary: block GPTBot if content is your core asset, but keep OAI-SearchBot open. And check your logs — from 15 September 2026 Cloudflare blocks training crawlers by default on new ad-carrying domains.