Your shopping cart is empty!
Cloudflare and AWS now let sites charge AI bots for crawling
Publication date: 26.07.2026
Charge or let them pass? Site owners now face a new choice: bill AI crawlers for access, or let them in free for the sake of being cited in AI answers. There's no clean answer — and that's the whole debate.
The case for charging. For thirty years crawlers paid in traffic: they indexed your content and sent visitors back. AI bots increasingly take the text and return nothing. The industry's answer is paid access via the HTTP 402 status. Cloudflare (July 2025) and AWS (June 2026) launched pay-per-crawl at the edge, joined by TollBit and Akamai — per-request billing runs on the x402 protocol.
The case against. The bot you'd invoice and the bot that sends you a customer from an AI answer are often the same pipeline. Per Adobe, AI-referred traffic to retailers jumped 393% year over year. Block access and you can easily vanish from the answers already bringing buyers. The full breakdown is on Search Engine Journal.
Who should do what:
- Charging makes sense for news outlets, reference databases and archives with independent distribution;
- Not risking it suits sites and e-commerce that live on visibility inside AI answers.
The paradox is that cutting access is technically easy now — Cloudflare and AWS both ship HTTP 402 — but telling apart a bot stealing content from one sending a paying customer is something almost nobody can do yet. Until that line gets clearer, every billing decision is a guess.
