How Bot-Check Screens Drop Pages From Google

Google's John Mueller flagged a subtle but dangerous issue: "are you a bot" verification screens and similar security interstitials can push a site's pages out of Google's index. Worse, a competitor's site can end up chosen as the canonical for your query.

The mechanism is simple. When a security layer (CDN, host, or security service) treats a visitor as suspicious, it serves an interstitial instead of the content. The trouble is that the same barrier sometimes fires for Googlebot too. As Mueller put it, "Google is able to reach the site and get a valid response, the issue is that it's receiving the wrong content."

• Google indexes the interstitial screen, not your actual page;
• these screens look nearly identical across many sites, so Google picks one as canonical and treats the rest as duplicates;
• your page can be marked a duplicate of another domain;
• normal browsing won't reveal it — you simply pass the screen yourself.

According to Search Engine Journal, the only way to catch this is via Search Console's indexing reports, which flag the page as a duplicate or show another domain's URL selected as canonical.

The irony is that a site owner is usually the last to find out — from a Search Console report quietly flagging the page as a duplicate of someone else's domain. As long as an anti-bot filter runs without an exception for Googlebot, part of a site can sit indexed under the wrong URL, bleeding traffic with no warning at all.

Related reading: Indexed and Non-Indexed Pages: Common robots.txt and Noindex Mistakes · SEO site audit