Why Google Ignores Robots.txt — and How to Fix It

"With robots.txt, the more specific rules win, so if you have a user-agent: Googlebot section, it will only use that section" — that's how Google's John Mueller explained why your robots.txt may not behave the way you intended. It isn't a bug; it's how the file is read.

The idea is simple. If your robots.txt has a general user-agent: * block and a separate user-agent: Googlebot block, Googlebot reads only its own section and ignores the general one entirely. Every rule you wrote under * effectively doesn't exist for it.

On a real case it played out like this: a Shopify client saw internal-search spam creep into the index even though /search was disallowed in the general block. Googlebot never saw it — the rule wasn't in its section. The full write-up is on Search Engine Journal.

What to do so you don't hit the same trap:

• open your robots.txt and look for a separate user-agent: Googlebot section — if it exists, audit it on its own;
• copy every needed Disallow from the general block into the Googlebot section, or group the bots under shared rules;
• remember: robots.txt controls crawling, not indexing — to guarantee a page stays out of search, use <meta name="robots" content="noindex">;
• after editing, validate the file in a robots.txt tester and watch whether the junk drops out of the index.