Your shopping cart is empty!
Googlebot reads only its own robots.txt section and ignores the general rules
Publication date: 26.07.2026
"More specific rules win: if there is a user-agent: Googlebot section, the bot will use only that one" — this is how Google's John Mueller explained why your robots.txt may not work the way you intended. The problem is not in the directives themselves but in how the file is read.
How Google reads robots.txt
The point is simple. If robots.txt contains a general user-agent: * block and a separate user-agent: Googlebot one, Googlebot reads only its own section and ignores the general one entirely. Every disallow written under * effectively does not exist for it.
This is not a bug or a new rule — it is how the standard works: a bot looks for the most precise name match and then follows only that block. So a file where "everything is closed" in the general section can be fully open to Google's main crawler, and nothing shows it from the outside until junk starts appearing in the index.
A case from practice
In a real case it played out like this: a client on Shopify had internal search spam crawling into the index even though /search was disallowed in the general block. Googlebot never saw it — that rule was not in its section. The full breakdown is on Search Engine Journal.
Such sections usually appear historically: someone once added a separate Googlebot rule for a single page, and then for years kept adding disallows to the general block without remembering they do not apply to Google.
Who is most exposed
The risk is highest where a site generates many utility URLs by itself: online stores with filters and sorting, sites with internal search, platforms with calendars and pagination. In those projects the general robots.txt block is usually long — and it is exactly the block that turns out to be inactive for Googlebot.
Sites migrated from another CMS or platform deserve a separate check: robots.txt is often carried over wholesale, including old bot-specific sections that ended up there by accident.
Crawling is not indexing
| Tool | What it does | When to use it |
|---|---|---|
| Disallow in robots.txt | prevents crawling | utility sections, filters, internal search |
| Meta robots noindex | prevents appearing in search | the page must stay crawlable but out of the SERP |
| Both at once | a conflict | the bot never sees noindex because it never loads the page |
This is the most common mistake: a page is blocked in robots.txt and then people wonder why it stays in search results. For noindex to work, the page has to be crawlable — that pair is covered in detail in the guide on indexation and robots.txt.
Duplicating rules for Googlebot
- open your robots.txt and look for a separate
user-agent: Googlebotsection — if it exists, review it on its own; - duplicate every required
Disallowfrom the general block into the Googlebot section, or merge the bots under shared rules; - remember that robots.txt controls crawling, not indexing — to keep a page out of search results reliably, use
<meta name="robots" content="noindex">; - after the edits, test the file in the robots.txt testing tool and watch whether the junk leaves the index;
- review Googlebot-Image and Googlebot-News sections separately if you have them — the same rule applies there.
Five minutes to open the file and line up the sections — and that's usually enough to explain why pages that looked closed for months were quietly sitting in the index the whole time.
