Google has added a section titled “Only index content meant for Search” to its guide to preventing user-generated spam, dated Oct. 26. The change to the Google spam guide recommends that platforms by default apply noindex or block crawling on pages created for direct sharing. For operators of sites that host user content, it is a concrete instruction about which pages should stay out of the Search index.
What Google added to the spam guide
According to Search Engine Journal, the new section concerns webpages designed “for direct sharing, collaboration, or utility rather than public search discovery.” Both the guide and its changelog carry the Oct. 26 addition, the outlet reports, and the changelog describes it as blocking indexing of link-shared content.
The recommendation is aimed at platforms, and it sets a default. Google advises that these pages be kept out of Search by default.
The guide itself addresses user-generated spam, meaning unwanted content that users post on sites. With the new section, that topic now extends from what users post to which of the resulting pages belong in the index at all.
The addition also widens the scope of the Google spam guide. Before, the guidance centered on comment forms and hosting services. It now also covers tools that let users publish shared pages, generated content or web apps. The opening sentence was adjusted to match and now refers to pages or sites rather than sites alone.
That change matters for products that were never thought of as user-generated content platforms. Shareable pages produced by these tools are exactly the ones the new default is meant for, even when the platform itself has no comments or forums.
The two options Google offers
Google names two ways to keep these pages out of Search. A platform can apply noindex, or it can block crawling in robots.txt.
Noindex
Noindex is a robots meta tag or an X-Robots-Tag HTTP header that tells search engines not to index a page. The meta tag sits in the page itself, while the HTTP header is sent with the page response. Google’s guidance covers both forms.
Robots.txt crawl block
Robots.txt is a file that tells search engine crawlers which URLs they can access on a site. A block there is the second route Google lists for these pages.
Why the choice between the two matters
The two options do not behave the same way, and Google’s own documentation explains the gap. Its robots.txt documentation says the file “is not a mechanism for keeping a web page out of Google.”
A page disallowed in robots.txt can still be indexed. That happens if other sites link to it. In that case its URL can appear in results without a description, according to Google for Developers.
The same documentation says that to keep a page out of Google, operators should block indexing with noindex or password-protect the page. That is a different mechanism from a crawl block, and the difference changes what a platform actually achieves.
Noindex needs crawl access
A noindex rule only works if Googlebot can crawl and read the page. Putting noindex on a URL that is also blocked in robots.txt would therefore keep Googlebot from seeing the rule.
The two options are listed as alternatives in the guidance. For a platform deciding between them, the documentation points to noindex as the way to keep a page out of the index, and to a crawl block as a way to control access.
Limits of robots.txt
Robots.txt rules cannot enforce crawler behavior. Reputable crawlers such as Googlebot obey them, while other crawlers might not. A block in the file is a request to well-behaved crawlers, not a lock.
Password protection, which the documentation also names, is the option that restricts who can open a page at all.
Monitoring advice in the guide
The guidance includes two monitoring steps. Google recommends verifying a website in Search Console so the owner receives notifications of manual actions and security issues.
It also recommends checking the Page Indexing and Search performance reports for “unexpected spikes.” A sudden jump in indexed pages or in Search activity can be a sign that pages were indexed or surfaced that nobody intended to expose.
Beyond the two reports, the guide now suggests tracking page creation and indexing across the whole platform. It recommends alerts for spikes in newly published public pages and for cases where a small number of users generate a large share of indexable URLs. Both patterns can point to abuse of a sharing feature before the problem becomes visible in Search Console indexing data.
What it means for site operators
Teams that run platforms, community features or tools that generate shareable pages are the most directly affected, including AI tools and app builders whose generated outputs end up on public URLs. Pages created for sharing, collaboration or utility are the category Google singles out, so those page templates are the first place to check for a default noindex rule.
The advice does not ask platforms to hide everything users create. Content that users publish to be found in Search can stay indexable, so the decision is made per page type rather than for the whole site. A private share link to a chat result or a web app built in one session is a different case from a public page someone wants others to discover.
Agencies and content teams that manage several sites can use the two Search Console reports named in the Google spam guide as a routine check on what is being indexed. The guide names these reports for spotting unexpected spikes, which makes them a practical starting point for watching changes. Based on the sources, a sensible order of checks is:
A practical order of checks follows from the sources. First, identify which page types exist for sharing rather than discovery. Second, confirm that Googlebot can crawl them if noindex is the chosen method. Third, watch the reports for spikes after any template change.
What is still not known
Neither the guide nor the changelog mentions any change in how Google will enforce these guidelines. It is not stated whether pages that remain indexable will face any consequence.
The sources do not say which platforms prompted the addition. They also do not say whether Google will publish further detail on what counts as a page made for sharing rather than for discovery in Search, or how platforms should classify borderline cases.
The guidance is a recommendation about defaults. No deadline for platforms is given in the material available.
Sources
Photo: Search Engine Journal
Frequently asked questions
Is there a new spam update from Google?
Is there a new spam update from Google?
The sources describe a guide update, not a ranking update. Google added a section to its user-generated spam guide on Oct. 26, and neither the guide nor the changelog mentions any change in enforcement.
What does Google recommend for pages made for sharing?
What does Google recommend for pages made for sharing?
Platforms should by default apply noindex or block crawling on pages designed for direct sharing, collaboration or utility rather than public search discovery.
Is a robots.txt block enough to keep a page out of Google?
Is a robots.txt block enough to keep a page out of Google?
Not by itself. Google’s documentation says robots.txt is not a mechanism for keeping a page out of Google, and a disallowed page can still be indexed if other sites link to it.
How can site owners watch for problems?
How can site owners watch for problems?
Google recommends verifying the site in Search Console to receive manual action and security notifications, and checking the Page Indexing and Search performance reports for unexpected spikes.
Does noindex work on a page blocked from crawling?
Does noindex work on a page blocked from crawling?
No. A noindex rule only works if Googlebot can crawl and read the page.