Robots.txt vs noindex comes down to crawling versus indexing: robots.txt tells crawlers which URLs they may fetch, while noindex tells a search engine to exclude an accessible page from search results. Google must crawl the page to read its noindex directive, so blocking that page in robots.txt can defeat the exclusion. Use password protection when the content must be private. Choose robots.txt for crawl management, noindex for search exclusion and authentication for restricted access.
Robots.txt vs noindex: which should you use?
Use robots.txt to manage crawler requests, noindex to exclude an accessible page from search, and authentication to restrict access. These controls are not interchangeable. Start with two questions: may an anonymous visitor read the page, and should the page be eligible for search? Record both answers for every template. A public thank-you page can be readable but excluded from search; a confidential draft should require access permission. Review the intended behavior with the editor before implementing a sitewide default.
- Public article: allow access and check that no accidental exclusion remains.
- Public utility page: allow access but apply noindex if search exclusion is intended.
- Private preview: require authentication and test an anonymous request.
Can robots.txt keep a page out of Google?
Robots.txt is not a reliable way to exclude an HTML page from Google Search. Google explains in its robots.txt introduction that a blocked URL can still appear when discovered through links. The file also reveals the paths it lists to anyone who reads it. Treat robots.txt as crawl configuration rather than a private-directory registry. Before changing a rule, save the existing file and select examples that should match and examples that should remain unaffected.
- Test a path covered by the proposed rule.
- Test an ordinary article outside that path.
- Check the production hostname as well as the preview hostname.
How do you apply noindex correctly?
Apply noindex in the HTML metadata or an appropriate HTTP response header, and leave the resource accessible to the crawler that must process it. Google’s noindex guidance identifies crawl access as a prerequisite. Verify the actual response rather than trusting the settings screen. A theme can emit one instruction while a server header adds another. Our technical audit process keeps those layers visible in a single repair ticket.
- Inspect the HTML head for the deployed robots directive.
- Inspect response headers for X-Robots-Tag instructions.
- Check that robots.txt does not prevent the intended crawl.
Should a staging site use a password?
A staging site should require authentication when its content must stay private. An obscure hostname does not restrict access. Inventory what the preview contains before deciding whether public access with noindex is sufficient. Unpublished contracts, customer records and private enquiries need a separate access decision from unfinished design work. Use sanitized test records during development. Test direct media and export URLs too; protecting a homepage does not prove that every file shares its access controls.
- Open the preview in a signed-out browser session.
- Request a private file directly, without first visiting the homepage.
- Document who can invite reviewers and revoke their access.
What should you check before removing staging restrictions?
Before launch, check the hostname, access rules, robots directives and canonical URLs together. Copying a staging database can carry an intentional noindex setting into production. Conversely, opening the public site should not make a private preview public. Keep environment settings explicit and separate from article content. Our canonical URL guide helps distinguish a preferred address from an exclusion instruction. Check a category and a normal article, not just the homepage.
- Confirm the intended public domain serves the final content.
- Remove only the restrictions that belong to the public launch.
- Retest the staging hostname and its access policy independently.
How do you fix an accidental noindex?
Fix an accidental noindex at the layer generating it, clear the relevant cache and fetch the page again. Record the affected URLs before making changes. Compare the CMS visibility setting, SEO plugin output, theme and server headers so the repair stays narrow. Removing every exclusion globally can expose utility pages that were intentionally excluded. Add the repair to your measurement change log and separate deployment verification from later search-engine processing.
- Save the original response as evidence of the defect.
- Read back the corrected anonymous response after cache invalidation.
- Monitor subsequent search processing without promising a recovery date.
What questions arise when choosing robots.txt or noindex?
Does noindex make a page private?
Noindex does not make a page private. Visitors can still read an accessible URL; use authentication when access must be restricted.
Can a canonical tag replace noindex?
A canonical tag communicates a preferred version of duplicate content. It is not an access control or a substitute for an intentional indexing exclusion.
How quickly will Google process a changed rule?
Processing requires a subsequent crawl and is not confirmed by saving the setting. Keep the deployment date and check search tools for evidence of processing.
How do you document access and indexing rules?
Create a short template inventory with access and search-inclusion columns. Have the site maintainer verify one example per template and keep the results with the configuration, so the next launch does not depend on memory.
Explore Technical SEO for related guidance.
Written by
Theo Bennett
Theo Bennett is an editorial pen name used by Approve SEO for technical SEO and measurement coverage. The byline focuses on crawler verification, website maintenance, analytics interpretation, software contracts and the limits of benchmark data. Articles attributed to Theo draw on official documentation and other identified primary sources, with practical checks that readers can reproduce on their own sites. Examples are labelled when they are illustrative rather than measured results. Theo represents an editorial function, not an individual specialist. The portrait is an original AI-generated illustration. Source suggestions and corrections are handled through Approve SEO’s editorial contact route.