What does the file control?
A website can use robots.txt to manage crawler access to particular paths. The rules identify the crawler group they apply to and which paths are allowed or disallowed.
Google explains the uses and limitations in its robots.txt introduction. Not all crawlers follow the same rules, so the file cannot enforce access control against every visitor or automated tool.
For confidential material, use appropriate authentication and permissions. Publishing a path in robots.txt does not make that path secret.
Blocking crawling is not the same as noindex
A URL blocked from crawling can still be known through links elsewhere. Blocking the request does not necessarily remove the address from search results.
A noindex instruction addresses indexing instead. For Google to see a noindex instruction on a page, it must be able to crawl that page. Blocking the page first can prevent it from seeing the instruction.
Decide which problem you are solving before changing settings: unwanted crawling, unwanted search inclusion or unauthorised access are different problems.
A common launch mistake
Imagine a new catalogue developed on a test website. A broad crawl restriction was added during development. If that restriction is copied to the public website without review, intended product pages may remain blocked from crawling.
The pages can still look normal when the owner opens them. That is why a visual check alone does not verify crawler access. Include the live robots.txt file in the launch review.
Review a rule before applying it
- Identify the specific URLs involved.
- Write down why crawling should be restricted.
- Check which crawler groups the rule affects.
- Test representative allowed and blocked URLs.
- Keep a copy of the previous file and record the change.
Ask your developer to explain broad patterns before using them. A rule aimed at one folder can have a much larger effect if the path is wrong. Also consider resources needed to display public pages properly.
Check the live result
Confirm the expected file is served by the correct public hostname. Use Search Console information for affected pages to investigate reported crawl restrictions, then compare that evidence with the intended rule.
Do not keep editing several controls at once without a record. You may fix one problem while introducing another, making the final result harder to explain.
Read crawling vs indexing before diagnosing missing pages, and the sitemap guide for a separate discovery mechanism.