ROBOTS.TXT GUIDE
Robots.txt vs noindex: choose the right control
These controls are often grouped together because all influence visibility, but they work at different layers. Selecting the wrong one can leave a page discoverable or prevent a search engine from seeing the instruction you intended.
Use robots.txt for crawl preferences
Robots.txt tells cooperative crawlers where you prefer they do not crawl. It is useful for crawler-specific policies and for avoiding unnecessary crawling, but it is not a privacy control.
Use noindex for search-result removal
Use noindex on a page that search engines can crawl when you want it excluded from search results. Do not block that same page in robots.txt before a crawler can read the noindex instruction.
Use authentication for private information
If a document must stay private, require access before serving it. Treat robots.txt and noindex as search controls, not as a replacement for permissions or login protection.
Ready to generate your rules?
Use the generator to create a readable, downloadable robots.txt, then publish it at your domain root.
Create robots.txt