ROBOTS.TXT GUIDE
XML sitemaps for public sites
A sitemap is a discovery aid, not a guarantee of indexing. It is most useful when it is a clean, current list of canonical pages that return a successful response and are allowed to be crawled.
Include only canonical public URLs
List the final version of each indexable page, not redirects, duplicate parameter URLs, draft pages, or private locations. Every listed URL should be consistent with its canonical tag and internal links.
Keep it current as content changes
When you publish a guide, add it to the sitemap. When you redirect or remove a page, update the sitemap at the same time so crawlers receive a consistent inventory.
- Return a valid XML document at a stable URL.
- Reference it from robots.txt.
- Submit it in Google Search Console and Bing Webmaster Tools.
Pair sitemaps with internal links
Sitemaps complement rather than replace normal navigation. Link guide indexes to individual guides and let every article link back to the generator or other relevant resources.
Ready to generate your rules?
Use the generator to create a readable, downloadable robots.txt, then publish it at your domain root.
Create robots.txt