ROBOTS.TXT GUIDE

Google-Extended and Google Search are separate controls

Google-Extended is a product token used for specific Google AI uses. It is separate from Googlebot, which is associated with Google Search crawling, so the names should not be treated as interchangeable.

Choose the token that matches your intent

Use a Google-Extended rule only when that is the AI-related preference you mean to express. Blocking Googlebot is a different decision and can affect discovery and indexing in Google Search.

Avoid broad Google blocks

A rule written for a broad Google user agent can have consequences beyond AI training or grounding preferences. Keep crawler groups explicit, readable, and limited to the behavior you intend to change.

Name the crawler exactly as documented.
Keep Googlebot and Bingbot out of AI-specific blocks.
Recheck crawler documentation before a policy change.

Validate the published result

After publishing, fetch the public robots.txt URL and make sure the intended groups are visible. Then use search-engine tools to verify that important public pages remain crawlable.

Ready to generate your rules?

Use the generator to create a readable, downloadable robots.txt, then publish it at your domain root.

Create robots.txt