media
Insights · GEO
Robots, sitemaps and llms.txt.
Different files. Different jobs.
Useful pages come first. Three files can help systems discover or understand them: robots.txt sets crawl instructions, sitemap.xml lists preferred URLs, and llms.txt offers an optional summary. There is no universal rule that an AI engine reads all three before your homepage.
robots.txt: choose the purpose, then the policy
Search crawling, model training and a visit triggered by a user are different activities. A provider can use different agents for each. Choose what you want to allow, then check that the response served by your host reflects that choice.
| Purpose | Example agents | What to check |
|---|---|---|
| Search discovery | OAI-SearchBot, Claude-SearchBot, Googlebot | Search access and indexability, including page-level controls. |
| Training and other uses | GPTBot, ClaudeBot, Google-Extended | Each provider’s stated purpose. These are not interchangeable search controls. |
| User-triggered retrieval | ChatGPT-User, Claude-User | The provider’s rules for a visit requested by a user. |
For Google’s AI features in Search, Googlebot and the normal Search controls apply. Google-Extended governs other specified uses. Blocking a training agent is not the same as opting out of search. See OpenAI’s crawler documentation, Anthropic’s guidance and Google’s AI Search guidance.
Robots.txt is not authentication or a way to protect confidential information. Access control belongs on the resource itself. A crawl instruction also does not guarantee removal of a URL already known to a search engine.
sitemap.xml: help discovery without promising indexing
A sitemap lists the preferred public URLs you want a search engine to discover. Keep it aligned with your canonical URLs, remove obsolete entries and check that each listed page works. Do not list private receipts, staging pages or deliberately unindexed offers.
Use a last-modified date only when it reliably describes a significant content change. Rebuilding a site does not mean every article was updated. A sitemap helps discovery; it does not force indexing. Google’s sitemap guidance explains the distinction.
llms.txt: an optional summary
The llms.txt proposal provides a concise summary and links for language-model use. We maintain one as another readable account of our services and work. Its presence is not evidence that a particular engine uses it, and we do not attribute a ranking or citation benefit to it.
Google explicitly states that its AI Search features require no special AI text file or additional schema. A summary should support accurate page content, not substitute for it.
The page still has to be useful
State what you do, who it is for and what supports your claims. Make important information available as readable text. Link related services and evidence so a visitor or crawler can follow the same sensible path.
You can inspect our robots.txt, sitemap.xml and llms.txt, then read the full readiness checklist.
Want your discovery setup checked? Our audit reviews access, indexability, content and evidence, then records what was observed and what remains uncertain.
Explore AI-search readiness