Your website can be perfect and stay invisible to AI if a single line of configuration blocks it. Here is the checklist we apply, on our own sites and on our clients’.
1. The robots.txt file
The robots.txt file, at the root of your site, tells crawlers what they may explore. By default, a crawler is allowed to explore anything the file does not forbid. An explicit allow rule only matters when a broader rule blocks it, for example a Disallow: / that applies to all crawlers. Here is an example that makes this choice explicit for the main AI crawlers:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
User-agent: Google-Extended
Allow: /
User-agent: Applebot-Extended
Allow: /
User-agent: *
Allow: /
Sitemap: https://www.your-site.com/sitemap-index.xml
A few things to remember:
- crawlers ending in -SearchBot power search inside the AI, crawlers ending in -User act when a user asks to read a page;
- GPTBot and ClaudeBot also collect training data: you can allow them or not, depending on your policy;
- Google-Extended and Applebot-Extended are not crawlers, but signals that control how Gemini and Apple use your content.
2. Your host or Cloudflare settings
robots.txt is not always enough. Some services block AI crawlers before they even read that file. On Cloudflare, check the policy applied to AI crawlers in the security settings, and set the categories you care about to Allow: search, agents, training.
It is a two-minute check, and it often explains why a site is never cited.
3. The llms.txt file
The llms.txt file is a proposal, not an official standard. It sits at the root of the site and summarises, in Markdown, what you do and which pages matter:
# Your company name
> One sentence on what you do and for whom.
## Services
- [Service 1](https://www.your-site.com/service-1): what it brings.
## Proof
- [Case studies](https://www.your-site.com/case-studies)
## Contact
- [Contact](https://www.your-site.com/contact)
Google Search does not use it, and no AI engine commits to reading it. It is a low-cost bonus, which also forces you to clarify the essentials.
4. Google Search Console as a Domain property
Add your site to Google Search Console and choose the Domain property: it covers every version of your site (with or without www, http or https). Verification is done with a TXT record in your DNS. Then submit your sitemap and request indexing of your key pages.
5. Bing Webmaster Tools
Create a Bing Webmaster Tools account and import your site from Search Console: the setup is copied in a few clicks. It matters for GEO: ChatGPT partly relies on third-party search providers, and Bing powers Copilot.
6. IndexNow
IndexNow lets you notify search engines as soon as a page is published or changed, instead of waiting for them to come by. It is used by Bing and a few other engines, but not by Google. To set it up:
- generate a key;
- publish a text file named after that key at the root of the site;
- send the list of changed pages to the IndexNow API, ideally automatically on every release.
7. Final checks
- the sitemap only lists indexable pages, with exactly the same URLs as your canonical tags;
- important pages return a 200 status code, not a redirect;
- preview or test versions are not indexed;
- structured data is valid and matches the visible content.
The checklist at a glance
- robots.txt allows AI crawlers and points to the sitemap
- your host or Cloudflare does not block AI crawlers
- llms.txt published at the root, as a bonus
- Search Console as a Domain property, sitemap submitted
- Bing Webmaster Tools imported from Search Console
- IndexNow set up, automated if possible
- sitemap, canonicals and status codes consistent
These settings do not guarantee a citation. They guarantee that nothing stops one.