SEO & GEO

robots.txt, llms.txt and AI bots: the checklist to set up on your website

The technical settings that open your site to AI: robots.txt, llms.txt, Cloudflare, Search Console, Bing Webmaster Tools and IndexNow, point by point.

Loan Fikri

By Loan Fikri, founder of LFK Agency

Published on · Updated on · 9 min read

Your website can be perfect and stay invisible to AI if a single line of configuration blocks it. Here is the checklist we apply, on our own sites and on our clients’.

1. The robots.txt file

The robots.txt file, at the root of your site, tells crawlers what they may explore. By default, a crawler is allowed to explore anything the file does not forbid. An explicit allow rule only matters when a broader rule blocks it, for example a Disallow: / that applies to all crawlers. Here is an example that makes this choice explicit for the main AI crawlers:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: *
Allow: /

Sitemap: https://www.your-site.com/sitemap-index.xml

A few things to remember:

  • crawlers ending in -SearchBot power search inside the AI, crawlers ending in -User act when a user asks to read a page;
  • GPTBot and ClaudeBot also collect training data: you can allow them or not, depending on your policy;
  • Google-Extended and Applebot-Extended are not crawlers, but signals that control how Gemini and Apple use your content.

2. Your host or Cloudflare settings

robots.txt is not always enough. Some services block AI crawlers before they even read that file. On Cloudflare, check the policy applied to AI crawlers in the security settings, and set the categories you care about to Allow: search, agents, training.

It is a two-minute check, and it often explains why a site is never cited.

3. The llms.txt file

The llms.txt file is a proposal, not an official standard. It sits at the root of the site and summarises, in Markdown, what you do and which pages matter:

# Your company name

> One sentence on what you do and for whom.

## Services
- [Service 1](https://www.your-site.com/service-1): what it brings.

## Proof
- [Case studies](https://www.your-site.com/case-studies)

## Contact
- [Contact](https://www.your-site.com/contact)

Google Search does not use it, and no AI engine commits to reading it. It is a low-cost bonus, which also forces you to clarify the essentials.

4. Google Search Console as a Domain property

Add your site to Google Search Console and choose the Domain property: it covers every version of your site (with or without www, http or https). Verification is done with a TXT record in your DNS. Then submit your sitemap and request indexing of your key pages.

5. Bing Webmaster Tools

Create a Bing Webmaster Tools account and import your site from Search Console: the setup is copied in a few clicks. It matters for GEO: ChatGPT partly relies on third-party search providers, and Bing powers Copilot.

6. IndexNow

IndexNow lets you notify search engines as soon as a page is published or changed, instead of waiting for them to come by. It is used by Bing and a few other engines, but not by Google. To set it up:

  1. generate a key;
  2. publish a text file named after that key at the root of the site;
  3. send the list of changed pages to the IndexNow API, ideally automatically on every release.

7. Final checks

  • the sitemap only lists indexable pages, with exactly the same URLs as your canonical tags;
  • important pages return a 200 status code, not a redirect;
  • preview or test versions are not indexed;
  • structured data is valid and matches the visible content.

The checklist at a glance

  • robots.txt allows AI crawlers and points to the sitemap
  • your host or Cloudflare does not block AI crawlers
  • llms.txt published at the root, as a bonus
  • Search Console as a Domain property, sitemap submitted
  • Bing Webmaster Tools imported from Search Console
  • IndexNow set up, automated if possible
  • sitemap, canonicals and status codes consistent

These settings do not guarantee a citation. They guarantee that nothing stops one.

FAQ

Frequently asked questions

Does allowing GPTBot let OpenAI train its models on my site?

GPTBot is used, among other things, to collect training data. OAI-SearchBot is used for search in ChatGPT. You can allow one without the other. It depends on your policy: many businesses allow everything to maximise visibility.

Does Google-Extended block Google's AI answers?

No. According to Google, Google-Extended controls the use of your content for Gemini, but does not affect your presence in Google Search, including AI answers.

Does IndexNow work with Google?

No. IndexNow is used by Bing and a few other search engines. For Google, use your sitemap and Search Console.

Loan Fikri

The author

Loan Fikri

Founder of LFK Agency. He orchestrates the agency’s projects and coordinates a selected specialist for each craft. Insights guides are written from that work.

Insights

Read next

Let’s talk about your growth.

A free 30-minute audit. You leave with clear recommendations, even if you do not go further.