robots.txt and AI bots: what to allow and what to block on your website
A practical guide to deciding which AI bots to let into your website with robots.txt: OAI-SearchBot, GPTBot, Google-Extended and Cloudflare, without losing visibility.
On Monday you check ChatGPT and, when you ask about your type of business in your city, three competitors show up and you don't. You open your robots.txt file and find a line someone added a year ago: “block all AI.” With good intentions, it closed the door on the very ones who could recommend you.
The solution is to tell which bot does what and decide one by one. Here is how.
Do all AI bots do the same thing?
No: some search to show your site in answers and others collect content to train models. According to OpenAI's documentation, OAI-SearchBot is for sites to appear in ChatGPT's search features, GPTBot crawls content that may be used to train its models, and ChatGPT-User acts when a person asks it to visit a page.
That difference is the basis of the decision: you may want to appear in answers (search) while not contributing content to training.
What happens if I block each bot?
Each block has a different effect, and only one removes you from search answers. This is what the official sources say:
| Bot or token | What it is for | If you block it |
|---|---|---|
| OAI-SearchBot | Search in ChatGPT | Your site is not shown in ChatGPT search answers |
| GPTBot | Training of OpenAI models | You signal that your content should not be used for training |
| ChatGPT-User | Actions requested by a person | OpenAI clarifies that robots.txt may not apply |
| Google-Extended | Control over Google's AI use (Gemini) | Does not affect your inclusion or your ranking in Google Search |
On the last row, Google explains it this way: Google-Extended is a control token, not a search crawler, and it is not used as a ranking signal.
Common mistake: pasting a generic “block AI” rule copied from a forum. If it includes OAI-SearchBot, you stop appearing in ChatGPT search without realizing it.
What does a balanced robots.txt look like?
A balanced robots.txt allows search bots and decides separately about training. An example for someone who wants to appear in answers but not train models:
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Deciding whether or not to block training is a business decision: if you live off your original content, you may want to protect it; if you want maximum visibility, you can allow it. What you should not do is mix both without knowing it. If you want to understand why appearing in those answers matters, read what GEO is.
How long does the change take to apply?
For ChatGPT search, it can take about 24 hours. OpenAI states this in its documentation: after you update robots.txt, its systems need that time to adjust. Publish the change, check that tudominio.com/robots.txt shows the new file and wait a day before concluding something failed.
If you have also just launched or moved your website, review what to do with Search Console and the sitemap so Google finds everything.
What do I do if I really need to block?
Use a security layer in front of your site, such as Cloudflare. robots.txt is a request that responsible crawlers respect, not a barrier. According to Cloudflare's documentation, in Security Settings you can configure AI bot policies by category (search, agent and training) and choose to block, block only on pages with ads, or allow.
Cloudflare also states that since September 15, 2026, new domains block training and agent bots by default on pages with ads, and allow search bots. Check your domain's settings and, if you don't know the tool yet, see what Cloudflare's free plan includes.
In short
Back to Monday: what removed your business from ChatGPT may have been a single inherited line. Separate search from training, write one rule per bot and check the result after a day. If you want us to review your robots.txt and your AI visibility strategy, see our SEO and GEO for AI service.
Frequently asked questions
If I block GPTBot, do I disappear from ChatGPT?
Not necessarily. According to OpenAI, blocking GPTBot signals that your content should not be used to train models. Appearing in ChatGPT search depends on OAI-SearchBot.
Does Google-Extended affect my Google ranking?
No. Google states that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal.
How long does a robots.txt change take to apply?
OpenAI states that, for search results, it can take about 24 hours after the robots.txt update.
Does robots.txt force bots to obey?
It is a signal that responsible crawlers respect. To actively block, there are tools such as Cloudflare's bot policies.
Do not stop at the theory: apply it to your business
We implement SEO and GEO with AI end to end, with your real operation. The first conversation is free.
