Sitewalk / Guides / Issue guides / AI crawlers and robots.txt
AI crawlers and robots.txt: what you block, and how to let ChatGPT search in
AI tools read your site with their own crawlers. Your robots.txt decides which ones may come in. Many sites block the wrong ones by accident, or block the right ones without knowing it.
Search bots and training bots are different
OpenAI runs three crawlers, each with its own robots.txt name, and each setting works alone:
- OAI-SearchBot finds pages to show in ChatGPT search. Block it and your site will not be shown in ChatGPT search answers.
- GPTBot collects content that may be used to train models. Block it and you ask OpenAI not to train on your content.
- ChatGPT-User opens a page when a person asks ChatGPT to. A person starts it, so OpenAI says robots.txt rules may not apply.
So you can allow search and block training. Other companies work the same way. PerplexityBot and Claude-SearchBot are search bots. ClaudeBot collects content that may be used for training, and Google-Extended is a token that controls whether Google may use your content to train its Gemini models.
Check your site in 2 minutes
Open https://yoursite.com/robots.txt. Look for lines that name these bots:
User-agent: OAI-SearchBot Disallow: /
A Disallow: / under a bot name closes the whole site to it. A Disallow with a path closes only that path. A User-agent: * group with Disallow: / closes the site to every bot that has no group of its own.
Then check two places that are easy to miss, because they are not in your page source:
curl -I https://yoursite.com/
Look for an x-robots-tag line that names a bot, or says noai. A plugin or a CDN setting often adds it.
In the page source, search for name="robots" and look for noai or noimageai.
The problems and the fixes
| Problem | What it means | Fix |
|---|---|---|
| Search bot blocked | OAI-SearchBot, PerplexityBot, Claude-SearchBot or ChatGPT-User is disallowed. The site cannot be cited or linked in those answers. | Remove the Disallow rule, or add an Allow rule for the bot. If the block is on purpose, leave it. |
| Training bot blocked | GPTBot, ClaudeBot or Google-Extended is disallowed. | Nothing to fix if it is on purpose. If you want your content used for training, remove the rule. |
| Partly blocked | A rule closes some paths, not the whole site. | Check the closed paths hold nothing you want cited. Narrow the rule if they do. |
| Blocked by a header | An x-robots-tag header names an AI search bot and says noindex, none or nosnippet. | Find where the header is set (server, CDN or plugin) and remove it. |
| Opt-out header or meta tag | noai or noimageai asks AI systems not to use your content. Not every system honours it. | Nothing to fix if on purpose. If not, remove it where it is set. |
| Crawler answers differently | Your site gives the search crawler an error, a redirect or far less content than a visitor gets. A firewall or bot-protection rule is usually the cause, and robots.txt says the bot is welcome. | Allow OAI-SearchBot in the firewall or CDN. OpenAI publishes its user-agent and address list. |
For the last row, test it:
curl -sI -A 'OAI-SearchBot/1.4' https://yoursite.com/
You want 200 and no redirect to another page.
What Sitewalk found on real sites
Of 54 audited sites (latest audit, since these checks went live on 4 Nov 2026):
- 0 blocked a search bot in robots.txt. None blocked it with a header either.
- 4 (7%) blocked at least one training bot.
- 3 (6%) had a rule that closes some paths to an AI bot.
- 0 had an opt-out header or tag, and 0 turned the search crawler away.
The fear that sites shut AI search out is not what these audits show. It is a small group of early users, so read it as a first look, not a survey.
Check it across your site
Sitewalk reads your robots.txt, your headers and how your site answers the search crawler, along with 200+ other technical checks. In ChatGPT, say:
Audit yoursite.com for technical SEO problems
Checked on 16 Nov 2026.