Sitewalk / Guides / Issue guides / AI crawlers and robots.txt

AI crawlers and robots.txt: what you block, and how to let ChatGPT search in

AI tools read your site with their own crawlers. Your robots.txt decides which ones may come in. Many sites block the wrong ones by accident, or block the right ones without knowing it.

Search bots and training bots are different

OpenAI runs three crawlers, each with its own robots.txt name, and each setting works alone:

So you can allow search and block training. Other companies work the same way. PerplexityBot and Claude-SearchBot are search bots. ClaudeBot collects content that may be used for training, and Google-Extended is a token that controls whether Google may use your content to train its Gemini models.

Check your site in 2 minutes

Open https://yoursite.com/robots.txt. Look for lines that name these bots:

User-agent: OAI-SearchBot
Disallow: /

A Disallow: / under a bot name closes the whole site to it. A Disallow with a path closes only that path. A User-agent: * group with Disallow: / closes the site to every bot that has no group of its own.

Then check two places that are easy to miss, because they are not in your page source:

curl -I https://yoursite.com/

Look for an x-robots-tag line that names a bot, or says noai. A plugin or a CDN setting often adds it.

In the page source, search for name="robots" and look for noai or noimageai.

The problems and the fixes

ProblemWhat it meansFix
Search bot blockedOAI-SearchBot, PerplexityBot, Claude-SearchBot or ChatGPT-User is disallowed. The site cannot be cited or linked in those answers.Remove the Disallow rule, or add an Allow rule for the bot. If the block is on purpose, leave it.
Training bot blockedGPTBot, ClaudeBot or Google-Extended is disallowed.Nothing to fix if it is on purpose. If you want your content used for training, remove the rule.
Partly blockedA rule closes some paths, not the whole site.Check the closed paths hold nothing you want cited. Narrow the rule if they do.
Blocked by a headerAn x-robots-tag header names an AI search bot and says noindex, none or nosnippet.Find where the header is set (server, CDN or plugin) and remove it.
Opt-out header or meta tagnoai or noimageai asks AI systems not to use your content. Not every system honours it.Nothing to fix if on purpose. If not, remove it where it is set.
Crawler answers differentlyYour site gives the search crawler an error, a redirect or far less content than a visitor gets. A firewall or bot-protection rule is usually the cause, and robots.txt says the bot is welcome.Allow OAI-SearchBot in the firewall or CDN. OpenAI publishes its user-agent and address list.

For the last row, test it:

curl -sI -A 'OAI-SearchBot/1.4' https://yoursite.com/

You want 200 and no redirect to another page.

What Sitewalk found on real sites

Of 54 audited sites (latest audit, since these checks went live on 4 Nov 2026):

The fear that sites shut AI search out is not what these audits show. It is a small group of early users, so read it as a first look, not a survey.

Check it across your site

Sitewalk reads your robots.txt, your headers and how your site answers the search crawler, along with 200+ other technical checks. In ChatGPT, say:

Audit yoursite.com for technical SEO problems

Get started

Checked on 16 Nov 2026.

More guides

All issue guides ยท All guides