Two pieces of advice circulate about AI crawlers: block them to protect your content, or allow everything to be visible. Both treat "the AI bot" as one thing. It is not, and the difference is the whole story.
When a publisher worries that OpenAI is training on its pages, the standard move is a line in robots.txt that disallows GPTBot. When a consultant worries about being invisible in ChatGPT, the standard move is the opposite. Both people are talking about the same company and, as it turns out, about different software.
OpenAI documents three separate user agents. GPTBot is used, in OpenAI's words, to make its foundation models "more useful and safe"; disallow it and your content should not be used for training. OAI-SearchBot is used "to surface websites in search results in ChatGPT's search features"; opt out of that one and a site "will not be shown in ChatGPT search answers." The third, ChatGPT-User, handles pages a person asks ChatGPT to open. OpenAI states plainly that "each setting is independent of the others," and that a change can take about 24 hours to take effect (OpenAI).
So the headline question has a clean answer. Blocking GPTBot does not remove you from ChatGPT's search answers. Blocking OAI-SearchBot does. Perplexity splits things the same way: PerplexityBot exists to surface and link sites in Perplexity's results and is not used to train models, and the company recommends allowing it (Perplexity).
That gap is the part worth a small firm's attention. Ahrefs looked at roughly 140 million websites and counted two things separately: sites that name a bot and disallow it, and sites where a general rule turns away every bot. For GPTBot the explicit rate was 0.50% against 5.89% overall. For OAI-SearchBot it was 0.08% against 5.54% (Ahrefs). The study does not say why those sites have blanket rules, and its data ends in December 2024, so treat the exact figures as a floor from an earlier period, not a current census. What it shows is that a few percent of the web is closed to AI search crawlers without anyone having decided to close it for that purpose.
The larger and more recent picture is deliberate. A June 2026 analysis of 7,870 top sites found 17.6% block GPTBot but only 6.6% block OAI-SearchBot; its author summarises that sites are nearly three times more willing to be cited than to be trained on (Humphrey). Large publishers have made this choice on purpose: keep the training door shut, keep the citation door open.
Cloudflare, which sees a large share of web traffic, reports that training now drives nearly 80% of AI bot activity. It also measured how little comes back: in July 2025 Anthropic's crawlers made about 38,065 requests for every referral they sent, and OpenAI's about 1,091 (Cloudflare). That asymmetry is why blocking training bots feels reasonable. It says nothing about whether blocking them helps you get named. The search bot is the one tied to answers.
A caveat in the other direction. ChatGPT-User and Perplexity-User act on behalf of a person who asked for a page, and both companies say those requests may not follow robots.txt; Perplexity's documentation says Perplexity-User "generally ignores" the file (Perplexity). Cloudflare also documented Perplexity fetching content with a generic Chrome user agent after its declared crawler was blocked, at an estimated 3 to 6 million requests a day, while its test found that ChatGPT stopped when blocked (Cloudflare). So robots.txt is a polite request that the major search bots mostly honour, not a lock. If your content is truly private, put it behind a login.
Open yourdomain.com/robots.txt in a browser. You are looking for one thing: a rule of the form
User-agent: * followed by Disallow: /, or a line that names OAI-SearchBot or
PerplexityBot. If you find either, ask whoever built the site whether it was intended. Staging settings
left on after launch and security plugins are the usual suspects, though we cannot put a number on how
often that happens to small sites. If you want to keep your pages out of training sets, disallow GPTBot
and leave OAI-SearchBot alone. Then wait about a day for the change to take effect.
This is a two-minute check, and it sits upstream of everything else in this series. A page that is well written, current and carefully structured earns nothing if the crawler that feeds the answer was told to stay out.
We check your robots.txt and your pages, then ask ChatGPT, Perplexity, Google AI and Claude the questions your customers ask, and send you the score. No account, no card.
run_check →