Can the AI even read your website?

Before any question about content, tone or structure comes a duller one: when ChatGPT opens your page to answer a customer, what does it actually get back? Two log studies from 2026 answer that, and the answer is less than most people assume.

Most advice on AI visibility starts at the writing desk. Answer the question early, add a number, keep the paragraph short. All of that is fine, and none of it matters if the model never receives the text. So it is worth knowing how the fetch works, because it is not a browser and it does not behave like one.

Most of the traffic is a live fetch, not a crawl

Duda hosts small-business websites, which makes its February 2026 numbers unusually relevant here. Of 858,457 sites, 59% received at least one AI crawler visit that month, 68.9 million visits in total. The majority, 56.9%, were live fetches triggered by someone asking a question; training crawls were 28.8% and index-building 14.3%. OpenAI accounted for 81% of everything (Search Engine Journal).

That shifts the question. The bot that decides whether your firm gets named is not the patient crawler that visits once a week. It is the one that opens your page while a person is waiting for a reply. And that bot is in a hurry.

What the live fetcher does with your page

In January 2026 the technical-SEO firm EdgeComet pointed ChatGPT at a fresh test domain and read the server logs. The user-triggered bot, ChatGPT-User, requested the HTML document and nothing else. No JavaScript files, no CSS, no images. Content that only appears after a script runs was never seen. Data placed exclusively in JSON-LD schema was ignored. And when a page took longer than five seconds to respond, the bot closed the connection and instead ran a Google search for the site to find a cached version of the text (EdgeComet).

The same lab ran the test against Gemini. The live fetcher there used a bare "Google" user agent rather than Googlebot, fetched raw HTML only, ignored schema, and gave up after four seconds. When the stock-keeping number of a test product sat only in the schema block, Gemini invented one (EdgeComet). The lesson is not about Gemini. It is that a fetcher with a four-second budget will answer anyway, with or without your page.

This is consistent with the larger dataset. Vercel and MERJ examined roughly 1.3 billion requests from AI crawlers across their network in late 2024 and found that none of the major ones, from OpenAI, Anthropic, Perplexity, Meta or ByteDance, executed JavaScript. They did download script files in some cases, ChatGPT in 11.5% of requests, Claude in 23.8%, but never ran them. Only Google's and Apple's crawlers rendered pages (Vercel). Eighteen months later, the vendor documentation still says the same.

Where the fetch goes wrong on a small site

A five-person firm rarely runs a React application. The failure modes are more mundane. Reviews pulled in by a widget after page load. Opening hours in an embedded map. Prices in a PDF or an image. A cookie wall that serves the banner instead of the page. A cheap shared host that takes six seconds to answer at lunchtime. Each of these is invisible to a bot that reads the first response and moves on.

Then there is the address book problem. In the Vercel data, ChatGPT's crawler spent 34.8% of its fetches on pages that returned 404, and another 14.4% following redirects; Claude's crawler hit 404s in 34.2% of fetches (Vercel). Old URLs from a relaunch, a sitemap nobody regenerated, a page that moved without a redirect: the bot keeps asking for what it once found, and a third of the time it gets nothing.

One more thing to check if your site sits behind Cloudflare. From 15 September 2026, new domains onboarding there will have "Training" and "Agent" crawlers blocked by default on pages that display ads, while "Search" crawlers stay allowed (Cloudflare). Most small-business sites carry no ads and are unaffected. If yours does, the live fetcher described above falls under "Agent", and the default now works against you.

The check takes two minutes

Open your most important page, the one that answers the question a customer would ask. Press Ctrl-U or Cmd-U to see the source the server sent. Search that text for your headline, for your opening hours, for a price. If they are there, the fetcher sees them. If the source is a short shell with a script tag and none of your words, the fetcher sees the shell.

Then time it. A page that takes longer than four seconds to start responding is out of Gemini's window and close to ChatGPT's. Shared hosting, uncached image galleries and stacked plugins are the usual culprits, and all three are fixable without a developer.

Finally, ask your host for the 404 log, or look at the "Not found" report in Search Console. Every entry there is a URL some crawler still believes in. Redirect the ones that had a successor, and regenerate the sitemap so the list stops growing.

None of this will get you cited on its own. It only removes the reason you might not be. But given that more than half of AI bot visits to sites like yours are now a person waiting for an answer, it is the cheapest afternoon in the whole method.