The research behindgetting a small businessrecommended by AI
We change real pages and measure what happens next, from request-level logs on small and medium-sized sites.
Share of sites an assistant visited on someone’s behalf, each month
On 98% of the small business sites we follow, an assistant fetched a page in September 2026 because someone asked it a question. In February it was 87%, and it has not dropped below 96% since June.
OpenAI leads by a distance. Counting its two crawlers together, ChatGPT-User and OAI-SearchBot, they reached 96.5% of monitored sites in September 2026 and fetched 86 pages on the middle site that month. The table below splits the same month crawler by crawler, so no single bar there matches this figure.
block an AI crawler without any sign of it. Their robots.txt invites the assistant in and the server then refuses it, so the owner has no way to tell. 43% refuse at least one crawler, and only 5 owners wrote a rule themselves, and most of the written rules came from Cloudflare’s managed list.
Which assistants actually reach a small business site
September 2026 · share of sites reachedEach bar is the share of monitored sites that assistant’s crawler reached. Reach rather than volume, because a single busy site can account for most of the requests in a month, and a request count would then tell you the opposite of what is happening.
Google is low here because its assistants mostly do not fetch. AI Overviews and AI Mode answer from the Search index Googlebot has already taken, so there is usually no second request to log: Googlebot reaches almost every site in this fleet and makes roughly a thousand requests for every one made by a Google agent our classifier labels as AI. Google-Extended, which people look for in this table, is a consent switch over training and grounding rather than a crawler. So for Google the lever is what you allow Googlebot and Google-Extended to do, not whether a Google AI crawler turns up.
Which single change moves the assistants?
We make several changes to a page at once, so the useful question is which of them is doing the work. This section ranks them. On each site we take pages of the same type, compare the ones that got a given change against the ones that did not, and count the sites where the change comes out ahead on AI referral visits. We re-run it every month, because what earns a citation keeps moving.
The changes themselves are ordinary. A five-question question-and-answer section, written from the page’s own content and marked up so an assistant can lift it, is ahead on 79% of sites. A title tag rewritten to name what the page actually offers, in the words someone would ask in, is ahead on 77%. Rewriting dense paragraphs so each one leads with its answer instead of building up to it, 69%. A single main heading that names the subject rather than greeting the visitor, 65%.
And one change we do not make, because it does nothing: llms.txt, the file this field has recommended for two years. AI crawlers fetched it 31 times across 82 million requests in six months.
Studies
6 publishedChatGPT didn’t stop sending visitors. It stopped telling you it sent them.
From 9 October 2026, the typical small business website lost 62% of its ChatGPT-tagged visits (39% across all sites combined), while ChatGPT's own page reads dipped far less and other assistants did not fall.
ChatGPT broken links: why visitors land on 404s
Across 323 small business websites, 36 of 4,053 ChatGPT visits landed on a 404, about 1 in 113. Only 5 went to pages that never existed; the rest followed links to real pages, old or garbled.
Does AI search optimization actually work?
Across 108,834 pages on the same websites, optimized pages earned 173.5 AI referral visits per 1,000 pages in 90 days against 15.8 for untouched pages, and led on 89 of 102 sites.
The Silent Block
We fetched 314 sites as 38 AI crawlers. 43% refused at least one, and 112 refused at the server with nothing in robots.txt to show for it.
The State of AI Traffic for Small Businesses 2026
71.5% of tracked properties received at least one AI-referred session in a 30-day window. The median site saw six.
Does anything actually read llms.txt?
Across 82 million crawler requests, AI bots fetched llms.txt 31 times in six months.
What the same fixes did on one live website
We applied the changes in these studies to a customer site: schema, an answer-led rewrite of the page headings, and unblocking the AI crawlers its firewall had been turning away. Its AI mentions climbed from roughly 1,200 a month to a peak near 2,950.
One site. No control group. A trend across months rather than a step change, and the site published new pages over the same period. Read it as an illustration of the mechanism, not as an effect size.
The effect size is the study above: 102 sites, 108,834 pages.
Running experiments
3 in flightEveryone in this field reports correlations, because observing model answers is all their data allows. A plugin install is a dated change to a real website, which makes before and after possible.
Does structured business data bring the bots back sooner?
Diff-in-differences · matched control cohort · 90-day pre/post
Which pages do AI crawlers actually re-read, and how often?
Per-URL crawl frequency · top vs bottom decile · 14 attributes
How volatile is AI crawling, week to week?
Same balanced panel · dispersion, not level
Every figure here is a share, a median or an index.
Cohorts are fixed and windows matched period over period, so a number never rises just because the panel did.
Medians carry the trend wherever a single heavy crawler or busy site would dominate a mean, which on this data is most of the time.
Partial periods are excluded or labelled. Never silently included.
Nothing identifies a site, a page or a customer.
“AI crawlers” means search and on-demand fetchers. Training crawlers are excluded: one alone makes more requests than every AI search bot combined, and tells you nothing about visibility.
Checking AI search readiness
The readiness score is a 0-100 check of whether a page states its facts in a form an assistant can lift: structured business data, a single clear heading, answerable copy, machine-readable contact and service detail, and nothing in robots.txt or the server config turning the crawlers away. It measures whether you can be quoted, not whether you rank.
Last 30 days · 1,494 scans · median 62. A near-zero score, which is 5.2% of the window, can mean we could not read the site rather than that it reads badly, so the bottom band is partly measurement.