Everyone asks the models. We change the page and read the server logs.
A small business should turn up when someone needs it. We measure what actually gets one found, from request-level logs on real small and medium-sized sites.
Median user-agent fetches per site, per month
The median site went from 27 to 75 user-triggered fetches a month. Fitted growth is 15.6% a month at R² 0.809, which is a real trend rather than two lucky endpoints.
OpenAI leads by a distance: its crawlers reached 96.9% of monitored sites in August 2026, and on the middle site they fetched 132 pages that month. No other assistant comes close.
block an AI crawler without any sign of it. Their robots.txt invites the assistant in and the server then refuses it, so the owner has no way to tell. 43% refuse at least one crawler, and only 5 owners wrote a rule themselves, and most of the written rules came from Cloudflare’s managed list.
Which assistants actually reach a small business site
August 2026 · share of sites reachedEach bar is the share of monitored sites that assistant’s crawler reached. Reach rather than volume, because a single busy site can account for most of the requests in a month, and a request count would then tell you the opposite of what is happening.
Google is low here because its assistants mostly do not fetch. AI Overviews and AI Mode answer from the Search index Googlebot has already taken, so there is usually no second request to log: Googlebot reaches almost every site in this fleet and makes roughly a thousand requests for every one made by a Google agent our classifier labels as AI. Google-Extended, which people look for in this table, is a consent switch over training and grounding rather than a crawler. So for Google the lever is what you allow Googlebot and Google-Extended to do, not whether a Google AI crawler turns up.
Which single change moves the assistants?
We make several changes to a page at once, so the useful question is which of them is doing the work. This section ranks them. On each site we take pages of the same type, compare the ones that got a given change against the ones that did not, and count the sites where the change comes out ahead on AI referral visits. We re-run it every month, because what earns a citation keeps moving.
The changes themselves are ordinary. A five-question question-and-answer section, written from the page’s own content and marked up so an assistant can lift it, is ahead on 79% of sites. A title tag rewritten to name what the page actually offers, in the words someone would ask in, is ahead on 77%. Rewriting dense paragraphs so each one leads with its answer instead of building up to it, 69%. A single main heading that names the subject rather than greeting the visitor, 65%.
And one change we do not make, because it does nothing: llms.txt, the file this field has recommended for two years. AI crawlers fetched it 31 times across 82 million requests in six months.
Studies
4 publishedDoes AI search optimization actually work?
Across 108,834 pages on the same websites, optimized pages earned 173.5 AI referral visits per 1,000 pages in 90 days against 15.8 for untouched pages, and led on 89 of 102 sites.
The Silent Block
We fetched 314 sites as 38 AI crawlers. 43% refused at least one, and 112 refused at the server with nothing in robots.txt to show for it.
The State of AI Traffic for Small Businesses 2026
71.5% of tracked properties received at least one AI-referred session in a 30-day window. The median site saw six.
Does anything actually read llms.txt?
Across 82 million crawler requests, AI bots fetched llms.txt 31 times in six months.
Running experiments
3 in flightEveryone in this field reports correlations, because observing model answers is all their data allows. A plugin install is a dated change to a real website, which makes before and after possible.
Does structured business data bring the bots back sooner?
Diff-in-differences · matched control cohort · 90-day pre/post
Which pages do AI crawlers actually re-read, and how often?
Per-URL crawl frequency · top vs bottom decile · 14 attributes
How volatile is AI crawling, week to week?
Same balanced panel · dispersion, not level
Every figure here is a share, a median or an index.
Cohorts are fixed and windows matched period over period, so a number never rises just because the panel did.
Medians carry the trend wherever a single heavy crawler or busy site would dominate a mean, which on this data is most of the time.
Partial periods are excluded or labelled. Never silently included.
Nothing identifies a site, a page or a customer.
“AI crawlers” means search and on-demand fetchers. Training crawlers are excluded: one alone makes more requests than every AI search bot combined, and tells you nothing about visibility.
Checking AI search readiness
The readiness score is a 0-100 check of whether a page states its facts in a form an assistant can lift: structured business data, a single clear heading, answerable copy, machine-readable contact and service detail, and nothing in robots.txt or the server config turning the crawlers away. It measures whether you can be quoted, not whether you rank.
Last 30 days · 1,494 scans · median 62. A near-zero score, which is 5.2% of the window, can mean we could not read the site rather than that it reads badly, so the bottom band is partly measurement.