An llms.txt file is a plain Markdown file at the root of your domain that lists the pages you most want AI models to read. It takes about ten minutes to write and it sits next to your robots.txt.
It has also become the poster boy of AI visibility. Somewhere in the last two years llms.txt turned into the default answer to "how do I get cited by ChatGPT," it sits near the top of most AEO checklists, and no shortage of people will tell you it transformed their AI traffic. Search the term and you will find case studies, before-and-after screenshots and a great deal of confidence.
The reality we can measure looks nothing like that, and we are in an awkward position to say so, because generating llms.txt files is a feature we sell. We make them for 1,283 websites and we log every bot that visits them. If the file worked, we would be the first to tell you, and we would have put it on the pricing page.
Here is what is coming: what the file is and what one looks like, how to add one to WordPress in a few minutes, and then the part where 82 million logged crawler requests show what AI bots actually do with it.
What is an llms.txt file?
An llms.txt file is a plain Markdown file served at the root of your domain, at yoursite.com/llms.txt, that lists the pages you most want a language model to read.
Jeremy Howard proposed it in September 2024 through Answer.AI. The reasoning is sound. Language models work inside a limited context window, and a modern web page is mostly navigation, scripts, cookie banners and footer links wrapped around a small amount of actual content. Converting that into something a model can use is wasteful and error prone. So rather than make every model do that work, you publish a clean index yourself.
It is worth being precise about what llms.txt is not. It is not a permission file, so it cannot allow or block any crawler: that is robots.txt, and the two are often confused. It is not a sitemap, though the two overlap in purpose. And it is not a standard in the formal sense: there is no W3C or IETF working group behind it, and no search engine has committed to consuming it. It is a proposal that some publishers adopted.
What an llms.txt file looks like
The format is deliberately simple. An H1 with the site name, an optional blockquote summary, then H2 sections containing Markdown link lists, each link followed by an optional one-line description.
# Acme Plumbing
> Emergency and scheduled plumbing across Greater Leeds, established 1994.
## Services
- [Emergency callouts](https://acme.example/emergency): 24/7 response, typical arrival under 90 minutes.
- [Boiler servicing](https://acme.example/boilers): Annual servicing and Gas Safe certification.
## About
- [Service areas](https://acme.example/areas): Postcodes covered and callout charges.
## Optional
- [Company history](https://acme.example/history)
Two conventions matter. A section named Optional marks links a model can skip if it is short on context. And some sites also publish llms-full.txt, which inlines the entire content of those pages rather than linking to them, so a model needs a single request instead of many.
That is the whole format. It takes minutes to write by hand, which is part of why it spread so fast among developer tool companies.
How to add an llms.txt to WordPress
Three ways, fastest first.
Let a plugin generate it. Several WordPress SEO and AI plugins will build the file from your published pages and keep it in sync as you publish, including ours and All in One SEO. This is the option worth choosing, for one reason that has nothing to do with crawlers: a hand-written llms.txt goes stale within a month or two, and a file listing deleted URLs is worse than no file at all. See our comparison of WordPress GEO plugins for which ones generate it.
Upload a static file. Write the Markdown, save it as llms.txt, and upload it to your web root over SFTP, the same directory that holds robots.txt and wp-config.php. Nothing to configure. It will not update itself.
Serve it from your theme. Add a rewrite rule that maps /llms.txt to a template which prints the content. Only worth it if you want the file generated dynamically and you are not using a plugin that already does this.
Whichever route you take, three things have to be true. The file must sit at the domain root, so yoursite.com/llms.txt and not yoursite.com/blog/llms.txt, because nothing looking for it will check anywhere else. It must return a 200 status. And it must be served as plain text rather than wrapped in your theme's HTML.
Then check it, because this is where it usually goes wrong. Open yoursite.com/llms.txt in a private browser window. You should see plain text, not a styled page and not a 404. We spot-checked 25 customer sites whose dashboards said the file was published: two returned a 404 and three were blocked by their own security plugin. That is one in five sites publishing nothing at all while believing otherwise.
Chrome's Lighthouse will catch this for you too. Its agentic browsing audits flag a page when the server errors while retrieving your llms.txt, which is exactly the failure mode above. More on what that audit does and does not imply in the Google section below.
Do AI crawlers read llms.txt?
Almost never. This is where our data comes in.
Between December 2025 and 24 May 2026 our plugin logged 82 million crawler requests across roughly 1,600 WordPress sites. In that window, every path matching llms.txt was requested 192 times in total.
Not 192 times per site. 192 times across the entire customer base, over nearly six months.
Only 31 of those came from an AI crawler. The rest were search engines and SEO tools.
Who fetched an llms.txt, Dec 2025 to 24 May 2026
All 192 llms.txt fetches
- Not AI16184%Googlebot, AhrefsBot, Bingbot, YandexBot, Baiduspider, Applebot
- AI crawlers3116%every AI bot combined, across six months
Search engines and SEO tools account for 84% of all llms.txt traffic. The busiest single requester is Googlebot, which Google says does not use the file.
The 31 AI fetches, by company
- OpenAI1755%OAI-SearchBot 11, ChatGPT-User 5, GPTBot 1
- Anthropic516%ClaudeBot 5
- Amazon413%Amazonbot 4
- Other516%Meta-ExternalAgent 2, GoogleOther 2, CCBot 1
GPTBot, the crawler that feeds ChatGPT's training data, managed a single fetch in six months across roughly 1,600 sites.
Sit with the right-hand ring for a second. OpenAI's three crawlers account for 17 of the 31, and GPTBot, the one feeding the most widely used assistant on earth, fetched an llms.txt once in six months. ClaudeBot managed five, across four sites.
The single busiest requester of llms.txt files on our entire network is Googlebot, at 78 fetches, and Google has said publicly that it does not use them.
Thirty-one AI fetches over 176 days is roughly one every six days across the whole customer base. That is not adoption. That is noise.
How AI crawlers actually find your pages
This is the part that changed how we think about the file. While we were counting 31 llms.txt fetches, the same bots were hammering sitemap XML.
AI crawler requests: llms.txt vs sitemap XML
both bars, one scale
same five AI crawlers, May 2026 alone
all AI crawlers, six months (Dec 2025 to 24 May 2026)
5,721× more requests for sitemap XML than for /llms.txt. And the windows favour llms.txt: six months of it against one month of sitemaps.
That is the same comparison most articles get backwards, so it is worth stating plainly: AI crawlers are not indifferent to structured discovery files. They are extremely interested in one of them. It just is not llms.txt.
The pattern holds bot by bot, not only in aggregate.
Same bots, two files: where AI crawlers actually go
/llms.txt
six months, Dec 2025 to 24 May 2026
scale: 0 to 11
sitemap XML
May 2026 only
scale: 0 to 83,916
16,783× its llms.txt count
34,925× its llms.txt count
1,722× its llms.txt count
3,212× its llms.txt count
1,434× its llms.txt count
Each panel has its own scale, shown in its header. Bars are not comparable across panels; the numbers are.
Note the windows again: the llms.txt column covers six months and the sitemap column covers May 2026 alone. The comparison is deliberately rigged in llms.txt's favour and it still loses by three to four orders of magnitude.
ClaudeBot requested sitemap XML 83,916 times across 377 sites in one month. GPTBot, which could not be bothered to fetch llms.txt more than once all year, pulled sitemaps 3,212 times across 262 sites.
The lesson is not that llms.txt is bad. It is that AI crawlers already have a discovery format they trust, it has been standardised since 2005, and they use it constantly. If you want AI crawlers to find a page, put it in your sitemap and keep that sitemap complete and current. That one file does more for AI visibility than anything llms.txt was proposed to do.
One caveat on our figures, and it cuts in the honest direction. We only log requests that reach WordPress PHP. Our plugin serves llms.txt through a PHP rewrite so we see those, but of the 25 sites we sampled, four served llms.txt as a static file from a different plugin, which our logger would miss. So treat 192 as a floor rather than a precise count. Doubling it does not touch a conclusion built on a ratio of thousands to one.
Does Google use llms.txt?
No, and Google has been unusually direct about it.
Google's documentation on AI features, last updated in December 2025, states that you do not need to create new machine readable files, AI text files, or markup to appear in those features. Gary Illyes said at Search Central Live in July 2025 that Google does not support llms.txt and has no plans to. John Mueller has compared it to the keywords meta tag.
There is one wrinkle worth clearing up, because it comes up constantly and it tripped us up too. Some Google-owned properties do publish an llms.txt. We found one: angular.dev, which serves a proper file. That gets passed around as evidence that Google secretly endorses the standard.
Mueller was asked that question directly and answered it in two words. Asked whether Google's own llms.txt files amounted to an endorsement, he said, "to be direct, no."
The stronger version of the argument is not angular.dev, though. It is that Google ships tooling which checks the file: Chrome's Lighthouse includes an llms.txt audit under agentic browsing. Read what it actually does and the nuance survives intact. Lighthouse flags your page only if the server throws an error while retrieving the file, and if there is no llms.txt at all the audit returns Not Applicable, because Google documents providing one as optional. It is a server-health check on a file you may or may not have chosen to publish. It does not score the file's quality, it does not reward having one, and it lives under agentic browsing rather than anything to do with Search.
Which is consistent with everything else here. Google's interest in llms.txt is on behalf of AI agents driving a browser, not the search index, and that is the same audience we found in our own numbers.
Our logs agree with the documentation, in a way that is almost funny. Googlebot is the top requester of llms.txt files on our network at 78 fetches. Google says it does not use them. Both can be true: crawlers fetch plenty of files they do nothing with.
Which companies publish an llms.txt?
We checked 30 well-known domains by hand. The split is more interesting than the totals.
Does this domain serve an llms.txt? Checked by hand, August 2026
AI companies do not use llms.txt themselves
Companies that operate AI crawlers
The intended consumers of the file.
- docs.claude.compublishes llms.txt
- angular.devGoogle-ownedpublishes llms.txt
- google.comno llms.txt
- developers.google.comno llms.txt
- ai.google.devno llms.txt
- openai.comno llms.txt
- platform.openai.comno llms.txt
- anthropic.comno llms.txt
- perplexity.aino llms.txt
Developer platforms and SEO vendors
The people selling to developers.
- vercel.compublishes llms.txt
- nextjs.orgpublishes llms.txt
- docs.stripe.compublishes llms.txt
- cloudflare.compublishes llms.txt
- wordpress.orgpublishes llms.txt
- yoast.compublishes llms.txt
- semrush.compublishes llms.txt
- hubspot.compublishes llms.txt
- shopify.compublishes llms.txt
- ahrefs.comno llms.txt
- moz.comno llms.txt
Look at who is missing. OpenAI, Google and Perplexity operate the crawlers llms.txt was written for, and their main domains do not publish one. Anthropic's documentation site does, while anthropic.com does not.
The companies that do publish are developer platforms and SEO tool vendors: exactly the companies whose users paste documentation into coding assistants, plus the companies selling AI visibility advice. Both have reasons unrelated to whether search crawlers consume the file.
Who does read llms.txt: coding agents
Coding agents. That is the real answer, and it is not nothing.
Cursor, Cline, Continue and various MCP integrations can be pointed at an llms.txt to pull clean documentation into a coding session, which is exactly the use case Howard described. It explains the publisher list above perfectly: Stripe, Vercel, Next.js and Cloudflare all have developer audiences who feed docs to an assistant while writing code.
So llms.txt has a working audience. It is developer tooling, not search. If you sell an API or maintain technical documentation, an llms.txt has a genuine job and you should keep it current. If you run a plumbing company in Leeds, that audience does not apply to you.
While we are on where AI attention actually goes, our referral data is blunt. Of the AI referrals we can identify a source for, ChatGPT accounts for 43,526 visits. Every other named assistant combined, Claude, Perplexity and Gemini, accounts for 7. Our LLM SEO guide breaks down that funnel across 1,100 sites.
llms.txt best practices for WordPress
Keep it. Expect nothing from it. Spend the time on your sitemap instead.
The cost of publishing llms.txt is a handful of kilobytes and one automated build step. The cost of not having one, if OpenAI or Google announces tomorrow that they consume it, is that you are late. When something costs approximately nothing and has a small chance of mattering, you buy it and stop thinking about it. What you should not do is hand-craft one, pay a retainer to maintain it, or believe a dashboard reporting AI crawler hits on it.
What matters more, in the order our data supports:
Make your sitemap complete and current. This is the file AI crawlers demonstrably pull tens of thousands of times a month. Every page you want cited belongs in it, it should be split sensibly rather than one enormous file, and dead URLs should leave it promptly.
Keep pages readable, because crawlers fetch HTML in volume. They are reading your pages, not a summary of your pages. Clean markup, real answers near the top, and working structured data all act on what crawlers actually consume.
Check your robots.txt. It is the file with real enforcement behind it, and a stray Disallow aimed at GPTBot or ClaudeBot will cost you far more than a missing llms.txt ever could. We find misconfigured robots.txt rules on customer sites regularly. We have never once found a site harmed by lacking an llms.txt.
Confirm your llms.txt actually loads, if you keep one. One in five of the sites we spot-checked were serving a 404 or a block page.
Where llms.txt sits in our own product
Since this article argues against the value of something we ship, it is worth being specific about how we ship it.
Every site on our plugin gets an llms.txt, free tier included. It is generated from your published pages and rebuilt automatically as you publish, which is the only part that genuinely matters, because a hand-written file goes stale and a stale file is worse than none. You can shape what goes in it: which sections appear, how many pages each one lists, a minimum importance threshold, your own site name and description, and a custom prompt controlling how the one-line descriptions get written. There are 20,034 of these files because it is automatic, not because 20,034 people sat down to write one.
Our own audit rates it Low impact. It is the only criterion in the whole report carrying that rating, and it is deliberately left out of the 0 to 100 readiness score, so publishing an llms.txt cannot move your number and lacking one cannot dent it. We check it, we report it, we fix it for you, and we do not let it flatter the score.
That is the position the data supports. Google documents the file as optional, and its own Lighthouse audit only complains when your server errors while serving one, so there is no penalty for having it. It costs a few kilobytes and one build step. Adoption could still change, and if it does you are already compliant. So we generate it, keep it current, and rate it low, rather than pretending a file that AI crawlers fetched 31 times in six months is a growth lever.
If a vendor tells you llms.txt is the first thing to fix for AI visibility, ask them what their own numbers say. This is ours.
How we know: the zero we nearly published
This section is for anyone who wants to run these numbers themselves. If you came for the verdict, you already have it.
Our first pass returned zero. Not a small number. Zero llms.txt fetches, and also zero robots.txt and zero sitemap.xml fetches, across thirteen months and 165 million logged bot visits.
Zero robots.txt is impossible. Every crawler on the internet fetches robots.txt constantly. So the number was not a finding, it was a filter in our own pipeline: an ingestion blacklist, live since 25 May 2026, strips any path containing llms.txt, robots.txt or sitemap, plus anything ending .txt or .xml, and a second filter removes the same paths from our aggregate tables. The blacklist exists for a good reason, which is keeping WordPress internals and vulnerability-scanner probes out of customer page reports. llms.txt and sitemaps were collateral damage.
The real data was in the archive of raw logs from before that filter shipped. That is where the 82 million rows and every number above came from.
Two things follow. A zero in a dashboard is not evidence of absence, and from the inside there is no way to tell "nothing happened" from "we are not recording it." More specifically: if you have read a post claiming AI bots hit an llms.txt some number of times, ask what produced the number, because a great many analytics setups cannot see file requests at all.
Does being listed in llms.txt help a page?
The fetch counts answer whether crawlers read the file. They do not answer the subtler question: do pages named inside an llms.txt get treated better?
Our generated files list a subset of each site's pages, chosen by importance score, so we can compare listed against unlisted pages on the same site. Same-site comparison removes every difference between one business and another: domain authority, publishing volume, industry, budget.
Two corrections were essential. We dropped every homepage, because homepages appear in every llms.txt and are always the most crawled page on any site, so leaving them in would manufacture a large fake effect. And we averaged within each site before taking a median across sites, because pooling lets three enormous sites speak for everyone.
Then the part that makes the result interpretable. We ran the identical comparison for Googlebot, a near-perfect control given that Google states Search does not use llms.txt. If listed pages attract more crawling simply because they are a site's important pages, Googlebot will show that pattern too.
Crawl rate on llms.txt-listed pages vs unlisted pages, same site
Pages listed in llms.txt get 1.13 times more AI search crawling than unlisted pages, and 1.39 times more Googlebot crawling. The crawler that cannot read the file shows a stronger preference for the listed pages than the crawlers that supposedly can.
On-demand agent traffic comes in slightly above Googlebot at 1.49x, and that gap deserves care rather than dismissal. It sits on the smallest sample of the three, and the spread is enormous: a quarter of sites show listed pages doing worse than unlisted ones. A tighter 61-site cut put AI crawling at 1.41x against 1.58x for search engines, flipping the ranking again. Numbers that reorder themselves when you change the sample are noise.
So listed pages do get crawled more, and it is because they are the good pages. Being listed adds nothing detectable on top.
The number we could have published instead
There is a much more exciting statistic in the same database, and it is worth showing because you will meet its cousins in vendor marketing.
Compare sites that have an llms.txt against sites that do not. Sites with one get a median of 369 AI crawler visits. Sites without get 48. That is 7.7 times more AI crawling for sites with llms.txt, it is technically accurate, and it would make a great chart.
One dataset, two study designs, two answers
Site vs site
7.7×
Confounded
Compares sites that have an llms.txt against sites that do not. The two groups differ in almost every other way too, so the file gets credit for the site owner's overall effort.
Within the same site
1.13×
Holds up
Compares listed against unlisted pages on one site, with Googlebot as a control. Googlebot scores 1.39× on the same pages, so even this gap is not about llms.txt.
It is also close to meaningless. Sites without an llms.txt are overwhelmingly sites that never finished onboarding. They are smaller, newer, less maintained and less invested in AI visibility across the board. The file is a symptom of an engaged site owner, not the cause of the crawling.
Any time you meet a headline of the form "sites with X get N times more AI traffic," this is the shape to suspect. The fix is not a bigger sample, because a million sites measured this way returns the same wrong answer with tighter error bars. The fix is comparing within sites, against a control that cannot respond to the thing you are testing.
The verdict
llms.txt was a good idea the market declined to adopt. The spec is sound, the problem it targets is real, and the companies with the power to make it matter have spent nearly two years not doing so.
Our numbers make that about as clean as measurement gets. Thirty-one AI crawler fetches in six months across roughly 1,600 sites, against 177,353 sitemap requests from the same bots in a single month. Pages inside llms.txt files get no more AI attention than a crawler blind to those files gives the very same pages. We generate 20,034 of these files and that is still the answer.
So: publish one, automate it so it never goes stale, spend ten minutes confirming it actually loads, and then go make your sitemap excellent.
If you want to know which AI crawlers are actually reaching your own pages, that needs server-side logging rather than an external scan: a crawler only shows up in your logs, never in a page audit. Our WordPress plugin records those hits, including on the free tier, which is where every number in this article came from.
