Robots.txt checker for AI crawlers

robots.txt rules. We read /robots.txt on your site and apply it to each crawler: which User-agent group applies, which Allow or Disallow line decides, and whether the page you tested is open. The parser follows Google’s documented behaviour: the most specific user agent wins, the longest matching path wins, * and $ wildcards work, and a tie goes to Allow.

What your server sends. We then request the page as each crawler and record the answer: served, refused (403, 401 and similar), rate limited, or a security challenge. A refusal here, while a normal browser gets the page, means a firewall, security plugin or hosting rule is blocking that crawler by name.

File health. A robots.txt that returns a server error, an HTML page instead of text, rules outside any User-agent group, relative Sitemap addresses, unsupported lines such as noindex, and files larger than the 500 KiB Google reads.

Two of them are rules, not crawlers. Google-Extended controls whether Google may use your content for Gemini training; Google crawls as Googlebot either way, and Google says this token does not affect inclusion or ranking in Search. Applebot-Extended does the same job for Apple’s AI training.

To block GPTBot only, add a group for it with Disallow: /. To check that you have not blocked something by accident, run the checker again after saving: it reads the live file, not a cached copy.

Allowed / Blocked is the robots.txt verdict for the path you tested. When a crawler is blocked, the deciding rule is shown under its name.

Served means your server returned the page to that crawler. Refused means it answered with an error such as 403 while a browser got the page: something on your server blocks that crawler by name. Connection cut is the same block done more bluntly: the server hangs up on that crawler at once, which is how many bot-blocker plugins and firewalls work. Security check means a bot challenge answered instead; see the limit above. Rate limited (429) means the server asked us to slow down, which says nothing about the real crawler.

Refused us means your site turned away a normal browser from our servers too, so the crawler rows can’t tell you anything. Many sites block data-center traffic this way while letting verified crawlers in.

The other common causes: a WordPress security plugin with a “block bad bots” list that includes AI crawlers, a host-level firewall, and a CDN rule that challenges unknown clients. Each shows up in the “Your server” column as Refused or Security check.

WordPress serves a virtual robots.txt when there is no file on the server, so there is nothing to find over FTP until you create one. SEO plugins such as Yoast SEO and Rank Math include a robots.txt editor under their tools settings, and a physical robots.txt file in your site root replaces the virtual one entirely.

After any change, run the checker again. Crawlers fetch robots.txt again on their own schedule; Google says it generally caches the file for up to 24 hours.

Crawler access is the first of many things that decide whether AI assistants recommend you. The free AI readiness check scores it alongside your content, schema and speed, and the sitemap checker tests the file your robots.txt should point to.