ARO · Operational
AI bot rules in robots.txt
AI bot rules in robots.txt is an Agent Readiness Optimization check in the Bot access group.
Version agent-readiness-2026-10
What is this benchmark?
AI bot rules in robots.txt
AI bot rules in robots.txt is an Agent Readiness Optimization check in the Bot access group.
Why it matters
Published crawler tokens such as ClaudeBot and GPTBot follow robots.txt.
How SiteGlue evaluates it
HTTP request to a well-known or published URL. Spec: https://developers.google.com/search/docs/crawling-indexing/robots/intro. Catalog agent-readiness-2026-10.
- Passing criteria
- The published signal for AI bot rules in robots.txt is present and readable.
- Failing criteria
- The scan requested the signal and it was missing, blocked, or malformed.
- Not verified
- The check was not selected, the probe did not run, or evidence was inconclusive. Not verified is never scored as 0.
- Not applicable
- Not applicable when the business has no corresponding surface (for example commerce protocols on a site with no cart).
Scoring weight: 1. Evidence: Public HTTP, HTML, or DNS evidence for AI bot rules in robots.txt. No private credentials.
Example results
Pass
Agents can read a valid AI bot rules in robots.txt signal on the public site.
Fail
AI bot rules in robots.txt is missing, blocked, or not machine-readable.
Recommended fixes
- The robots.txt block allows the published crawler tokens. Merge it into the existing file.
Related benchmarks
Frequently asked questions
Does a missing optional check lower my ARO score?
Yes, if the check is applicable and failed. Not verified items are excluded.
Is this a live ChatGPT test?
No. ARO is a website audit. Optional live walks are Test With AI Agents.
Run the ARO audit.