ARO · Operational
robots.txt
robots.txt is an Agent Readiness Optimization check in the Discoverability group.
Version agent-readiness-2026-10
What is this benchmark?
robots.txt
robots.txt is an Agent Readiness Optimization check in the Discoverability group.
Why it matters
Crawlers need a robots file to know which pages they may read.
How SiteGlue evaluates it
HTTP request to a well-known or published URL. Spec: https://developers.google.com/search/docs/crawling-indexing/robots/intro. Catalog agent-readiness-2026-10.
- Passing criteria
- The published signal for robots.txt is present and readable.
- Failing criteria
- The scan requested the signal and it was missing, blocked, or malformed.
- Not verified
- The check was not selected, the probe did not run, or evidence was inconclusive. Not verified is never scored as 0.
- Not applicable
- Not applicable when the business has no corresponding surface (for example commerce protocols on a site with no cart).
Scoring weight: 1. Evidence: Public HTTP, HTML, or DNS evidence for robots.txt. No private credentials.
Example results
Pass
Agents can read a valid robots.txt signal on the public site.
Fail
robots.txt is missing, blocked, or not machine-readable.
Recommended fixes
- Add this block to the existing robots.txt. Do not replace the file.
Related benchmarks
Frequently asked questions
Does a missing optional check lower my ARO score?
Yes, if the check is applicable and failed. Not verified items are excluded.
Is this a live ChatGPT test?
No. ARO is a website audit. Optional live walks are Test With AI Agents.
Run the ARO audit.