ARO · Operational

Content Signals in robots.txt

Content Signals in robots.txt is an Agent Readiness Optimization check in the Bot access group.

Version agent-readiness-2026-10

What is this benchmark?

Content Signals in robots.txt is an Agent Readiness Optimization check in the Bot access group.

Why it matters

A Content-Signal line tells crawlers how the page may be used.

How SiteGlue evaluates it

HTTP request to a well-known or published URL. Spec: https://contentsignals.org/. Catalog agent-readiness-2026-10.

Passing criteria: The published signal for Content Signals in robots.txt is present and readable.
Failing criteria: The scan requested the signal and it was missing, blocked, or malformed.
Not verified: The check was not selected, the probe did not run, or evidence was inconclusive. Not verified is never scored as 0.
Not applicable: Not applicable when the business has no corresponding surface (for example commerce protocols on a site with no cart).

Scoring weight: 1. Evidence: Public HTTP, HTML, or DNS evidence for Content Signals in robots.txt. No private credentials.

Example results

Pass

Agents can read a valid Content Signals in robots.txt signal on the public site.

Fail

Content Signals in robots.txt is missing, blocked, or not machine-readable.

Recommended fixes

The robots.txt block includes a Content-Signal line. Change the values before publishing if this business wants a different training or search preference.

Related benchmarks

Frequently asked questions

Does a missing optional check lower my ARO score?

Yes, if the check is applicable and failed. Not verified items are excluded.

No. ARO is a website audit. Optional live walks are Test With AI Agents.