robots.txt (RFC 9309) rule reading with llms.txt and AI-use signals

$0.005 USDC per call · Developer and security · POST /api/v1/robots-check · Id robots-check

Reads a supplied robots.txt the RFC 9309 way for an agent deciding whether to fetch up to 25 paths as one product-token user agent: group selection (merged groups, "*" fallback), longest-match with the Allow-wins-tie rule, "*" and "$" patterns, percent-encoding normalization, fetch-status handling (4xx = no rules, 5xx = disallow all), the rule and line that decided each path. It also lints a supplied llms.txt (llmstxt.org format) and lists supplied X-Robots-Tag / meta robots / noai / TDM-reservation signals. This is a reading of published rules, not permission: it does NOT fetch the file, interpret terms of service or licences, follow redirects, honour Crawl-delay, or prove the file you supplied is the live one.

What you get

Stops an agent from guessing at robots.txt precedence; every verdict names the deciding line.

Call it

Send the JSON body below to https://agents.writerslogic.com/api/v1/robots-check. An unpaid request answers 402 with the price and how to pay; a request that fails with a 5xx status is not charged.

curl -i -X POST https://agents.writerslogic.com/api/v1/robots-check \
  -H 'content-type: application/json' \
  -d '{"userAgent":"foobot","paths":["/example/page.html","/example/other.html","/index.html"],"robotsTxt":"User-Agent: *\nDisallow: *.gif$\nDisallow: /example/\nAllow: /publications/\n\nUser-Agent: foobot\nDisallow:/\nAllow:/example/page.html\nAllow:/example/allowed.gif\n"}'

Input

Field	Type	Required	Description
`userAgent`	string; length 0..64	yes	Product token only (letters, underscore, hyphen), e.g. "Googlebot"; no version.
`paths`	array<string; length 0..2048>; max 25 items	yes	Path plus optional query beginning with /, or a full http(s) URL (path and query are used).
`robotsTxt`	string; length 0..512000	no	robots.txt body, at most 512,000 UTF-8 bytes (500 KiB).
`fetchStatus`	integer; 100..599	no	HTTP status the robots.txt fetch ended with (after at most 5 redirects). Omit when robotsTxt is supplied and was a 2xx.
`purpose`	string; one of "crawl", "ai-training"	no	ai-training also treats noai, noimageai and tdm-reservation: 1 as restrictions. Default crawl.
`llmsTxt`	string; length 0..200000	no
`headers`	array<object>; max 50 items	no	Response headers of the page: X-Robots-Tag, tdm-reservation, tdm-policy.
`metaTags`	array<object>; max 50 items	no	HTML meta elements: robots, googlebot, tdm-reservation, tdm-policy.

Example request body

{
  "userAgent": "foobot",
  "paths": [
    "/example/page.html",
    "/example/other.html",
    "/index.html"
  ],
  "robotsTxt": "User-Agent: *\nDisallow: *.gif$\nDisallow: /example/\nAllow: /publications/\n\nUser-Agent: foobot\nDisallow:/\nAllow:/example/page.html\nAllow:/example/allowed.gif\n"
}

Result

The response is { "data": ..., "_provenance": ... }. data carries these fields; _provenance is a signed W3C Verifiable Credential with a C2PA-style manifest, checked free at POST /api/v1/verify-provenance.

Field	Type	Required	Description
`status`	string; one of "pass", "fail", "unknown", "unsupported"	no	pass: every rule held. fail: a rule was violated (see witnesses). unknown: the input lacks the evidence to decide. unsupported: outside what this engine covers. Missing evidence is never a pass.
`fileUsed`	string; one of "parsed", "unavailable-allow-all", "unreachable-disallow-all", "none"	no
`matchedGroup`	string; one of "agent", "*", "none"	no
`decisions`	array<object>	no
`sitemaps`	array<string>	no
`ignoredRulesBeforeGroup`	integer	no
`llms`	any	no	llms.txt lint: title, summary, sections, link count and problems; null when not supplied.
`signals`	array<object>	no
`tdm`	object	no
`note`	string	no
`witnesses`	array<object>	no
`versions`	object	no	Every semantic dependency of this answer (engine, tables and their as-of dates, tokenizer, calendar, rounding profile).
`limits`	array<string>	no
`deterministic`	boolean	no	Always true: pure code, no model.

Pay and connect

Available through: x402 (USDC on Base), MCP. MCP clients reach it through the infra-preflight-kit package.

Machine-readable

This agent as JSON (schemas, example, OpenAPI operation)
Free signed sample response
OpenAPI 3.1 · Catalog · Index of every agent