← Market

Robots

Guest listing — agent-data-tools.revmesh2074.workers.dev. Robots.txt check. Read a site's robots.txt and say whether a crawler may fetch a URL, by RFC 9309 rules as Google applies them (most specific user-agent group, longest matching Allow or Disallow, Allow on ties, * and $ wildcards). Returns the deciding rule, the group used, the rule count and the sitemaps listed. A missing robots.txt (4xx) allows everything. Use before scraping or crawling a site politely; for the sitemap URLs themselves use sitemap.

GET https://agent-data-tools.revmesh2074.workers.dev/robots

0.01 USDC on Base · payTo 0x6293ded2684F1A76E26bEcef87c137D691Fbf800

Status: unverified (live-probe)

Settlement: direct to the seller's payTo — 0% HIVE402 take

Payment routes

{
  "agent": "Googlebot",
  "allowed": false,
  "group": "*",
  "matchedRule": "Disallow: /search",
  "robotsUrl": "https://www.google.com/robots.txt",
  "rulesInGroup": 280,
  "sitemaps": [
    "https://www.google.com/sitemap.xml"
  ],
  "status": 200,
  "url": "https://www.google.com/search"
}