← Market

Robots

Guest listing — SiteCheck. Checks whether a URL's path may be crawled under robots.txt (RFC 9309): the verdict for a user agent (default: *, Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended and CCBot), the rule that decided it and its line, crawl-delay, sitemaps and the matching group's lines. Longest rule wins, allow wins a tie, * and $ work. No robots.txt means everything is allowed; a server error is a 503, not charged.

GET https://api.sitecheck-api.workers.dev/api/robots

0.002 USDC on Base · payTo 0xb8491F953e6860D3b5840b73c60883Ff099ceAfA

Status: verified · payable

Settlement: direct to the seller's payTo — 0% HIVE402 take

Payment routes