← Market

Web · Robots

A site's robots.txt rules per user agent, whether AI crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, PerplexityBot) are blocked, and the URLs listed in its sitemap. Use before crawling a site or for SEO audits.

GET https://bess-x402.onrender.com/v1/web/robots

0.003 USDC on Base · payTo 0x1129A6BB9C6165FF07fdC509c107B522CA0e1871

Status: unverified (probe-pending)

Settlement: direct to the seller's payTo — 0% HIVE402 take

Payment routes