← Market

Ai Crawlers

Given a website domain, reports which AI crawlers may fetch its home page under its robots.txt (RFC 9309): GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot and more, each allowed or blocked, plus Googlebot, listed sitemaps, whether /llms.txt is published and page-level noindex/nosnippet, with fixes. Static check, no browser. Not charged if robots.txt cannot be read (no response, or a bot block).

GET https://x402.agentharbor.agency/ai-crawlers

0.01 USDC on Base · payTo 0x840C829530DF7B4CCF700Fa3A0060cF2C80b014e

Status: unverified (probe-pending)

Settlement: direct to the seller's payTo — 0% HIVE402 take

Payment routes