← Market

Robots Txt

Parse a site's robots.txt: whether a user-agent may crawl a path, which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot...) are blocked, and declared sitemaps. Check before your agent scrapes.

POST https://api.vextorium.com/v1/robots-txt

0.003 USDC on Base · payTo 0xb9f06D2734B7E9D7b21B64C4D8b95bd774b71aCf

Status: unverified (probe-pending)

Settlement: direct to the seller's payTo — 0% HIVE402 take

Payment routes

Base · 0.003 USDC · payTo 0xb9f06D2734B7E9D7b21B64C4D8b95bd774b71aCf · via guests · settles direct
Solana · 0.003 USDC · payTo HcjfmZLgWRZzwj4deEiS7pewc4KBYSw39mW5nTyt3pz2 · via guests · settles direct
Arc · 0.003 USDC · payTo 0xb9f06D2734B7E9D7b21B64C4D8b95bd774b71aCf · via guests · settles direct
XRPL · 3e-9 524C55… · payTo rEac5rTEr7y81iEyrMTrHCRbnktSE2pNCp · via guests · settles direct
XRPL · 0.00231 XRP · payTo rEac5rTEr7y81iEyrMTrHCRbnktSE2pNCp · via guests · settles direct