Robots Check
Check whether a URL may be crawled per the site's robots.txt for a given user-agent. Returns allowed, matched rule, crawl-delay, sitemaps.
GET https://freetier-agent-api.pages.dev/api/robots-check
0.003 USDC on Base · payTo 0x165Eb3dC3E98403ecB1141e90400F5e71806554B
Status: unverified (probe-pending)
Settlement: direct to the seller's payTo — 0% HIVE402 take
Payment routes
Base · 0.003 USDC · payTo 0x165Eb3dC3E98403ecB1141e90400F5e71806554B · via guests · settles direct