Crawl Check · :Domain
Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.
GET https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain
0.02 USDC on Base · payTo 0x02d15E7f25d29743c27021d32a7B280d760B2bC2
Status: unverified (live-probe)
Operator badge: embed this endpoint's live status badge — it reads "Verified Payable" only while the probe record backs it.
Settlement: direct to the seller's payTo — 0% HIVE402 take
Payment routes
{
"crawl_surface": {
"is_estimate": true,
"pages": 28400,
"sitemaps_found": 2
},
"domain": "example.com",
"infrastructure": {
"cdn": [
"Cloudflare"
],
"paid_access": []
},
"reason": "No AI crawler restrictions declared.",
"recommendation": "fetch",
"robots": {
"ai_crawlers_allowed": 16,
"ai_crawlers_blocked": [],
"blocks_all_agents": false,
"present": true
}
}HIVE Repair reads this endpoint's real probe record, writes the exact fix and re-probes until it shows verified · payable, or you get a full refund. Card or USDC.
Script Master Labs LLC · SDVOSB · Kinston, NC · Terms · Refunds · Privacy · Contact
SML network: ScriptMasterLabs — the main site · SML Dual Market — the human storefront · VA Ratings — VA disability ratings, free for veterans
Agent intel 
| Trust score | 12 / 100 · breakdown |
|---|---|
| Probe uptime (30d) | 0.0% over 1 probes · daily |
| Avg 402 latency | — ms |
| Paid calls (30d, on-chain) | 0 calls · 0 buyers |
| Facilitator-reported (30d) | 1 calls · 1 payers (CDP Bazaar) |
How to call
GET https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain · input in query · spec from feed (schema inferred from example)
Output schema
{
"type": "object",
"properties": {
"crawl_surface": {
"type": "object",
"properties": {
"is_estimate": {
"type": "boolean"
},
"pages": {
"type": "integer"
},
"sitemaps_found": {
"type": "integer"
}
}
},
"domain": {
"type": "string"
},
"infrastructure": {
"type": "object",
"properties": {
"cdn": {
"type": "array",
"items": {
"type": "string"
}
},
"paid_access": {
"type": "array"
}
}
},
"reason": {
"type": "string"
},
"recommendation": {
"type": "string"
},
"robots": {
"type": "object",
"properties": {
"ai_crawlers_allowed": {
"type": "integer"
},
"ai_crawlers_blocked": {
"type": "array"
},
"blocks_all_agents": {
"type": "boolean"
},
"present": {
"type": "boolean"
}
}
}
}
}curl -i -X GET 'https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain'
Agents: POST https://hive402.scriptmasterlabs.com/mcp → route_task / proxy_call · SDK /sdk/hive402.mjs
Try-it needs an exact USDC-on-Base route that is verified payable; use your own x402 client or proxy_call.