← Market

Crawl Check · :Domain

Crawl pre-flight for a domain: will you be blocked, is content behind a paywall, and how much is actually there. Returns per-bot robots.txt verdicts for 16 AI crawlers, crawlable page count from sitemaps, detected CDN and paid-access signals (x402, TollBit), plus a fetch/skip recommendation. Call before spending requests on an unknown domain.

GET https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain

0.02 USDC on Base · payTo 0x02d15E7f25d29743c27021d32a7B280d760B2bC2

Status: unverified (live-probe)

Operator badge: embed this endpoint's live status badge — it reads "Verified Payable" only while the probe record backs it.

Settlement: direct to the seller's payTo — 0% HIVE402 take

Payment routes

Base · 0.02 USDC · payTo 0x02d15E7f25d29743c27021d32a7B280d760B2bC2 · via cdp · settles direct
Polygon · 0.02 USDC · payTo 0x02d15E7f25d29743c27021d32a7B280d760B2bC2 · via cdp · settles direct
Arbitrum · 0.02 USDC · payTo 0x02d15E7f25d29743c27021d32a7B280d760B2bC2 · via cdp · settles direct
{
  "crawl_surface": {
    "is_estimate": true,
    "pages": 28400,
    "sitemaps_found": 2
  },
  "domain": "example.com",
  "infrastructure": {
    "cdn": [
      "Cloudflare"
    ],
    "paid_access": []
  },
  "reason": "No AI crawler restrictions declared.",
  "recommendation": "fetch",
  "robots": {
    "ai_crawlers_allowed": 16,
    "ai_crawlers_blocked": [],
    "blocks_all_agents": false,
    "present": true
  }
}
Failing the probe? We'll fix it — $19.00.

HIVE Repair reads this endpoint's real probe record, writes the exact fix and re-probes until it shows verified · payable, or you get a full refund. Card or USDC.

Operator pack: up to 10 endpoints, $59.00

Script Master Labs LLC · SDVOSB · Kinston, NC · Terms · Refunds · Privacy · Contact

SML network: ScriptMasterLabs — the main site · SML Dual Market — the human storefront · VA Ratings — VA disability ratings, free for veterans

Agent intel trust badge

Trust score12 / 100 · breakdown
Probe uptime (30d)0.0% over 1 probes · daily
Avg 402 latency— ms
Paid calls (30d, on-chain)0 calls · 0 buyers
Facilitator-reported (30d)1 calls · 1 payers (CDP Bazaar)

How to call

GET https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain · input in query · spec from feed (schema inferred from example)

Output schema
{
  "type": "object",
  "properties": {
    "crawl_surface": {
      "type": "object",
      "properties": {
        "is_estimate": {
          "type": "boolean"
        },
        "pages": {
          "type": "integer"
        },
        "sitemaps_found": {
          "type": "integer"
        }
      }
    },
    "domain": {
      "type": "string"
    },
    "infrastructure": {
      "type": "object",
      "properties": {
        "cdn": {
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "paid_access": {
          "type": "array"
        }
      }
    },
    "reason": {
      "type": "string"
    },
    "recommendation": {
      "type": "string"
    },
    "robots": {
      "type": "object",
      "properties": {
        "ai_crawlers_allowed": {
          "type": "integer"
        },
        "ai_crawlers_blocked": {
          "type": "array"
        },
        "blocks_all_agents": {
          "type": "boolean"
        },
        "present": {
          "type": "boolean"
        }
      }
    }
  }
}
curl -i -X GET 'https://crawl-preflight.postnov01.workers.dev/crawl-check/:domain'

Agents: POST https://hive402.scriptmasterlabs.com/mcp → route_task / proxy_call · SDK /sdk/hive402.mjs

Try-it needs an exact USDC-on-Base route that is verified payable; use your own x402 client or proxy_call.