← Market

Site Crawl

Everything a crawler or SEO agent needs before visiting a site: robots.txt rules for your user agent (is this URL allowed? crawl delay?), the site's sitemaps with page URLs and last-modified dates (following one sitemap-index level), and the internal/external links on the page.

GET https://aipaygen.com/v1/site_crawl

0.002 USDC on Base · payTo 0xebA22e8CD05Cb72A2F913099D6cCA666DD6D35B0

Status: verified · payable

Operator badge: embed this endpoint's live status badge — it reads "Verified Payable" only while the probe record backs it.

Settlement: direct to the seller's payTo — 0% HIVE402 take

Payment routes

Base · 0.002 USDC · payTo 0xebA22e8CD05Cb72A2F913099D6cCA666DD6D35B0 · via cdp · settles direct
{
  "max_urls": 20,
  "url": "https://www.cloudflare.com/learning/",
  "user_agent": "GPTBot"
}

Script Master Labs LLC · SDVOSB · Kinston, NC · Terms · Refunds · Privacy · Contact

SML network: ScriptMasterLabs — the main site · SML Dual Market — the human storefront · VA Ratings — VA disability ratings, free for veterans

Agent intel trust badge

Trust score66 / 100 · breakdown
Probe uptime (30d)100.0% over 1 probes · daily
Avg 402 latency360 ms
Paid calls (30d, on-chain)64 calls · 1 buyers · shared price cohort of 20
Facilitator-reported (30d)4 calls · 1 payers (CDP Bazaar)

How to call

GET https://aipaygen.com/v1/site_crawl · input in query · spec from live-challenge

Input schema
{
  "type": "object",
  "properties": {
    "url": {
      "type": "string",
      "description": "Page or site URL, e.g. https://example.com/blog"
    },
    "user_agent": {
      "description": "Crawler name to check rules for (default \"*\"), e.g. \"GPTBot\", \"Googlebot\".",
      "type": "string",
      "maxLength": 100
    },
    "include": {
      "description": "Sections to return (default all three).",
      "type": "array",
      "items": {
        "type": "string",
        "enum": [
          "robots",
          "sitemap",
          "links"
        ]
      }
    },
    "max_urls": {
      "description": "Sitemap URLs and links to list (default 100 each).",
      "type": "integer",
      "minimum": 1,
      "maximum": 500
    }
  },
  "required": [
    "url"
  ],
  "additionalProperties": false
}
curl -i -X GET 'https://aipaygen.com/v1/site_crawl?url=https%3A%2F%2Fwww.cloudflare.com%2Flearning%2F&user_agent=GPTBot&max_urls=20'

Agents: POST https://hive402.scriptmasterlabs.com/mcp → route_task / proxy_call · SDK /sdk/hive402.mjs

Try it (your wallet signs; HIVE402 never holds funds)