Site Crawl
Everything a crawler or SEO agent needs before visiting a site: robots.txt rules for your user agent (is this URL allowed? crawl delay?), the site's sitemaps with page URLs and last-modified dates (following one sitemap-index level), and the internal/external links on the page.
GET https://aipaygen.com/v1/site_crawl
0.002 USDC on Base · payTo 0xebA22e8CD05Cb72A2F913099D6cCA666DD6D35B0
Status: verified · payable
Operator badge: embed this endpoint's live status badge — it reads "Verified Payable" only while the probe record backs it.
Settlement: direct to the seller's payTo — 0% HIVE402 take
Payment routes
{
"max_urls": 20,
"url": "https://www.cloudflare.com/learning/",
"user_agent": "GPTBot"
}Script Master Labs LLC · SDVOSB · Kinston, NC · Terms · Refunds · Privacy · Contact
SML network: ScriptMasterLabs — the main site · SML Dual Market — the human storefront · VA Ratings — VA disability ratings, free for veterans
Agent intel 
| Trust score | 66 / 100 · breakdown |
|---|---|
| Probe uptime (30d) | 100.0% over 1 probes · daily |
| Avg 402 latency | 360 ms |
| Paid calls (30d, on-chain) | 64 calls · 1 buyers · shared price cohort of 20 |
| Facilitator-reported (30d) | 4 calls · 1 payers (CDP Bazaar) |
How to call
GET https://aipaygen.com/v1/site_crawl · input in query · spec from live-challenge
Input schema
{
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Page or site URL, e.g. https://example.com/blog"
},
"user_agent": {
"description": "Crawler name to check rules for (default \"*\"), e.g. \"GPTBot\", \"Googlebot\".",
"type": "string",
"maxLength": 100
},
"include": {
"description": "Sections to return (default all three).",
"type": "array",
"items": {
"type": "string",
"enum": [
"robots",
"sitemap",
"links"
]
}
},
"max_urls": {
"description": "Sitemap URLs and links to list (default 100 each).",
"type": "integer",
"minimum": 1,
"maximum": 500
}
},
"required": [
"url"
],
"additionalProperties": false
}curl -i -X GET 'https://aipaygen.com/v1/site_crawl?url=https%3A%2F%2Fwww.cloudflare.com%2Flearning%2F&user_agent=GPTBot&max_urls=20'
Agents: POST https://hive402.scriptmasterlabs.com/mcp → route_task / proxy_call · SDK /sdk/hive402.mjs