Crawl
Crawl a site and get clean markdown for every page, in one call. Same-host BFS with depth and page-budget limits, robots.txt honoured by default. $0.0015 per requested page (1-25).
POST https://api.weblens.dev/crawl
0.015 USDC on Base · payTo 0x5f71Ce4135004b5fa4dF81014721598E1C7c5AF8
Status: verified · payable
Operator badge: embed this endpoint's live status badge — it reads "Verified Payable" only while the probe record backs it.
Settlement: direct to the seller's payTo — 0% HIVE402 take
Payment routes
{
"limit": 10,
"maxDepth": 2,
"url": "https://example.com"
}{
"crawledAt": "2026-07-31T12:00:00.000Z",
"pages": [
{
"content": "# Example\\n\\nWelcome...",
"depth": 0,
"status": "success",
"title": "Example",
"truncated": false,
"url": "https://example.com"
},
{
"content": "# About us...",
"depth": 1,
"status": "success",
"title": "About",
"truncated": false,
"url": "https://example.com/about"
}
],
"requestId": "req_crawl123",
"summary": {
"crawled": 2,
"discovered": 5,
"failed": 0,
"limit": 10,
"maxDepth": 2,
"robotsRespected": true,
"successful": 2
},
"url": "https://example.com"
}Script Master Labs LLC · SDVOSB · Kinston, NC · Terms · Refunds · Privacy · Contact
SML network: ScriptMasterLabs — the main site · SML Dual Market — the human storefront · VA Ratings — VA disability ratings, free for veterans
Agent intel 
| Trust score | 62 / 100 · breakdown |
|---|---|
| Probe uptime (30d) | 100.0% over 1 probes · daily |
| Avg 402 latency | 1625 ms |
| Paid calls (30d, on-chain) | 0 calls · 0 buyers |
| Facilitator-reported (30d) | 1 calls · 1 payers (CDP Bazaar) |
How to call
POST https://api.weblens.dev/crawl · input in body · spec from live-challenge
Input schema
{
"type": "object",
"properties": {
"url": {
"type": "string",
"format": "uri",
"description": "The URL to process",
"example": "https://example.com"
},
"limit": {
"default": 10,
"description": "Page budget — you are charged per requested page",
"type": "number",
"minimum": 1,
"maximum": 25,
"example": 10
},
"maxDepth": {
"default": 2,
"description": "Link depth from the start URL (0 = start page only)",
"type": "number",
"minimum": 0,
"maximum": 3,
"example": 2
},
"include": {
"default": [],
"description": "Only crawl URLs whose path+query contains one of these",
"maxItems": 20,
"type": "array",
"items": {
"type": "string",
"minLength": 1,
"maxLength": 200
}
},
"exclude": {
"default": [],
"description": "Skip URLs whose path+query contains one of these",
"maxItems": 20,
"type": "array",
"items": {
"type": "string",
"minLength": 1,
"maxLength": 200
}
},
"respectRobots": {
"default": true,
"description": "Honour robots.txt (default true; disable only for sites you control)",
"type": "boolean"
},
"maxChars": {
"default": 8000,
"description": "Per-page content character cap",
"type": "number",
"minimum": 500,
"maximum": 50000
},
"timeout": {
"default": 10000,
"description": "Request timeout in milliseconds",
"type": "number",
"minimum": 5000,
"maximum": 30000
}
},
"required": [
"url"
]
}Output schema
{
"type": "object",
"properties": {
"url": {
"type": "string"
},
"pages": {
"type": "array",
"items": {
"type": "object",
"properties": {
"url": {
"type": "string"
},
"depth": {
"type": "integer"
},
"status": {
"type": "string",
"enum": [
"success",
"failed"
]
},
"title": {
"type": "string"
},
"content": {
"type": "string",
"description": "Page as clean markdown"
},
"truncated": {
"type": "boolean"
},
"error": {
"type": "string"
}
}
},
"description": "Per-page results with url, depth, status, title, content, truncated, error"
},
"summary": {
"type": "object",
"properties": {
"crawled": {
"type": "integer"
},
"successful": {
"type": "integer"
},
"failed": {
"type": "integer"
},
"discovered": {
"type": "integer"
},
"limit": {
"type": "integer"
},
"maxDepth": {
"type": "integer"
},
"robotsRespected": {
"type": "boolean"
}
},
"description": "crawled, successful, failed, discovered, limit, maxDepth, robotsRespected"
},
"crawledAt": {
"type": "string"
},
"requestId": {
"type": "string"
}
}
}curl -i -X POST 'https://api.weblens.dev/crawl' \
-H 'content-type: application/json' -d '{"url":"https://example.com","limit":10,"maxDepth":2}'
Agents: POST https://hive402.scriptmasterlabs.com/mcp → route_task / proxy_call · SDK /sdk/hive402.mjs