← Market

Crawl

Crawl a site and get clean markdown for every page, in one call. Same-host BFS with depth and page-budget limits, robots.txt honoured by default. $0.0015 per requested page (1-25).

POST https://api.weblens.dev/crawl

0.015 USDC on Base · payTo 0x5f71Ce4135004b5fa4dF81014721598E1C7c5AF8

Status: verified · payable

Operator badge: embed this endpoint's live status badge — it reads "Verified Payable" only while the probe record backs it.

Settlement: direct to the seller's payTo — 0% HIVE402 take

Payment routes

Base · 0.015 USDC · payTo 0x5f71Ce4135004b5fa4dF81014721598E1C7c5AF8 · via cdp · settles direct
Solana · 0.015 USDC · payTo 34sd6wPrmoR9UG9rUgYAFtcWYSVzqDNsvH9VkCkycUeb · via cdp · settles direct
{
  "limit": 10,
  "maxDepth": 2,
  "url": "https://example.com"
}
{
  "crawledAt": "2026-07-31T12:00:00.000Z",
  "pages": [
    {
      "content": "# Example\\n\\nWelcome...",
      "depth": 0,
      "status": "success",
      "title": "Example",
      "truncated": false,
      "url": "https://example.com"
    },
    {
      "content": "# About us...",
      "depth": 1,
      "status": "success",
      "title": "About",
      "truncated": false,
      "url": "https://example.com/about"
    }
  ],
  "requestId": "req_crawl123",
  "summary": {
    "crawled": 2,
    "discovered": 5,
    "failed": 0,
    "limit": 10,
    "maxDepth": 2,
    "robotsRespected": true,
    "successful": 2
  },
  "url": "https://example.com"
}

Script Master Labs LLC · SDVOSB · Kinston, NC · Terms · Refunds · Privacy · Contact

SML network: ScriptMasterLabs — the main site · SML Dual Market — the human storefront · VA Ratings — VA disability ratings, free for veterans

Agent intel trust badge

Trust score62 / 100 · breakdown
Probe uptime (30d)100.0% over 1 probes · daily
Avg 402 latency1625 ms
Paid calls (30d, on-chain)0 calls · 0 buyers
Facilitator-reported (30d)1 calls · 1 payers (CDP Bazaar)

How to call

POST https://api.weblens.dev/crawl · input in body · spec from live-challenge

Input schema
{
  "type": "object",
  "properties": {
    "url": {
      "type": "string",
      "format": "uri",
      "description": "The URL to process",
      "example": "https://example.com"
    },
    "limit": {
      "default": 10,
      "description": "Page budget — you are charged per requested page",
      "type": "number",
      "minimum": 1,
      "maximum": 25,
      "example": 10
    },
    "maxDepth": {
      "default": 2,
      "description": "Link depth from the start URL (0 = start page only)",
      "type": "number",
      "minimum": 0,
      "maximum": 3,
      "example": 2
    },
    "include": {
      "default": [],
      "description": "Only crawl URLs whose path+query contains one of these",
      "maxItems": 20,
      "type": "array",
      "items": {
        "type": "string",
        "minLength": 1,
        "maxLength": 200
      }
    },
    "exclude": {
      "default": [],
      "description": "Skip URLs whose path+query contains one of these",
      "maxItems": 20,
      "type": "array",
      "items": {
        "type": "string",
        "minLength": 1,
        "maxLength": 200
      }
    },
    "respectRobots": {
      "default": true,
      "description": "Honour robots.txt (default true; disable only for sites you control)",
      "type": "boolean"
    },
    "maxChars": {
      "default": 8000,
      "description": "Per-page content character cap",
      "type": "number",
      "minimum": 500,
      "maximum": 50000
    },
    "timeout": {
      "default": 10000,
      "description": "Request timeout in milliseconds",
      "type": "number",
      "minimum": 5000,
      "maximum": 30000
    }
  },
  "required": [
    "url"
  ]
}
Output schema
{
  "type": "object",
  "properties": {
    "url": {
      "type": "string"
    },
    "pages": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "url": {
            "type": "string"
          },
          "depth": {
            "type": "integer"
          },
          "status": {
            "type": "string",
            "enum": [
              "success",
              "failed"
            ]
          },
          "title": {
            "type": "string"
          },
          "content": {
            "type": "string",
            "description": "Page as clean markdown"
          },
          "truncated": {
            "type": "boolean"
          },
          "error": {
            "type": "string"
          }
        }
      },
      "description": "Per-page results with url, depth, status, title, content, truncated, error"
    },
    "summary": {
      "type": "object",
      "properties": {
        "crawled": {
          "type": "integer"
        },
        "successful": {
          "type": "integer"
        },
        "failed": {
          "type": "integer"
        },
        "discovered": {
          "type": "integer"
        },
        "limit": {
          "type": "integer"
        },
        "maxDepth": {
          "type": "integer"
        },
        "robotsRespected": {
          "type": "boolean"
        }
      },
      "description": "crawled, successful, failed, discovered, limit, maxDepth, robotsRespected"
    },
    "crawledAt": {
      "type": "string"
    },
    "requestId": {
      "type": "string"
    }
  }
}
curl -i -X POST 'https://api.weblens.dev/crawl' \
  -H 'content-type: application/json' -d '{"url":"https://example.com","limit":10,"maxDepth":2}'

Agents: POST https://hive402.scriptmasterlabs.com/mcp → route_task / proxy_call · SDK /sdk/hive402.mjs

Try it (your wallet signs; HIVE402 never holds funds)