← Market

Parse · Robots

Fetch and parse robots.txt (+ llms.txt if present) into structured JSON, including which AI crawlers are blocked.

GET https://toolvend.dev/parse/robots

0.005 USDC on Solana · payTo FSnwAPTNNSxqnR8MtuNrtwLkyYusZ4hncp684HxYpdLb

Status: verified · payable

Operator badge: embed this endpoint's live status badge — it reads "Verified Payable" only while the probe record backs it.

Settlement: direct to the seller's payTo — 0% HIVE402 take

Payment routes

Solana · 0.005 USDC · payTo FSnwAPTNNSxqnR8MtuNrtwLkyYusZ4hncp684HxYpdLb · via cdp · settles direct
Base · 0.005 USDC · payTo 0x55ae5d177cdc9493391a632794fd048e8dba4e20 · via cdp · settles direct
{
  "url": "https://example.com"
}
{
  "llmsTxt": {
    "present": false
  },
  "robots": {
    "aiAgentsBlocked": [
      "GPTBot"
    ],
    "groups": [],
    "sitemaps": []
  }
}

Script Master Labs LLC · SDVOSB · Kinston, NC · Terms · Refunds · Privacy · Contact

SML network: ScriptMasterLabs — the main site · SML Dual Market — the human storefront · VA Ratings — VA disability ratings, free for veterans

Agent intel trust badge

Trust score64 / 100 · breakdown
Probe uptime (30d)100.0% over 1 probes · daily
Avg 402 latency649 ms
Paid calls (30d, on-chain)0 calls · 0 buyers
Facilitator-reported (30d)1 calls · 1 payers (CDP Bazaar)

How to call

GET https://toolvend.dev/parse/robots · input in query · spec from live-challenge

Input schema
{
  "type": "object",
  "properties": {
    "url": {
      "type": "string",
      "description": "Public HTTP(S) site origin or URL"
    }
  },
  "required": [
    "url"
  ]
}
Output schema
{
  "type": "object"
}
curl -i -X GET 'https://toolvend.dev/parse/robots?url=https%3A%2F%2Fexample.com'

Agents: POST https://hive402.scriptmasterlabs.com/mcp → route_task / proxy_call · SDK /sdk/hive402.mjs

Try it (your wallet signs; HIVE402 never holds funds)