Parse · Robots
Fetch and parse robots.txt (+ llms.txt if present) into structured JSON, including which AI crawlers are blocked.
GET https://toolvend.dev/parse/robots
0.005 USDC on Solana · payTo FSnwAPTNNSxqnR8MtuNrtwLkyYusZ4hncp684HxYpdLb
Status: verified · payable
Operator badge: embed this endpoint's live status badge — it reads "Verified Payable" only while the probe record backs it.
Settlement: direct to the seller's payTo — 0% HIVE402 take
Payment routes
{
"url": "https://example.com"
}{
"llmsTxt": {
"present": false
},
"robots": {
"aiAgentsBlocked": [
"GPTBot"
],
"groups": [],
"sitemaps": []
}
}Script Master Labs LLC · SDVOSB · Kinston, NC · Terms · Refunds · Privacy · Contact
SML network: ScriptMasterLabs — the main site · SML Dual Market — the human storefront · VA Ratings — VA disability ratings, free for veterans
Agent intel 
| Trust score | 64 / 100 · breakdown |
|---|---|
| Probe uptime (30d) | 100.0% over 1 probes · daily |
| Avg 402 latency | 649 ms |
| Paid calls (30d, on-chain) | 0 calls · 0 buyers |
| Facilitator-reported (30d) | 1 calls · 1 payers (CDP Bazaar) |
How to call
GET https://toolvend.dev/parse/robots · input in query · spec from live-challenge
Input schema
{
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "Public HTTP(S) site origin or URL"
}
},
"required": [
"url"
]
}Output schema
{
"type": "object"
}curl -i -X GET 'https://toolvend.dev/parse/robots?url=https%3A%2F%2Fexample.com'
Agents: POST https://hive402.scriptmasterlabs.com/mcp → route_task / proxy_call · SDK /sdk/hive402.mjs