Robots Txt · Analyze
Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked for any bot — so agents can respect crawl policies and locate sitemaps before scraping.
GET https://netintel.dev/robots-txt/analyze
0.01 USDC on Base · payTo 0xdaDc335482AD545296Fd7b28518A251fFCbEb9Df
Status: verified · payable
Operator badge: embed this endpoint's live status badge — it reads "Verified Payable" only while the probe record backs it.
Settlement: direct to the seller's payTo — 0% HIVE402 take
Payment routes
{
"domain": "example.com"
}{
"domain": "example.com",
"findings": [],
"found": true,
"grade": "B",
"path_check": {
"allowed": false,
"matched_rule": "Disallow: /private/",
"path": "/private/page",
"rule_type": "disallow",
"user_agent": "*"
},
"raw_content_preview": "User-agent: *\nDisallow: /private/\n...",
"robots_url": "https://example.com/robots.txt",
"rules": [
{
"allow": [
"/public/"
],
"crawl_delay": null,
"disallow": [
"/private/",
"/admin/"
],
"user_agent": "*"
}
],
"score": 85,
"sitemaps": [
"https://example.com/sitemap.xml"
],
"status_code": 200
}Script Master Labs LLC · SDVOSB · Kinston, NC · Terms · Refunds · Privacy · Contact
SML network: ScriptMasterLabs — the main site · SML Dual Market — the human storefront · VA Ratings — VA disability ratings, free for veterans
Agent intel 
| Trust score | 68 / 100 · breakdown |
|---|---|
| Probe uptime (30d) | 100.0% over 1 probes · daily |
| Avg 402 latency | 168 ms |
| Paid calls (30d, on-chain) | 2 calls · 2 buyers · shared price cohort of 24 |
| Facilitator-reported (30d) | 1 calls · 1 payers (CDP Bazaar) |
How to call
GET https://netintel.dev/robots-txt/analyze · input in query · spec from live-challenge
Input schema
{
"type": "object",
"properties": {
"domain": {
"type": "string",
"description": "Domain name to fetch robots.txt from (e.g. example.com)"
},
"path": {
"type": "string",
"description": "URL path to check permission for (e.g. /api/data)"
},
"user_agent": {
"type": "string",
"description": "Bot name to check rules for (default: *)"
}
},
"required": [
"domain"
]
}Output schema
{
"type": "object",
"properties": {
"domain": {
"type": "string"
},
"robots_url": {
"type": "string"
},
"found": {
"type": "boolean"
},
"status_code": {
"type": "number"
},
"raw_content_preview": {
"type": "string"
},
"sitemaps": {
"type": "array",
"items": {
"type": "string"
}
},
"rules": {
"type": "array",
"items": {
"type": "object"
}
},
"path_check": {
"type": "object"
},
"score": {
"type": "number"
},
"grade": {
"type": "string"
},
"findings": {
"type": "array",
"items": {
"type": "object"
}
}
}
}curl -i -X GET 'https://netintel.dev/robots-txt/analyze?domain=example.com'
Agents: POST https://hive402.scriptmasterlabs.com/mcp → route_task / proxy_call · SDK /sdk/hive402.mjs