← Market

Robots Txt · Analyze

Fetch and parse a domain's robots.txt file — returns all crawl rules by user-agent, sitemap URLs, crawl delay settings, and checks whether a specific path is allowed or blocked for any bot — so agents can respect crawl policies and locate sitemaps before scraping.

GET https://netintel.dev/robots-txt/analyze

0.01 USDC on Base · payTo 0xdaDc335482AD545296Fd7b28518A251fFCbEb9Df

Status: verified · payable

Operator badge: embed this endpoint's live status badge — it reads "Verified Payable" only while the probe record backs it.

Settlement: direct to the seller's payTo — 0% HIVE402 take

Payment routes

Base · 0.01 USDC · payTo 0xdaDc335482AD545296Fd7b28518A251fFCbEb9Df · via cdp · settles direct
Solana · 0.01 USDC · payTo 8FVeybWnB4dYrxTuPK7WzFZNAiEcFVaDLFnKpRwVGr5s · via cdp · settles direct
{
  "domain": "example.com"
}
{
  "domain": "example.com",
  "findings": [],
  "found": true,
  "grade": "B",
  "path_check": {
    "allowed": false,
    "matched_rule": "Disallow: /private/",
    "path": "/private/page",
    "rule_type": "disallow",
    "user_agent": "*"
  },
  "raw_content_preview": "User-agent: *\nDisallow: /private/\n...",
  "robots_url": "https://example.com/robots.txt",
  "rules": [
    {
      "allow": [
        "/public/"
      ],
      "crawl_delay": null,
      "disallow": [
        "/private/",
        "/admin/"
      ],
      "user_agent": "*"
    }
  ],
  "score": 85,
  "sitemaps": [
    "https://example.com/sitemap.xml"
  ],
  "status_code": 200
}

Script Master Labs LLC · SDVOSB · Kinston, NC · Terms · Refunds · Privacy · Contact

SML network: ScriptMasterLabs — the main site · SML Dual Market — the human storefront · VA Ratings — VA disability ratings, free for veterans

Agent intel trust badge

Trust score68 / 100 · breakdown
Probe uptime (30d)100.0% over 1 probes · daily
Avg 402 latency168 ms
Paid calls (30d, on-chain)2 calls · 2 buyers · shared price cohort of 24
Facilitator-reported (30d)1 calls · 1 payers (CDP Bazaar)

How to call

GET https://netintel.dev/robots-txt/analyze · input in query · spec from live-challenge

Input schema
{
  "type": "object",
  "properties": {
    "domain": {
      "type": "string",
      "description": "Domain name to fetch robots.txt from (e.g. example.com)"
    },
    "path": {
      "type": "string",
      "description": "URL path to check permission for (e.g. /api/data)"
    },
    "user_agent": {
      "type": "string",
      "description": "Bot name to check rules for (default: *)"
    }
  },
  "required": [
    "domain"
  ]
}
Output schema
{
  "type": "object",
  "properties": {
    "domain": {
      "type": "string"
    },
    "robots_url": {
      "type": "string"
    },
    "found": {
      "type": "boolean"
    },
    "status_code": {
      "type": "number"
    },
    "raw_content_preview": {
      "type": "string"
    },
    "sitemaps": {
      "type": "array",
      "items": {
        "type": "string"
      }
    },
    "rules": {
      "type": "array",
      "items": {
        "type": "object"
      }
    },
    "path_check": {
      "type": "object"
    },
    "score": {
      "type": "number"
    },
    "grade": {
      "type": "string"
    },
    "findings": {
      "type": "array",
      "items": {
        "type": "object"
      }
    }
  }
}
curl -i -X GET 'https://netintel.dev/robots-txt/analyze?domain=example.com'

Agents: POST https://hive402.scriptmasterlabs.com/mcp → route_task / proxy_call · SDK /sdk/hive402.mjs

Try it (your wallet signs; HIVE402 never holds funds)