← Market

Robots Check

Read a site's robots.txt and say whether a crawler (user_agent, default *) may fetch a given URL or path, which rule decided it, the crawl delay and the sitemaps listed.

GET https://lookups.mtfit-mcp.workers.dev/api/robots_check

0.001 USDC on Base · payTo 0x2eDB61aE5aAAfDC59b715C2cBB5672cA0C402B80

Status: verified · payable

Operator badge: embed this endpoint's live status badge — it reads "Verified Payable" only while the probe record backs it.

Settlement: direct to the seller's payTo — 0% HIVE402 take

Payment routes

Base · 0.001 USDC · payTo 0x2eDB61aE5aAAfDC59b715C2cBB5672cA0C402B80 · via cdp · settles direct
{
  "url": "https://www.google.com/search",
  "user_agent": "Googlebot"
}
{
  "allowed": false,
  "crawl_delay": null,
  "group": [
    "*",
    "google-extended",
    "yandex"
  ],
  "matched_rule": "Disallow: /search",
  "path": "/search",
  "robots_status": 200,
  "robots_url": "https://www.google.com/robots.txt",
  "sitemaps": [
    "https://www.google.com/sitemap.xml"
  ],
  "tool": "robots_check",
  "user_agent": "Googlebot",
  "version": "1.1.0"
}

Script Master Labs LLC · SDVOSB · Kinston, NC · Terms · Refunds · Privacy · Contact

SML network: ScriptMasterLabs — the main site · SML Dual Market — the human storefront · VA Ratings — VA disability ratings, free for veterans

Agent intel trust badge

Trust scorepending / 100 · breakdown
Probe uptime (30d)— over 0 probes · daily
Avg 402 latency— ms
Paid calls (30d, on-chain)0 calls · 0 buyers

How to call

GET https://lookups.mtfit-mcp.workers.dev/api/robots_check · input in query · spec from live-challenge

Input schema
{
  "type": "object",
  "properties": {
    "url": {
      "type": "string",
      "minLength": 1,
      "maxLength": 2000
    },
    "user_agent": {
      "default": "*",
      "type": "string",
      "maxLength": 100
    }
  },
  "required": [
    "url"
  ]
}
Output schema
{
  "type": "object"
}
curl -i -X GET 'https://lookups.mtfit-mcp.workers.dev/api/robots_check?url=https%3A%2F%2Fwww.google.com%2Fsearch&user_agent=Googlebot'

Agents: POST https://hive402.scriptmasterlabs.com/mcp → route_task / proxy_call · SDK /sdk/hive402.mjs

Try it (your wallet signs; HIVE402 never holds funds)