Robots Check
Read a site's robots.txt and say whether a crawler (user_agent, default *) may fetch a given URL or path, which rule decided it, the crawl delay and the sitemaps listed.
GET https://lookups.mtfit-mcp.workers.dev/api/robots_check
0.001 USDC on Base · payTo 0x2eDB61aE5aAAfDC59b715C2cBB5672cA0C402B80
Status: verified · payable
Operator badge: embed this endpoint's live status badge — it reads "Verified Payable" only while the probe record backs it.
Settlement: direct to the seller's payTo — 0% HIVE402 take
Payment routes
{
"url": "https://www.google.com/search",
"user_agent": "Googlebot"
}{
"allowed": false,
"crawl_delay": null,
"group": [
"*",
"google-extended",
"yandex"
],
"matched_rule": "Disallow: /search",
"path": "/search",
"robots_status": 200,
"robots_url": "https://www.google.com/robots.txt",
"sitemaps": [
"https://www.google.com/sitemap.xml"
],
"tool": "robots_check",
"user_agent": "Googlebot",
"version": "1.1.0"
}Script Master Labs LLC · SDVOSB · Kinston, NC · Terms · Refunds · Privacy · Contact
SML network: ScriptMasterLabs — the main site · SML Dual Market — the human storefront · VA Ratings — VA disability ratings, free for veterans
Agent intel 
| Trust score | pending / 100 · breakdown |
|---|---|
| Probe uptime (30d) | — over 0 probes · daily |
| Avg 402 latency | — ms |
| Paid calls (30d, on-chain) | 0 calls · 0 buyers |
How to call
GET https://lookups.mtfit-mcp.workers.dev/api/robots_check · input in query · spec from live-challenge
Input schema
{
"type": "object",
"properties": {
"url": {
"type": "string",
"minLength": 1,
"maxLength": 2000
},
"user_agent": {
"default": "*",
"type": "string",
"maxLength": 100
}
},
"required": [
"url"
]
}Output schema
{
"type": "object"
}curl -i -X GET 'https://lookups.mtfit-mcp.workers.dev/api/robots_check?url=https%3A%2F%2Fwww.google.com%2Fsearch&user_agent=Googlebot'
Agents: POST https://hive402.scriptmasterlabs.com/mcp → route_task / proxy_call · SDK /sdk/hive402.mjs