KitScout

Categories › Web & Browser › SEO & site audits

RobotsGate

MCP serverVerified domain · robotsgate.mike-tusa.workers.devRemote● LiveFree to try

Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.

Install

Remote server: your requests and data go to robotsgate.mike-tusa.workers.dev. Claude Code: claude mcp add --transport http robotsgate https://robotsgate.mike-tusa.workers.dev/mcp · Claude app: Settings → Connectors → Add custom connector.

What it can do

The tools this server listed when we connected to it, in its own words.

  • list_crawlers — Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other
  • generate_robots — Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token o
  • validate_robots — Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or block
  • check_site — Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes;

KitScout test notes

Not tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.

Similar tools