KitScout

Categories › Web & Browser › Scraping, crawling & page-to-text

Robots.txt & AI Crawler Checker

MCP serverVerified domain · robots-check.quietworklabs.workers.devRemote● LiveFree to try

robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.

Install

Remote server: your requests and data go to robots-check.quietworklabs.workers.dev. Claude Code: claude mcp add --transport http robots-txt-ai-crawler-checker https://robots-check.quietworklabs.workers.dev/mcp · Claude app: Settings → Connectors → Add custom connector.

What it can do

The tools this server listed when we connected to it, in its own words.

  • robots_check — Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309
  • ai_crawler_access — AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Clau
  • sitemap_check — Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry

KitScout test notes

Not tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.

Similar tools