KitScout

Categories › Web & Browser › Scraping, crawling & page-to-text

Botcard

MCP serverVerified domain · whosthatbot.comRemote● LiveFree to try

Cited AI crawler registry (training, search or user fetch) plus robots.txt checks. No fetching.

Install

Remote server: your requests and data go to whosthatbot.com. Claude Code: claude mcp add --transport http botcard https://whosthatbot.com/mcp · Claude app: Settings → Connectors → Add custom connector.

What it can do

The tools this server listed when we connected to it, in its own words.

  • check_robots — Match a robots.txt file the caller pastes against Botcard's AI crawler tokens, using RFC 9309. Each row is allow or disallow plus that token
  • list_crawlers — List the AI crawler robots.txt tokens in this dataset. Each row names the vendor page the summary was checked against. This tool does not fe
  • get_crawler — Return one card by id, including the vendor source URL and the date it was read. This tool does not fetch any website. Do not send url, host
  • suggest_robots — Write a robots.txt from this dataset for the kinds you allow or block. Each group cites the vendor page. effect request_only means the vendo

KitScout test notes

Not tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.

Similar tools