Categories › Web & Browser › Scraping, crawling & page-to-text
Check whether a website works for AI agents before visiting it, and the steps that worked.
Remote server: your requests and data go to botworthy.dev. Claude Code: claude mcp add --transport http botworthy https://botworthy.dev/mcp · Claude app: Settings → Connectors → Add custom connector.
The tools this server listed when we connected to it, in its own words.
check_site — Look up how well an AI agent could use a website when Botworthy last scanned it: whether it blocked agents, the score out of 100, advice, anrequest_scan — Queue a website that Botworthy hasn't scanned yet. Scans run in batches by Botworthy's own agent, so expect days, not minutes; don't wait foreport_visit — Tell Botworthy how one of its 5 tasks went when you did it on a website. Reports aren't published: they decide what gets scanned or rescanneNot tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.