Categories › Web & Browser › Scraping, crawling & page-to-text
Ask before you scrape: is a site open, may I fetch this URL, and how do I extract a page. No key.
Remote server: your requests and data go to rat-radar.github-a18.workers.dev. Claude Code: claude mcp add --transport http rat-radar https://rat-radar.github-a18.workers.dev/mcp · Claude app: Settings → Connectors → Add custom connector.
The tools this server listed when we connected to it, in its own words.
check_site — Can I scrape this site, and how? Returns a verdict (open, guarded, hostile, blocked), a hostility score, a recommended approach with steps aget_playbook — What should I build with this site, and how? Give a domain and, optionally, a goal in plain words (for example "price alerts", "sales leads"list_sites — List the sites in the dataset, optionally by category or verdict. Use it to find easy targets or to see what is covered.xray_page — How do I extract the data from THIS page? Fetches one public page and returns a ranked extraction plan: embedded JSON, JSON-LD, repeated recxray_diagram — Draw a page's anatomy as an Excalidraw diagram: its regions, the ranked ways to get its data, proof rows, the best route and what the page isearch_sites — Find sites in the dataset by a word: a name, host, category (jobs, shop, news, gov), country or verdict. Use it to discover what is covered can_fetch — Does robots.txt allow fetching this exact URL? Fetches the live robots.txt and applies the longest-match rule for a user agent (default: * psummary — Headline numbers across the dataset: verdict mix, how many sites block AI crawlers (and which), protection vendors, by category and country.Not tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.