Categories › Web & Browser › Scraping, crawling & page-to-text
Check AI crawler robots.txt policy and monitor public-site policy, sitemap, and llms.txt changes.
Remote server: your requests and data go to actablesite.com. Claude Code: claude mcp add --transport http actablesite-ai-crawler-monitor https://actablesite.com/mcp/ · Claude app: Settings → Connectors → Add custom connector.
The tools this server listed when we connected to it, in its own words.
audit_public_website — Run ActableSite's free three-signal readiness scan for a public website. Returns observable evidence for business identity, semantic readingcheck_ai_crawler_policy — Check the homepage robots.txt rule for eight named OpenAI, Anthropic, Perplexity, and Google AI crawler tokens. The free result stays compleget_crawler_watch_offer — Return the exact $9 monthly price, one-site scope, 15-minute check cadence, two-check confirmation, delivery and cancellation paths, and synget_full_report_offer — Return the exact one-time price, deliverables, delivery path, checkout URL, support contact, and limitations for ActableSite's paid 15-checkget_practice_radar_offer — Return the current Practice Radar weekly-edition receipt, public sample, exact $19 one-time and $39 monthly options, included fields, deliveNot tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.