Categories › Web & Browser › Scraping, crawling & page-to-text
Check a page, AI-crawler access, llms.txt and backlinks of a site. Free, read-only.
Remote server: your requests and data go to citedby.eu. Claude Code: claude mcp add --transport http citedby https://citedby.eu/mcp · Claude app: Settings → Connectors → Add custom connector.
The tools this server listed when we connected to it, in its own words.
check_page — Fetch one public web page as CitedBot (robots.txt respected) and return what is wrong with it for AI answers and search: title, description,check_ai_crawler_access — Read a site's robots.txt and test the server for GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, ClaudeBot, Google-Extended, Bingbot andcheck_llms_txt — Fetch /llms.txt of a site and report whether it exists, whether its links work, and whether /llms-full.txt exists. When there is none, the fbacklinks — How many other domains link to a domain, and the strongest referrers, from the Common Crawl domain web graph. Covers domains with 100 or morNot tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.