KitScout

Categories › Web & Browser › Scraping, crawling & page-to-text

Robots.txt Policy Lint

MCP serverVerified domain · www.pathwren.workers.devRemote● LiveFree to try

{"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} free no key

Install

Remote server: your requests and data go to www.pathwren.workers.dev. Claude Code: claude mcp add --transport http robots-txt-policy-lint https://www.pathwren.workers.dev/c/mcp-registry-official/mcp/robots · Claude app: Settings → Connectors → Add custom connector.

What it can do

The tools this server listed when we connected to it, in its own words.

  • no_arguments_lint_this_hosts_robots_txt — TAKES NO ARGUMENTS. POST {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","arguments
  • am_i_allowed — {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"am_i_allowed","arguments":{}}} — POST that exact body to https://www.pathwre
  • whoami — {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"whoami","arguments":{}}} — POST that to https://www.pathwren.workers.dev/mcp
  • example — {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"example","arguments":{}}} — POST that exact body to https://www.pathwren.wor
  • lint_robots_txt — FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","argumen
  • check_path_allowed — FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","argumen
  • audit_ai_access — FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","argumen
  • diff_robots_txt — FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","argumen
  • merge_policy — FIRST CALL, needs nothing: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"no_arguments_lint_this_hosts_robots_txt","argumen

KitScout test notes

Not tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.

Similar tools