KitScout

Categories › AI & Agents › Thinking, planning & prompt tools

CompletionKit

MCP serverVerified domain · completionkit.comRemote● LiveAPI key needed

Prompt evals over MCP: run a prompt on your dataset, score each output 1-5 with an LLM judge.

Install

Remote server: your requests and data go to completionkit.com. Claude Code: claude mcp add --transport http completionkit https://completionkit.com/mcp · Claude app: Settings → Connectors → Add custom connector.

What it can do (first 40)

The tools this server listed when we connected to it, in its own words.

  • prompts_list — List all prompts
  • prompts_get — Get a prompt by ID
  • prompts_create — Create a prompt
  • prompts_update — Update a prompt. If the prompt already has runs, this creates a new DRAFT version (current=false) rather than editing in place or publishing
  • prompts_delete — Delete a prompt
  • prompts_publish — Publish a prompt version, making it the current version
  • prompts_suggest_improvement — Suggest an improved version of a prompt, grounded in a run's test results and judge feedback. Analyzes the run's responses, scores, and revi
  • runs_list — List all runs
  • runs_get — Get a run by ID, including "metric_averages": a per-metric breakdown with each metric's average score (or pass rate for checks), how many ro
  • runs_create — Create a run. Omit prompt_id and provide output_column to score existing outputs by grading a pre-existing dataset column instead of generat
  • runs_update — Update a run
  • runs_delete — Delete a run
  • runs_generate — Start a run. Required for every run, including score-only runs (no prompt): generates responses with the prompt when there is one, otherwise
  • runs_regrade — Re-grade a run's existing responses with its currently attached metrics, without regenerating. Use after attaching or editing metrics on an
  • runs_rerun — Create and start a fresh copy of a run with the same prompt, dataset, metrics, and settings. Use when the judge changed and you want a clean
  • runs_retry_failures — Re-run only the failed responses of a run, optionally limited to specific response ids via "only".
  • responses_list — List responses for a run, in row order. Returns {total, limit, offset, returned, responses}. Defaults to 50 rows because full payloads are l
  • responses_get — Get a specific response
  • datasets_list — List all datasets
  • datasets_get — Get a dataset by ID
  • datasets_create — Create a dataset with CSV data. First row is the header. Two column names are recognized specially: "expected_output" is each row's answer k
  • datasets_update — Update a dataset
  • datasets_delete — Delete a dataset
  • datasets_create_from_url — Create a dataset by downloading CSV from a URL instead of inlining it. Use this for large datasets: pass a public http(s) URL and the server
  • metrics_list — List all metrics
  • metrics_get — Get a metric by ID
  • metrics_create — Create a metric with evaluation criteria. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value
  • metrics_update — Update a metric. For a deterministic check set metric_type:"check" and check_config. Per-kind required keys: value (contains/not_contains/eq
  • metrics_delete — Delete a metric
  • metrics_suggest_variants — Ask the model to rewrite the metric's judge instruction in N variants targeted at the recent disagreements. Each variant is saved as a draft
  • metric_groups_list — List all metric groups
  • metric_groups_get — Get a metric group by ID
  • metric_groups_create — Create a metric group
  • metric_groups_update — Update a metric group
  • metric_groups_delete — Delete a metric group
  • metric_versions_list — List every MetricVersion (drafts + published) for a metric, newest first. Each row carries version_number, state, source, current flag, and
  • metric_versions_publish — Publish a MetricVersion as the live version of its metric. Works for both 'draft → published' and 'revert to an older published version → cu
  • metric_versions_dismiss — Destroy a draft MetricVersion (use for either source: 'edit' or source: 'suggestion'). Published versions are refused. To demote a published
  • provider_credentials_list — List all provider credentials (API keys are not exposed)
  • provider_credentials_get — Get a provider credential by ID (API key is not exposed)

KitScout test notes

Not tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.

Similar tools