Categories › AI & Agents › Models, LLM APIs, routing, serving, training & GPUs
Benchmark your own prompt on 100+ LLMs with real API calls; pick the cheapest model that passes.
Remote server: your requests and data go to openmark.ai. Claude Code: claude mcp add --transport http openmark https://openmark.ai/mcp · Claude app: Settings → Connectors → Add custom connector.
The tools this server listed when we connected to it, in its own words.
get_task_format — How to write an OpenMark task: the YAML structure, all 18 scoring modes, and tips. Call this before writing a task. Pass a topic for the fulvalidate_task — Check a task YAML the way OpenMark will before running it: structure, scoring settings, and the test limit of the user's plan. Free, saves nlist_models — Current OpenMark model ids with list prices ($ per 1M tokens), as text and as a structured list. Use these ids in run_benchmark. Append @lowprepare_attachments — Get an upload_uid and a one-hour upload link (no API key needed) for images, PDFs or text documents, with the exact curl command. Upload theestimate_cost — Credits a run_benchmark call would use, before spending anything: the expected total, the split per model, the balance needed to start, the run_benchmark — Save a task (YAML, see get_task_format) to the user's OpenMark account and run it on the given models with real API calls. Returns a job_id get_results — Status of a benchmark job, and once completed the ranked results (score, stability, cost per run, latency), the tests each model missed withexport_openclaw_csv — A completed job's results as the CSV the OpenMark Router plugin for OpenClaw reads (the same file as the app's Export > OpenClaw). Save it iget_account — The user's plan, credit balance and limits, and what a credit pack, Pro or Expert would unlock. Use it before large runs, or when a run is rwatch_task — Watch the task of a completed job: when new or cheaper models ship, OpenMark re-tests up to 4 of them next to the user's pick and emails theNot tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.