Categories › Wildcards › Unusual, one-of-a-kind tools
Audit an LLM eval pipeline and surface problems: missing error analysis, unvalidated judges, vanity metrics, etc. Use when inheriting an eval system, when unsure whether evals are trustworthy, or as a starting point when no eval infrastructure exists. Do NOT use when the goal is to build a new evaluator from scratch (use error-discovery, write-judge-prompt, or validate-evaluator instead).
Not tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.