01-graders-and-pipeline

Build custom LLM evaluation pipelines using the OpenJudge framework. Covers selecting and configuring graders (LLM-based, function-based, agentic), running batch evaluations with GradingRunner, combining scores with aggregators, applying evaluation strategies (voting, average), auto-generating graders from data, and analyzing results (pairwise win rates, statistics, validation metrics). Use when the user wants to evaluate LLM outputs, compare multiple models, design scoring criteria, or build an

Install

KitScout test notes

Not tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.

Similar tools

• REA 456,474 ★
  Reverse engineer anything from your terminal or agent with one CLI and MCP server.
• ask-matt 323,337 ★
  Ask which skill or flow fits your situation. A router over the skills in this repo.
• chief-of-staff 323,337 ★
  Pursue a long-running goal in a single session by co-ordinating subagents.
• claude-handoff 323,337 ★
  Hand the current conversation off to a fresh background agent that picks up the work immediately.
• grill-me 323,337 ★
  A relentless interview to sharpen a plan or design.