nemo-evaluator-sdk

Evaluates LLMs across 100+ benchmarks from 18+ harnesses (MMLU, HumanEval, GSM8K, safety, VLM) with multi-backend execution. Use when needing scalable evaluation on local Docker, Slurm HPC, or cloud platforms. NVIDIA's enterprise-grade platform with container-first architecture for reproducible benchmarking.

Install

KitScout test notes

Not tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.

Similar tools

• reverse-engineer-anything 456,474 ★
  Reverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and b
• mattpocock-skills 323,337 ★
  Matt Pocock's agent skills for real engineering — grilling, spec/ticket flows, TDD, code review, domain modell
• scaffold-exercises 323,337 ★
  Create exercise directory structures with sections, problems, solutions, and explainers that pass linting. Use
• wait-what 323,337 ★
  Stop. That last message did not land: re-pitch it.
• template-skill 194,218 ★
  Replace with description of the skill and when Claude should use it.