Categories › AI & Agents › Thinking, planning & prompt tools
Use when the user has a judge/grader and human-labeled data, and wants to measure how well the judge agrees with humans, detect systematic biases, determine whether automatic evaluation can replace human review, or build a human-reduction roadmap. Also use when the user mentions calibration, TPR/TNR, judge validation, inter-rater agreement, Cohen's kappa, bias detection, or "is my automatic evaluation trustworthy." Merges the calibrate and align functions into one skill.
Not tested by KitScout yet. When our harness runs it, pros, cons and the tasks we tried appear here, with the date and evidence.