A guide to replacing some generative judge calls with typed propositions, calibrated thresholds, and an evaluation system that stores evidence and routes uncertain cases.
Original by Made with JevEvaluationMade with Jev guideSource reviewed
Before you dive in
What you’ll find in the original
Ask one narrow proposition per criterion rather than requesting an overall prose grade.
Calibrate thresholds against labelled examples and route the middle confidence band to a stronger judge or a person.
Keep the judge separate from the evaluation system that stores runs, compares regressions, and displays evidence.
Worth knowing
A Jev probability is not a percentage grade or proof. Published comparisons are workload-specific and several use model-generated reference answers.