Our summary
Paying a large model to check thousands of AI answers can cost more than producing them. The study tries Jev 1.13 first, comparing it with sixteen other judges and blinded human decisions. It separates reading a verdict from supplied text from reasoning through a solution.
Jev comes closer to larger judges on text-reading tasks than on math, code and logic. The authors accept a judgment only when confidence is sufficient in both answer orders; otherwise GPT-6 takes over. Their frozen-cutoff comparison on fresh cases slightly improves accuracy at 41% of the cost of GPT-6 alone.
The cutoff is not universal. Routing by confidence weakens with manipulated writing styles or prose lacking a reference answer. The study uses one proprietary Jev version and sequential calls. Its local-validation recipe checks labeled examples to see whether a proposed cutoff meets the accuracy a task requires.
Key takeaways
- Distinguish reading a verdict from solving a problem. The gap between Jev and larger judges changes with that distinction.
- Check both answer orders and freeze the acceptance rule before testing cases that did not help choose the cutoff.
- Test your own labeled examples, including altered writing styles and prose without reference answers, before adopting this cutoff.