Our summary
Checking thousands of AI answers with a large model can cost more than producing them. This study tries Jev 1.13 as the first judge, comparing it with sixteen other judges and blinded human decisions. It distinguishes reading a verdict from supplied text from working out a solution through reasoning.
Jev is closer to the larger judges on text-reading tasks and farther behind on math, code and logic. The authors then accept only sufficiently confident judgments in both answer orders, sending the rest to GPT-6. With a frozen cutoff, their held-out comparison costs 41% of GPT-6 alone while slightly improving accuracy.
That result does not make the cutoff universal. Confidence routing weakens when writing style is manipulated or prose lacks a reference answer. The study covers one proprietary Jev version and sequential calls. Its local-validation recipe uses labeled examples to check whether a proposed cutoff meets the task's required accuracy.
Key takeaways
- Reading a verdict from given text is different from solving the problem. Jev's gap to larger judges differs between those tasks.
- Check both answer orders, then fix the acceptance rule before evaluating fresh cases that did not set the cutoff.
- Try your own labeled cases, including changed writing styles and prose without a reference answer, before adopting the study's cutoff.