Jev-as-a-Judge for Agent Evals

Daniel Shea and Seán Roche replay five fixed weather-agent responses through Jev and three LLM judges, comparing human-label agreement, score variance, cost, and latency.

Original by Daniel Shea and Seán RocheEvaluationLangChain blogOriginal published 2026-09-20Source reviewed 2026-09-25

Before you dive in

What you’ll find in the original

Freeze the agent outputs before comparing evaluators. A human reviewer labels each response against the same rubric, while repeated judgments measure stability on identical evidence.
Jev matched the human pass/fail labels on all 500 repeated decisions in this test, but those decisions cover only five distinct responses. Low variance and broad accuracy are different claims.
The authors report 92–913× lower quality-score variance and an average of 0.44 seconds and $0.00035 per Jev call. Test whether those gains survive different agents, rubrics, and failure cases.

Worth knowing

This is the writeup of danielgshea/jev-as-a-judge, already listed among the experiments, not a new experiment. Provider sampling defaults were used and Jev's service version was missing from the run metadata. JevMade did not reproduce the benchmark.