We Tested Jev on 791 Labeled Decisions Against Four LLMs
Compares Jev with four LLMs on 791 labeled routing and prompt-injection decisions. The article links the Python harness, raw per-call results, paired tests, and confidence-gated cascade analysis, including cases where Jev is confidently wrong.
Original by Adel DahaniEvaluationAY Automate benchmarkOriginal published Source reviewed
Before you dive in
What you’ll find in the original
Compare accuracy against human labels, not only agreement with a stronger model.
Measure both automatic coverage and errors when tuning a confidence cutoff.
Replay the published harness on your own data before projecting cost or speed savings.
Worth knowing
This is one OpenRouter run on public datasets with label-only choices; the cascade thresholds were examined on the same cases and need separate validation.