Benchmarks & research

Fast Models, Slow Evidence

Jiawei Li compares Jev and Laya at eleven agent decision points, then audits where the reported savings and thresholds were misleading.

Bookmark Keep this in your collection.

Leave a noteWhat would you try with this?

Only you can see your notes.

Source screenshot of Fast Models, Slow Evidence — SOURCE SCREENSHOTFull screenshot ↗

What it does

Laya’s accuracy falls sharply with many or similar candidates, and reversing option order changes 30% of its answers. The self-audit found that leaving out pre-screen token costs inflated two-tier safety-review savings from 4.3% to 23.9%. Thresholds chosen on the same data used to tune them also failed to guarantee target miss rates on held-out data.

How you can use it

Compare the models on identical inputs and account for the whole workflow. This study tests tool choices, unwanted instructions and search relevance, including what happens when option order changes. Its self-audit is a reason to check savings and cutoffs on new examples, not just the examples used for tuning.