AnyJev develops readout levels from label-free option rotation to supervised calibration and hidden-state heads, then evaluates order bias, calibration error, and selective-automation coverage on local Qwen models.
Original by Jiamu Zhang, Tianze Yang, Yucheng Shi, and Liang WuEvaluationGitHub repositorySource reviewed
Before you dive in
What you’ll find in the original
Rotate option labels and divide out a label prior before treating vocabulary logits as decision probabilities.
Measure expected calibration error and coverage at a fixed tolerated error, not accuracy alone.
Keep task competence separate from calibration: a calibrator cannot rescue a model that cannot solve the underlying decision.
Worth knowing
On the project's BANKING77 test, coverage at 5% risk is explicitly high variance at n=300. Its typed-decisions accuracy is agreement with a teacher LLM, and the letter readout supports at most 26 options.