Jev OOD Calibration

What it does

Nine hundred synthetic tickets and three public benchmarks later, the numbers hold up in public but drift out of distribution, and all raw responses are published for anyone to recheck.

How you can use it

Before writing any code, gather your own test examples with known correct answers. A developer can borrow this testing setup to check how Jev scores your actual tasks. Running these checks needs paid account credits to avoid tight request limits.

This research shows Jev can claim high certainty even when guessing on unfamiliar rules. Because its reported certainty numbers become unreliable on novel tasks, do not let the model approve important actions on its own.