Benchmarks & research

Same Scores, Different Decisions

Test whether a contract supports a statement, then check how answers change when the request changes but the contract does not.

Bookmark Keep this in your collection.

Leave a noteWhat would you try with this?

Only you can see your notes.

Source screenshot of Same Scores, Different Decisions — SOURCE SCREENSHOTFull screenshot ↗

What it does

Existing dataset labels define correct answers, not another AI's opinion. Similar overall scores can hide changed answers. The small repeated-test panel does not establish a general stability advantage for Jev.

How you can use it

Compare the paper's four request conditions for the same contract statement, keeping its established correct answer fixed. Track answers that become right and those that become wrong, rather than just the total score. Saved predictions permit offline comparison; missing original requests and responses prevent a complete check of the historical API runs.