JevMade Sign in
← Back to experiments

Benchmarks & research

When the Right Answer Is Missing

A paired arithmetic study tests whether Jev can reject a menu when every numerical answer is wrong.

Source screenshot of When the Right Answer Is Missing
SOURCE SCREENSHOTFull screenshot ↗

What it does

Across 100 cases, the authors report 99 correct selections but only 7 correct rejections. Checking each answer through Jev's native yes-or-no probability response works better here than a True/False choice menu. The paper does not identify the exact Jev model tested.

How you can use it

Read the paper's paired three-answer arithmetic examples, where the correct answer is known. Keep the question fixed and compare menus with and without that answer. Compare native yes-or-no probability responses with a True/False choice menu; count a case correct only if all three checks are right. If using probabilities to reject choices, tune the rule on separate examples and count newly wrong selections too.

Maker-reported (not independently measured by JevMade): Authors report Choice selection 99/100 and answer-absent rejection 7/100 on paired synthetic arithmetic cases. · Native Boolean verification gets 99/100 answer-absent cases fully correct; each case requires all three candidate judgments to be correct. · A 0.03 rejection cutoff selected on 20 separate development problems gives 79/100 rejection and 97/100 selection on the disjoint evaluation set. It is not a universal cutoff.

Primitives
choice
Platform
Hosted Jev API / research preprint
Added

Keep this for later

Sign in to bookmark experiments, guides and videos, and keep notes only you can see.

Continue to sign in

We’ll bring you back to this listing.