Our summary
A decision model can act as a quick, cheap checker for another AI. Hamel Husain and Antaripa Saha gave six of them, including Jev and OpenAI Decisions, 98 conversation logs from an apartment-leasing assistant that people had marked as passing or failing. Each model scored how likely a log contained a mistake.
Most tools flag a log once its score reaches 0.50. The authors show this default can make a good model look weak, so they picked a better cut-off for each model on some logs and measured it on logs kept aside. They also checked whether a score of 0.8 really meant an 80 percent chance.
With tuned cut-offs, OpenAI Decisions scored best but cost about 4.5 times as much as Perplexity Decider, the cheapest of the three finalists. Jev's scores already matched reality well. The test used only 98 logs from one business, so the authors urge readers to repeat these checks on their own data.
Key takeaways
- Do not trust the default 0.50 cut-off: pick each model's cut-off using logs people have already marked, then test it on logs you kept aside.
- Decide first how many real mistakes you must catch and how many false alarms you can accept; here only three of the six models could reach 80 percent on both.
- Check that a score means what it says: Jev's scores already matched how often mistakes really happened, while a simple adjustment called Platt scaling helped three other models.