Benchmarks & research

Benchmarking decision models against generative large language models on medical multiple-choice questions

By Oscar Freyer, Daniel Glauert, Moritz Hofmann, Max Ostermann, Stephen Gilbert, Jakob Nikolas Kather

Six researchers put 13 decision models and 15 generative AI setups through 1,273 four-option MedQA exam questions, comparing accuracy, cost, speed and how far each model's confidence can be trusted.

Bookmark Keep this in your collection.

Leave a noteWhat would you try with this?

Only you can see your notes.

Source screenshot of Benchmarking decision models against generative large language models on medical multiple-choice questions — SOURCE SCREENSHOTFull screenshot ↗

What it does

Decision models scored 67.71% to 89.08% and generative setups 78.32% to 96.86%; Jev reached 88.30%. The decision models were usually cheaper and faster, yet answers they gave at least 0.90 probability were still wrong 1.71% to 12.77% of the time. These are exam questions with the options already supplied, so the results say nothing about clinical safety.

How you can use it

You can adapt this evaluation method to test how well different AI systems pick answers from a fixed list of options. A developer can use the researchers' open-source code and a sample set of multiple-choice questions to compare accuracy, cost, and speed across various models.

The study also shows how to track the confidence scores that decision models report. Even when a model claims high certainty, it can still choose the wrong answer. Because these were standardized exam questions, the results do not prove that any model is safe for actual medical care.