Our summary
Rather than writing a reply, a decision model selects from supplied answers. Artificial Analysis tests that ability in four areas, each worth a quarter of the index: knowledge without invented answers, difficult multiple-choice reasoning, tool choice and deciding whether text is safe.
The launch comparison measures fees and response times, too, and checks whether probabilities line up with correct answers. Jev tops the decision-model total in this snapshot, but other models lead on price or speed. A larger language model scores higher overall at greater cost and delay under the tested settings.
Confidence often runs ahead of accuracy, especially for knowledge and reasoning. Classification and tool choice fare better. The tests cover hosted services on fixed tasks, and the leaderboard can change. Neither a high probability nor a high total tells you which model will suit your own work.
Key takeaways
- Look at all four task scores. Knowledge and difficult reasoning account for half the total, which can hide individual weaknesses.
- The launch comparison has different leaders for price, speed and overall score. None of those measures chooses a model for you.
- Check probabilities against known outcomes before allowing high confidence to guide an action.