Head-to-Head Benchmarks Show Where OpenAI Decisions, Jev, and Parallel Split
A November specification and benchmark comparison maps pricing, request limits, and leaderboard gaps between TypeSafe Jev, OpenAI’s Decisions API, and Parallel.
Evaluations, academic papers, architecture deep dives and empirical benchmarks for decision models.
A November specification and benchmark comparison maps pricing, request limits, and leaderboard gaps between TypeSafe Jev, OpenAI’s Decisions API, and Parallel.
An independent test on 300 customer-support chats finds Jev and OpenAI Decisions tied on zero-shot triage, while few-shot examples push Jev ahead on playbook actions.
An independent five-evaluation snapshot across 23 decision models places Liquid d1-ultra first at 65.2, Jev 1.14 at 57.1, and OpenAI’s GPT-6 Luna Decisions at 48.5.
Technical breakdowns of open decision architectures explain how non-causal readout layers replace token-by-token decoding with one parallel forward pass.
A new 300-question benchmark shows Jev and OpenAI Decisions drop sharply when answers require current web facts, underscoring the boundary between state-only and search-backed calls.
Research from Shanghai AI Lab shows a compact 4-billion-parameter decision model can rival or exceed larger hosted System One APIs on targeted evaluation suites.
AI eval specialist Hamel Husain argues that judging decision models at a default 0.5 cut-off hides their biggest advantage: tunable precision-recall and escalation bands.
A Towards Data Science evaluation tests seven open-weight decision models against a generative LLM on real banking intent routing, highlighting accuracy and self-hosting trade-offs.
Two October 7 technical evaluations compare eight early decision models and probe when fast single-pass System One models match or trail non-reasoning LLMs.
An October 3 arXiv paper identifies a new adversarial attack surface unique to decision models, showing how option wording and ordering can steer fixed-schema probabilities.
Late-September arXiv preprints put TypeSafe’s calibration claims under mathematical scrutiny, testing logical coherence, framing sensitivity, and deferral thresholds.