Parallel Open-Sources PWS-Decisions to Test Where Static Decision Models Fall Short
A new 300-question benchmark shows Jev and OpenAI Decisions drop sharply when answers require current web facts, underscoring the boundary between state-only and search-backed calls.
Parallel open-sourced PWS-Decisions on November 1, releasing a 300-question evaluation suite designed specifically to test decision questions whose answers depend on live facts from the open web—such as whether a vendor currently supports SSO or whether two company records refer to the same entity.
Sharp drop when evidence is missing from the prompt
In Parallel’s evaluation, state-only decision models performed strongly when all facts sat inside the input payload (scoring 88.3% for TypeSafe Jev and 90.0% for OpenAI Decisions on a 231-question JevBench set), but plunged on PWS-Decisions when called natively without supplied web context: OpenAI Decisions dropped 2.2x to 41.7%, and Jev fell 2.6x to 33.4%. Parallel’s own web-searching Decisions API scored 76.2% on the same set.
Drawing the right architectural boundary
While Parallel builds a competing web-grounded API—a commercial bias the authors acknowledge directly—the benchmark clarifies a vital engineering rule for Jev adopters. System One models are engineered to evaluate supplied state, not to act as a live encyclopedia. When a workflow depends on external or rapidly changing facts, teams either need a retrieval step before calling Jev or a search-integrated decision endpoint, and should factor retrieval latency and search API fees into total cost per decision.