JevMade

Sign in
← Back to Jev News Benchmarks & Research

Parallel Open-Sources PWS-Decisions to Test Where Static Decision Models Fall Short

A new 300-question benchmark shows Jev and OpenAI Decisions drop sharply when answers require current web facts, underscoring the boundary between state-only and search-backed calls.

1 min read By JevMade Source: Parallel Blog ↗
Editorial collage of a brass nautical sextant and cutaway globe wired with copper to an orange magnetic compass dial.

Parallel open-sourced PWS-Decisions on November 1, releasing a 300-question evaluation suite designed specifically to test decision questions whose answers depend on live facts from the open web—such as whether a vendor currently supports SSO or whether two company records refer to the same entity.

Sharp drop when evidence is missing from the prompt

In Parallel’s evaluation, state-only decision models performed strongly when all facts sat inside the input payload (scoring 88.3% for TypeSafe Jev and 90.0% for OpenAI Decisions on a 231-question JevBench set), but plunged on PWS-Decisions when called natively without supplied web context: OpenAI Decisions dropped 2.2x to 41.7%, and Jev fell 2.6x to 33.4%. Parallel’s own web-searching Decisions API scored 76.2% on the same set.

Drawing the right architectural boundary

While Parallel builds a competing web-grounded API—a commercial bias the authors acknowledge directly—the benchmark clarifies a vital engineering rule for Jev adopters. System One models are engineered to evaluate supplied state, not to act as a live encyclopedia. When a workflow depends on external or rapidly changing facts, teams either need a retrieval step before calling Jev or a search-integrated decision endpoint, and should factor retrieval latency and search API fees into total cost per decision.

Keep what you save

Sign in to keep bookmarks and private notes on every device. We email you a link; there is no password, and the same link creates your account.