Support-Chat Benchmark Pits Jev Against Databricks ai_decide and OpenAI Decisions
An independent test on 300 customer-support chats finds Jev and OpenAI Decisions tied on zero-shot triage, while few-shot examples push Jev ahead on playbook actions.
An independent benchmark published on November 13 by Bev Manz at Advancing Analytics tested three major hosted decision endpoints—TypeSafe’s Jev, OpenAI’s Decisions API, and Databricks’ SQL-native ai_decide—against 300 unseen customer-support conversations from the Action-Based Conversations Dataset (ABCD).
Zero-shot parity, then Jev pulls ahead with examples
| Endpoint | Broad intent (10) | Exact case (54) | Next action (30) | Next action + 5 examples |
|---|---|---|---|---|
| TypeSafe Jev | 0.87 | 0.79 | 0.56 | 0.71 |
| OpenAI Decisions | 0.88 | 0.79 | 0.57 | 0.63 |
| Databricks ai_decide | 0.86 | 0.70 | 0.39 | 0.43 |
On easy calls all three are level. On fine-grained tasks, Jev and OpenAI’s Decisions API stay level with each other and well ahead of ai_decide. Add examples to the prompt and Jev pulls ahead on the hardest job.
Bev Manz, Advancing Analytics
Speed, cost, and data perimeter
Over the public internet, Jev averaged ~230 ms per call at $0.06–$0.10 per 1,000 calls, compared with ~300 ms and $0.11–$0.20 for OpenAI Decisions. Databricks ai_decide ran all 300 chats in 19 seconds as one batch query (~$0.55 per 1,000 calls) while keeping customer rows strictly inside the Databricks workspace.

Cross-agreement probes also found ai_decide matching community model OpenJev on 100% of yes/no questions with near-identical probabilities—consistent with Databricks’ documentation noting Apache 2.0 underlying weights rather than hosted Jev.