JevMade

Sign in
← Back to Jev News Benchmarks & Research

Support-Chat Benchmark Pits Jev Against Databricks ai_decide and OpenAI Decisions

An independent test on 300 customer-support chats finds Jev and OpenAI Decisions tied on zero-shot triage, while few-shot examples push Jev ahead on playbook actions.

1 min read By JevMade Source: Advancing Analytics ↗
Editorial collage of a telephone switchboard operator plugging cables into three brass pressure gauges with an orange central needle.

An independent benchmark published on November 13 by Bev Manz at Advancing Analytics tested three major hosted decision endpoints—TypeSafe’s Jev, OpenAI’s Decisions API, and Databricks’ SQL-native ai_decide—against 300 unseen customer-support conversations from the Action-Based Conversations Dataset (ABCD).

Audio summary: Advancing Analytics’ support-chat comparison of Jev, Databricks ai_decide, and OpenAI Decisions.

Zero-shot parity, then Jev pulls ahead with examples

EndpointBroad intent (10)Exact case (54)Next action (30)Next action + 5 examples
TypeSafe Jev0.870.790.560.71
OpenAI Decisions0.880.790.570.63
Databricks ai_decide0.860.700.390.43
Accuracy across 300 unseen ABCD customer-support chats (majority-class baseline on next action: 0.21).

On easy calls all three are level. On fine-grained tasks, Jev and OpenAI’s Decisions API stay level with each other and well ahead of ai_decide. Add examples to the prompt and Jev pulls ahead on the hardest job.

Bev Manz, Advancing Analytics

Speed, cost, and data perimeter

Over the public internet, Jev averaged ~230 ms per call at $0.06–$0.10 per 1,000 calls, compared with ~300 ms and $0.11–$0.20 for OpenAI Decisions. Databricks ai_decide ran all 300 chats in 19 seconds as one batch query (~$0.55 per 1,000 calls) while keeping customer rows strictly inside the Databricks workspace.

Editorial collage of a telephone switchboard operator plugging cables into three brass pressure gauges with an orange central needle.

Cross-agreement probes also found ai_decide matching community model OpenJev on 100% of yes/no questions with near-identical probabilities—consistent with Databricks’ documentation noting Apache 2.0 underlying weights rather than hosted Jev.

Keep what you save

Sign in to keep bookmarks and private notes on every device. We email you a link; there is no password, and the same link creates your account.