JevMade

Sign in
← Back to guides

JevMade field notes / Three-way benchmark

Compare Jev, Databricks ai_decide and OpenAI Decisions on support chats

Bev Manz of Advancing Analytics tests Jev, Databricks ai_decide and OpenAI's Decisions API on 300 customer-support chats, with and without worked examples in the prompt.

Original by Bev ManzEvaluation

Listen to this guide

JevMade’s plain-English explanation

Expanded guide unavailable: the complete source text could not be retrieved.

0:00 /

Our summary

Bev Manz at Advancing Analytics published this test on 13 November 2026. It uses the ABCD dataset of role-played clothing-store support chats and asks each service to pick a broad intent, an exact case and the agent's next action, on 300 unseen chats.

On broad intent all three score about the same. On the exact case, Jev (0.79) and OpenAI Decisions (0.79) lead ai_decide (0.70). Adding the five most similar labelled chats lifts Jev's next-action score from 0.56 to 0.71, against 0.63 for OpenAI. The author also finds ai_decide behaves almost like Databricks' open OpenJev model.

It is one role-played dataset with some noisy labels, so treat it as a template for your own test. The author tuned the questions against Jev's mistakes, which may favour Jev, and speed was not measured like for like. The ai_decide identification is an inference, not a Databricks statement.

Key takeaways

  1. Test decision services on your own fine-grained questions, because easy broad-intent calls did not separate them here.
  2. Adding a few similar labelled examples to each request was the biggest single gain for Jev in this test.
  3. Treat Databricks ai_decide as a separate model from Jev, not as Jev under another name.

This is one practitioner's test on one dataset, and its code is on GitHub. JevMade did not rerun it. The author says tuning favoured Jev, and the prices and speeds quoted are the author's own figures from November 2026, so check current pricing before relying on them.

Advancing Analytics · Original published

Read the original guide Opens the author’s site in a new tab.

Your note

Only you can see this. Clear the text and save to remove it.

Keep what you save

Sign in to keep bookmarks and private notes on every device. We email you a link; there is no password, and the same link creates your account.