Our summary
Bev Manz at Advancing Analytics published this test on 13 November 2026. It uses the ABCD dataset of role-played clothing-store support chats and asks each service to pick a broad intent, an exact case and the agent's next action, on 300 unseen chats.
On broad intent all three score about the same. On the exact case, Jev (0.79) and OpenAI Decisions (0.79) lead ai_decide (0.70). Adding the five most similar labelled chats lifts Jev's next-action score from 0.56 to 0.71, against 0.63 for OpenAI. The author also finds ai_decide behaves almost like Databricks' open OpenJev model.
It is one role-played dataset with some noisy labels, so treat it as a template for your own test. The author tuned the questions against Jev's mistakes, which may favour Jev, and speed was not measured like for like. The ai_decide identification is an inference, not a Databricks statement.
Key takeaways
- Test decision services on your own fine-grained questions, because easy broad-intent calls did not separate them here.
- Adding a few similar labelled examples to each request was the biggest single gain for Jev in this test.
- Treat Databricks ai_decide as a separate model from Jev, not as Jev under another name.
