Benchmarks & research
Datadog's Agent Observability team shows Jev doing double duty in their eval stack: notebooks score production agent spans as they arrive through online evals and run the same focused questions offline, mapping typed answers onto Datadog metrics.
Only you can see your notes.
Screenshot unavailable. Open the experiment ↗
The questions stay deliberately narrow — whether claims are supported, what failure occurred, how severe the impact is — and the post is candid about what Jev shouldn't be asked, like arithmetic or dates, where a parser belongs. Code ships as online and offline notebooks in DataDog's llm-observability repo. It's judge-style evaluation moving into the live path rather than after execution.
For your customer-facing AI, test questions it cannot answer from the rules it was given. Start with the airline example, where some answers are missing from the policies. A developer can adapt its checks for your replies. Compare the AI judge's answers with human reviews before relying on it. Judging sends the examples to TypeSafe's outside AI service.