Our summary
A follow-up such as changing a chart’s dates can require several connected choices. Isotopes AI tested whether its app could answer from data already in the browser or needed a larger agent team. The company compared four model versions on eighty-eight scenarios, repeating the same scenarios three times.
The app kept a confidence cutoff of 0.50 and checked whether decisions preserved the requested conditions. Jev 1.13 passed 100 percent, Jev 1.14 preview 93.6 percent, OpenAI Decisions 68.2 percent and Microsoft Decision 1 62.1 percent. Those are the app’s requirement and routing checks, not a general ranking of intelligence.
Median decision-and-composition times were about 0.81, 0.84, 1.55 and 2.71 seconds in that order. They exclude final routing, execution and later fallback work. Early failures can shorten a run, so the article also compares the same thirty-eight scenarios that succeeded for every model. The private suite covers one application.
Key takeaways
- Keep application rules fixed and check that every requested filter survives the model’s decisions.
- A confident answer can still omit a date restriction; a confidence cutoff is not a correctness test.
- Compare the same successful cases as well as all attempts, and separate stage timing from total user wait.