Our summary
A retry can clear a temporary failure or waste more time. Astronomer tests AI used by Airflow, which organizes data-processing jobs, against four hundred errors. Written rules deal with familiar failures first. For the remaining failed jobs, models decide whether to retry or stop.
The confidence number is not interchangeable across models. At a 0.8 cutoff, Jev stops more recoverable errors than decider-0.8b, an open model. Comparing equal total stop counts narrows that gap on this small set of recoverable cases. Confidence estimates how sure an answer looks, not a shared standard of certainty.
AI raters supplied the grading answers, the test sets are not public, and only forty-one recoverable errors were scored. Astronomer also finds decisions changing when answers are reworded. Its safer starting point logs proposed stops but permits retries, with each model's cutoff tuned separately.
Key takeaways
- Use examples with known outcomes to set each model's cutoff, rather than sharing one confidence number.
- Count the recoverable tasks that were wrongly stopped, not only fast or cheap decisions.
- Log proposed stops first while continuing to allow retries.