JevMade

Sign in
← Back to guides

JevMade field notes / Written guide

Testing Whether an AI Should Retry a Failed Task

Astronomer tests Jev and seven open models on failed data-pipeline tasks, asking how often each wrongly stops work that could recover.

Original by Kaxil Naik / AstronomerEvaluation
Leave a noteWhat would you try with this? : Testing Whether an AI Should Retry a Failed Task

Only you can see your notes.

Listen to this guide

JevMade’s plain-English explanation

0:00 /

Our summary

A retry can clear a temporary failure or waste more time. Astronomer tests AI used by Airflow, which organizes data-processing jobs, against four hundred errors. Written rules deal with familiar failures first. For the remaining failed jobs, models decide whether to retry or stop.

The confidence number is not interchangeable across models. At a 0.8 cutoff, Jev stops more recoverable errors than decider-0.8b, an open model. Comparing equal total stop counts narrows that gap on this small set of recoverable cases. Confidence estimates how sure an answer looks, not a shared standard of certainty.

AI raters supplied the grading answers, the test sets are not public, and only forty-one recoverable errors were scored. Astronomer also finds decisions changing when answers are reworded. Its safer starting point logs proposed stops but permits retries, with each model's cutoff tuned separately.

Key takeaways

  1. Use examples with known outcomes to set each model's cutoff, rather than sharing one confidence number.
  2. Count the recoverable tasks that were wrongly stopped, not only fast or cheap decisions.
  3. Log proposed stops first while continuing to allow retries.

This is an Astronomer engineering comparison with AI-supplied grading answers and non-public test sets. Forty-one recoverable errors are too few to establish a dependable winner.

astronomer.io · Original published

Read the original guide Opens the author’s site in a new tab.

Your note

Only you can see this. Clear the text and save to remove it.

Keep what you save

Sign in to keep bookmarks and private notes on every device. We email you a link; there is no password, and the same link creates your account.