JevMade

Sign in
← Back to guides

JevMade field notes / Confidence and threshold tutorial

Decide when an AI choice needs a person

Flavio Copes shows how to sort support messages into automatic action, confirmation and human review, then compare confidence cutoffs with labeled examples and shadow logs.

Original by Flavio CopesEvaluation
Leave a noteWhat would you try with this? : Decide when an AI choice needs a person

Only you can see your notes.

Listen to this guide

JevMade’s plain-English explanation

0:00 /

Our summary

A support message can fit more than one team. Flavio Copes shows how Jev's confidence number can help software decide whether to assign a message, suggest a team for confirmation or leave it for a person. Confidence describes the model's answer probabilities, not a promise that its choice is right.

The example saves each chosen team, its probabilities and the team that actually handled the message. A separate script compares different cutoffs, showing how much work becomes automatic and how often those choices match the labels. Shadow logging gathers these comparisons while people keep making the real decisions.

The displayed results use twenty invented answers, so they cannot establish a working cutoff. Test with real examples and check again after changing the model or question. The article says exact confidence formulas are not fully published; TypeSafe's official documentation now explains those calculations for choices and ordered scales.

Key takeaways

  1. Choose separate cutoffs for separate actions. Sending a refund form is not the same action as issuing a refund.
  2. Measure both the share of work handled automatically and mistakes among those automatic choices. Invented rows teach the calculation, not real accuracy.
  3. Keep the full probabilities and model version. A low-confidence choice can sometimes map to a broader category, but that category can still be wrong.

The article and its code were read, not run. Its twenty-row table is explicitly made up; figures quoted from other tutorials were not independently reproduced. Correction checked October 4, 2026: TypeSafe's official Confidence documentation explains the exact calculations for choices and ordered scales, contrary to the article's statement. The original archive is unchanged. Example cutoffs are not validated safety or permission checks.

Flavio Copes · Original published

Read the original guide Opens the author’s site in a new tab.

Your note

Only you can see this. Clear the text and save to remove it.

Keep what you save

Sign in to keep bookmarks and private notes on every device. We email you a link; there is no password, and the same link creates your account.