JevMade

Sign in
← Back to guides

JevMade field notes / Written guide

Build Your Own Decision Model

Nish Tahir shows how to turn a language model into a multiple-choice decision tool, then test and adjust its confidence estimates.

Original by Nish TahirClassification
Bookmark: Build Your Own Decision Model Keep this in your collection.
Leave a noteWhat would you try with this? : Build Your Own Decision Model

Only you can see your notes.

Listen to this guide

JevMade’s plain-English explanation

0:00 /

Our summary

What if an ordinary language model chose an answer from a short list instead of writing a reply? Nish Tahir's tutorial explores that decision-model idea, also used by Jev. The model scores the allowed answers in one pass rather than producing words one at a time.

Tahir uses Qwen3-1.7B and assigns a letter to each possible answer. His program reads the scores for those letters and ignores the other words the model could produce. He then tests the answers, trains on examples and adjusts how sure the answers appear.

The untuned model looks too sure of itself: even a high score can belong to a wrong answer. The companion code covers testing, training and confidence adjustment. Tahir reports his own results, though, and this remains a tutorial rather than TypeSafe's model.

Key takeaways

  1. Use the scores for allowed answers without asking the model to write a response.
  2. Before trusting a score, check whether the answers that look certain are right.
  3. Train to change the answers; adjust confidence to change how sure they look. These are separate steps.

This is an independent tutorial, not a TypeSafe release or a ready-made replacement for Jev. Tahir reports his own results on CommonsenseQA, a multiple-choice question set; they have not been reproduced here.

nishtahir.com · Original published

Read the original guide Opens the author’s site in a new tab.

Your note

Only you can see this. Clear the text and save to remove it.

Keep what you save

Sign in to keep bookmarks and private notes on every device. We email you a link; there is no password, and the same link creates your account.