JevMade Sign in
← Back to guides

JevMade field notes / Method and evaluation guide

Helping an AI assistant make reliable choices from a list

You will learn how the AnyJev project helps an AI assistant choose from options instead of writing text. It tests ways to measure how confident the AI is, so people can safely automate routine choices.

Original by Jiamu Zhang, Tianze Yang, Yucheng Shi, and Liang WuEvaluation

Listen to this guide

JevMade’s plain-English explanation

0:00 /

Our summary

AnyJev is a method for turning a standard AI assistant into a tool that chooses from options rather than writing an answer. People use this to sort incoming messages or route tasks. It helps the software provide a trustworthy confidence score so users can automate safe choices and send uncertain ones to humans.

The software tests different ways to improve these choices. First, it shuffles the order of the options to prevent the AI from just picking the first one. Then, it uses a few hundred examples to build a small add-on that looks at the AI's internal process to give a final, reliable answer.

This guide is useful for developers building automated sorting tools. However, these methods cannot help an AI that simply does not know the answer. The small add-on only works for one specific question at a time, and the current software can only handle up to twenty-six options.

Key takeaways

  1. Shuffle the order of choices to stop the AI from favoring the first option it sees.
  2. Measure how often the software makes safe choices, rather than just looking at overall correctness.
  3. A confidence score cannot fix an AI assistant that lacks the knowledge to solve the problem.

In the project's tests, the automation rates at a five percent risk level varied highly because they only used 300 examples. Also, the accuracy scores only measure how often the tool agreed with a larger AI, not a human expert.

GitHub repository

Read the original guide Opens the author’s site in a new tab.

Keep this for later

Sign in to bookmark experiments, guides and videos, and keep notes only you can see.

Continue to sign in

We’ll bring you back to this listing.