JevMade Sign in
← Back to guides

JevMade field notes / Model and benchmark report

Testing an AI tool that chooses options without generating text

This guide explores Von, a local AI tool that picks from predefined choices instead of writing new text. You will learn how to test if an AI changes its answer just because the options are shuffled.

Original by wfzyxEvaluation

Listen to this guide

JevMade’s plain-English explanation

0:00 /

Our summary

Von is a downloadable AI tool that categorizes information or chooses from a list of options instead of writing out a response. People use this type of software to quickly sort incoming messages, route support tickets, or control simple game actions without waiting for text generation.

The author tests how the software handles shuffled choices. In many systems, changing the order of the options changes the final answer. Von attempts to fix this by forcing the software to look at each option completely separately from the others before making a final decision.

This guide helps developers who need fast, consistent sorting tools. The author separates language tasks from interactive game tests because speed and software design affect game results differently. The reported perfect consistency scores come directly from the creator and were not independently verified in this review.

Key takeaways

  1. Shuffle identical options and measure how often the software changes its answer to test consistency.
  2. Separate test results into easy, standard, and hard groups instead of hiding difficult cases in an average.
  3. Report language accuracy and interactive game outcomes separately because software speed affects interactive tasks.

The author reports that Von achieved perfect consistency when shuffling hard-tier options, compared to a 49.5% failure rate in the baseline. All comparisons, including those to Jev—an AI tool that chooses from options rather than writing answers—were not independently rerun.

GitHub repository

Read the original guide Opens the author’s site in a new tab.

Keep this for later

Sign in to bookmark experiments, guides and videos, and keep notes only you can see.

Continue to sign in

We’ll bring you back to this listing.