Our summary
Many apps only need to pick one option, such as which support team should handle a customer's message. Lambert Leong tested seven open projects that score a list of options directly, as Jev does, against Qwen3-0.6B, a small model that writes its answer as text. He used BANKING77, a public set of bank requests.
In one round, every message had the same eight routes. In the other, each message came with four shuffled routes, one of them right. Kev was correct 98 percent of the time in both rounds. Verdict reached 89 percent in the shuffled round, while the text-writing model reached 83 and then 66 percent.
Nine of the text model's shuffled-round answers failed only because valid output came wrapped in extra formatting that a strict checker refused. Median response times were about 18 milliseconds for Verdict, 130 for Kev and 340 for Qwen. This was one small banking task against a tiny model, so Leong treats the results as leads to test.
Key takeaways
- Kev picked the right route 98 percent of the time, whether the list of routes stayed the same or changed with each message.
- Handling a new list is not the same as choosing well from it: NanoJev accepted changing lists but got only 17 percent right.
- A model that writes its answer can fail on formatting alone: nine of Qwen's answers were refused because they came wrapped in extra symbols.