Our summary
Jev is an AI tool that chooses from options rather than writing an answer. People use these tools to sort incoming messages or make automated workflow decisions. This paper tests if these tools actually follow the provided rules or just guess based on familiar labels.
The authors swapped the definitions attached to labels like yes and no across 1,200 test questions. They checked if the software still picked the correct definition or if it blindly followed the normal meaning of the label. They compared these results against neutral labels like numbers.
This guide is useful for people designing automated choices. The study only covered English questions. The authors suggest randomizing labels during training to fix this issue, but they did not test that idea. Hosted tools act as hidden systems, so the exact internal causes remain unknown.
Key takeaways
- Swapping definitions for familiar labels caused the tested AI models to frequently change their final decisions.
- Using neutral labels like numbers or random letters helped the software follow the actual instructions better.
- The open Laya model's accuracy dropped sharply, while the hosted Jev model showed a smaller decline.