Our summary
The creators of a workout app tested Jev, an AI tool that chooses from options instead of writing answers. They wanted to check coaching messages and replies, then find out whether Jev could replace software already running on a user's phone.
The team checked whether messages concerned exercise, needed more thought, or contained instructions trying to mislead the assistant. They also checked drafted replies against workout records. Separately, they compared Jev with their existing software for identifying muscle groups from a person's workout request.
Jev did well in the reported message and reply checks, but lost the muscle-group comparison overall. The app kept its existing phone-based tool. These results describe one team's tests, with different examples and languages for each task; they do not establish that Jev works equally well everywhere.
Key takeaways
- The app tested specific choices rather than asking Jev to write coaching advice.
- Message-topic checks covered twenty languages; the other checks used different test sets.
- The existing phone-based tool beat Jev overall at identifying requested muscle groups.