Our summary
The author builds a way to check if a person disagrees with an AI assistant. It uses Jev, an AI tool that chooses from options rather than writing an answer. This helps software builders automatically review how well their automated conversations are going.
The method groups all the hidden steps an assistant takes into a single turn. It then compares the assistant's final reply to the person's next message. Jev decides if the person is arguing and gives a percentage showing how likely that choice is true.
This is useful for developers tracking software performance over time. The tool only detects if a person sounds like they disagree, not if the assistant was actually wrong. Builders must choose their own cutoff scores and lock in the specific software version they use.
Key takeaways
- Group hidden software steps together so you only judge the final reply against the next user message.
- Save the percentage score alongside the final yes or no decision to help adjust your settings later.
- Use consistent names and timestamps for your scores so running the test again updates the same record.