Before you dive in
What you’ll find in the original
- Collapse consecutive generations with the same user message into one turn, then judge its final reply against the next distinct user message. Otherwise an internal tool step can be mistaken for a user reaction.
- A Noul returns the probability of true, without a separate confidence field. Keep that probability alongside the verdict, pin the model, and choose a threshold using examples from your own application.
- Page through observations, sort them chronologically, and use stable score identifiers and timestamps so reruns update the same score. Flush buffered events before the script exits.
Worth knowing
The example detects the user's expressed disagreement, not whether the assistant was objectively wrong. Its 0.7 threshold is an example policy, not a validated default. The code was source-reviewed, not run against a Langfuse account.