JevMade

Sign in
← Back to guides

JevMade field notes / Written article

Check whether a chat answer is complete and supported

scottgal's twelve-case support-chat probe checks whether an answer is present, supported by evidence and clear, separating a missing answer from a wrong one.

Original by scottgal / mostlylucidEvaluation

Listen to this guide

JevMade’s plain-English explanation

0:00 /

Our summary

An answer can name a refund status and still get that status wrong. It can also sound polite while never giving a status at all. This article treats coverage, evidence support and communication as separate checks, so the system knows whether to rewrite a claim or collect missing information.

Exact IDs and statuses can be compared in code. For indirect wording, a language-inference model asks whether the evidence supports, contradicts or says too little about a claim. A local Nimble decision model can also choose the next response, such as revising the answer or asking another question.

The author's small probe uses twelve made-up support cases, not live customer traffic. Models sometimes confuse missing evidence with contradiction, and the suggested cutoffs are not calibrated. Nimble here runs locally through Ollama; it is not a hosted Jev test. None of the labels grants permission to issue a refund.

Key takeaways

  1. Check presence and truth separately: an incorrect status still answers the question, while a polite non-answer may not.
  2. In a support check, supply evidence first and the claim second. Too little evidence differs from a contradiction.
  3. Let model judgments guide how prose is revised, while code retains the authoritative transaction state and permission to act.

This is scottgal's .NET experiment with author-reported synthetic results. Its local-model timings perform different work and do not establish a general speed ranking or production review accuracy.

mostlylucid · Original published

Read the original guide Opens the author’s site in a new tab.

Keep this for later

Sign in to bookmark experiments, guides and videos, and keep notes only you can see.

Continue to sign in

We’ll bring you back to this listing.