Our summary
This guide tests a way to check if an AI assistant's answers are actually backed by the documents it read. People use this method to catch mistakes when software writes summaries or answers questions based on a specific text.
The process first looks for the exact quoted words in the original document. If found, an AI tool named Jev chooses whether the surrounding sentences support the claim, contradict it, or say nothing at all. Jev picks from options rather than writing a new answer.
This method is useful for fact-checkers, but it has strict limits. In these tests, the software required exact word matches and rejected quotes with minor typos. A supported quote does not prove the original document is true, and uncertain answers still need a person to check them.
Key takeaways
- Find the exact quoted text in the original document before judging if it supports the claim.
- Treat a document that says nothing about a claim differently than one that directly contradicts it.
- Ask a person to review the results if the software is not highly confident in its choice.