Our summary
A publication or document system may need to sort material into named categories. This guide asks Jev to choose from a defined list rather than write an answer. Separate questions can check other properties, and the program keeps the estimates behind those judgments.
Start with clear label descriptions and examples that people have already labelled. Keep some examples separate from the ones used to adjust instructions or cutoffs. Compare mistakes, then decide which answers are clear enough for automatic handling and which need a person's review.
Success on one collection does not guarantee success on another. Missing or vague categories can force the wrong answer, even when it looks confident. Keep an unknown option, watch changing inputs, and examine mistakes among automatically accepted items as well as the review queue.
Key takeaways
- Give every label a clear boundary and include an unknown option when needed.
- Test with examples that were not used to tune the rules.
- Check mistakes among automatic decisions, not just how much work was automated.