Our summary
About 16,000 calls underpin Aman Kumar's comparison of Jev with small language models on public text datasets and private business workflows. He asks whether Jev should replace a whole classifier or filter just the cases that look clearly positive or clearly negative.
His proposed filter takes cases from both ends of the probability scale. Very high and very low scores can use a cheaper route, while the uncertain middle goes to another model. Test the cutoffs on your own examples; a score that worked elsewhere is not a ready-made cutoff.
Results differ across tasks. Short text and clear labels fare better than long documents or decisions requiring detailed business context. Kumar shares code and public cases, but reports the production results himself. This supports a selective filter, not a promise that a confident answer is correct.
Key takeaways
- Try the comparison on your own task before replacing the classifier you already use.
- Possible filter cutoffs sit at both ends: test very low scores as well as very high ones.
- Leave the uncertain cases on the more capable route rather than sending every decision through Jev.