Seven Open Decision Models Tested Against a Generative LLM on Banking Request Routing
A Towards Data Science evaluation tests seven open-weight decision models against a generative LLM on real banking intent routing, highlighting accuracy and self-hosting trade-offs.
An October 15 benchmark published in Towards Data Science put the rapidly growing wave of open-weight Jev alternatives to the test on a classic regulated-industry task: routing incoming banking customer requests across fine-grained intent categories.
Testing open decision models against a generative baseline
The study compared seven open-weight decision models against a standard generative LLM baseline, evaluating not only top-1 routing accuracy but also latency stability and operational practicality for teams that cannot send sensitive banking queries to external hosted APIs.
Across the seven open entrants, results varied sharply by parameter count and post-training quality. While the best open decision models approached generative accuracy at a fraction of the latency, smaller or poorly calibrated checkpoints struggled on closely worded financial intents—reinforcing that the “decision model” label covers a wide quality spectrum.
Takeaways for regulated enterprises
For banks and fintechs exploring System One architectures, the findings highlight a practical trade-off: self-hosted open models eliminate third-party data transfer and per-token vendor lock-in, but demand careful GPU provisioning and calibration auditing before replacing either hosted Jev or established supervised classifiers.