What it does
The whole experiment cost about $1.13. The write-up also records the frontier run being cut short by an OpenRouter HTTP 402 cost reservation.
Benchmarks & research
jev-bench
An independent benchmark of Jev 1.13 against a cheap and a frontier LLM on SMS spam and Banking77, measuring accuracy, ECE, Brier, latency and cost.
Bookmark: jev-bench
Keep this in your collection.
Leave a noteWhat would you try with this?
: jev-bench
Only you can see your notes.
Screenshot unavailable. Open the experiment ↗
How you can use it
You can use this project to test whether specialized software is cheaper or faster than general artificial intelligence for sorting messages. Start by collecting sample messages with known correct categories, like unwanted spam or banking questions. A developer will need to adapt the benchmark scripts to test your own data.
Your developer will need an access key from an OpenRouter account to send the messages to external AI services. Keep in mind that message contents travel to outside servers, so do not include private personal details. Testing can also pause or stop early if your account balance runs out.