Benchmarks & research
Sys1Cal-v1 is a synthetic benchmark by Riccardo Porcedda that tests if decision models can accurately recover exact probabilities across different but equivalent question formats.
Only you can see your notes.
Screenshot unavailable. Open the experiment ↗
The fixed release contains 365 true-or-false questions queried through Noul, Choice, and Score formats. The author suggests that Jev hides an "I don't know" option when answering Choice questions, but this is an interpretation of the results rather than a proven internal mechanism of the model.
Researchers can use the provided dataset files to test how consistently a model assigns probabilities to the same underlying facts presented in different ways. The project includes tools to compare these results against exact mathematical answers.
To ensure your tests can be compared with others, use the pre-generated question files rather than creating new ones. You will need to provide your own access credentials if you want to test the live Jev service.