The benchmark mixes tasks adapted from public NLP datasets with synthetic ones, and records dataset revisions and inference settings with each result. The Japanese and vision evaluation datasets are private and need authorized access to run.
How you can use it
Clone the repository and install the Python evaluator with uv, then run a smoke test with the dummy adapter to check the pipeline works.
To evaluate a real model, pick an adapter such as TypeSafe/Jev, supply the needed API key or model weights, and run the full benchmark scope.
Open the Hugging Face leaderboard to compare published models, or submit your own validated results as a dataset pull request.