JevMade

Sign in
← Back to experiments

Benchmarks & research

BeelineBench

Run decision models like Jev through seven multi-step puzzles and see how directly each one reaches the goal instead of wandering.

Bookmark: BeelineBench Keep this in your collection.
Leave a noteWhat would you try with this? : BeelineBench

Only you can see your notes.

Source screenshot of BeelineBench
SOURCE SCREENSHOTFull screenshot ↗

What it does

Each step sends the whole frontier of open states as one choice question, capped at 255 options, and the score is the shortest path length divided by the nodes explored. The README notes that classic search such as A* will almost certainly stay more efficient, and that Jev's answers vary between identical calls.

How you can use it

Clone the repository and install it with uv, then run the baseline command to see the heuristic and oracle scores without sending any requests.

Run the mini benchmark to send paid requests to the configured models, then run the report command to see the scores.

You need an API key in a .env file, and the full runs cost money because every explored node is a request.

Your note

Only you can see this. Clear the text and save to remove it.

Keep what you save

Sign in to keep bookmarks and private notes on every device. We email you a link; there is no password, and the same link creates your account.