A reproducible pipeline for ingesting ESCO occupations and skills, applying explicit automation and augmentation rubrics, and comparing batched LLM scoring with Jev typed judgments.
Original by Joris de VreedeEvaluationRepository READMESource reviewed
Before you dive in
What you’ll find in the original
Start from ESCO occupation-skill relations and preserve taxonomy links before adding model judgments.
Define anchored 1–10 rubrics for automation risk and AI leverage so scores have interpretable meanings.
Use checkpointing, resume behavior, retries, and configurable batches when scoring a large taxonomy; the Jev experiment stores typed answers separately.
Worth knowing
The README names Gemini Flash for the main scorer and jev-1.13.0 for the typed-judgment experiment; results were not rerun.