CaLLM is a post hoc multicalibration procedure designed to mitigate position, verbosity, and model-family biases in pairwise language model evaluations.
Large language models deployed to judge pairwise text comparisons frequently display systematic biases favoring specific presentation positions, longer outputs, or their own model families. CaLLM applies multicalibration over input representations—specifically PCA-reduced difference embeddings and initial confidence scores—to adjust predictions across identifiable error patterns. In reported evaluations across five preference datasets, CaLLM reduced documented bias metrics while simultaneously lowering log loss and raising judgment accuracy. However, theoretical protections apply only to error structures captured by the representation rather than all unknown biases, and labeled calibration data remains necessary.
How you can use it
If you use AI to compare two answers, borrow this test before trusting its preferences. Keep human-labelled examples apart from training and check whether order, length or model family tilts the verdict. The released code can adjust confidence using those examples, but cannot remove biases its input representation fails to capture.
