Examining the Instrument Behind LLM Evaluation

The problem

LLM-as-a-Judge is used to compare models and assess generated answers. Even when answers have equal quality, presentation order or surface form can affect the verdict. This study starts by examining the evaluator itself.

The research idea

Drawing on dark current in a sensor with no incident light, we examine judgments when no quality difference is present. Empty inputs, identical answers, quality-preserving surface changes and reversed answer order are combined with a controlled quality ladder. The Judge Datasheet organizes these measurements into an instrument profile.

Dark current

Judgments without a quality difference

Cross-sensitivity

Responses to quality-preserving surface changes

Position bias

Effects of answer presentation order

Quality sensitivity

Discrimination on a controlled quality ladder

What was evaluated

Three public models are evaluated for judgments under null conditions, responses to surface form and presentation order, and sensitivity to controlled quality differences. Experiments also examine how prompts favoring ties change both bias measures and sensitivity. The design examines distinct judge properties rather than collapsing them into one accuracy score.

Considering applications

For answer ranking and quality assessment with LLMs, the study provides a way to examine bias, quality sensitivity and tie criteria separately. It motivates treating the model, prompt and verdict readout as one evaluation instrument, with explicit measurement conditions.