Skip to main content
Usami Laboratory
Research explainers

Testing LLM judges as measuring instruments

We test LLM judges as measuring instruments: whether verdicts can be read as specified, whether identical answers receive non-tie judgments, and how accurately a target is selected under declared criteria.

arXiv · Preprint · 2026

Joint research with Mitsubishi Heavy Industries, Research & Innovation Center

Illustrative research overview of output validity, identical-answer preferences, and target selection by LLM judges.
English research overview, about two minutes. Output validity, identical-answer preferences, and target selection are assessed separately; the animation is illustrative.

The problem

LLM judges are used to compare models and assess answer quality. Returning a tie for identical answers and selecting a target among different answers are distinct properties. The study pairs tests of both under the same declared evaluation settings.

The research idea

Each configuration fixes the model profile, prompt, decoding conditions, and verdict parser. We first check output validity, then measure non-tie preferences between byte-identical answers as dark current. Matched target-selection tests are evaluated by input stratum and grading criterion.

Output validity
Can the declared parser read the verdict?
Dark current
Does the judge prefer one exact duplicate?
Target-selection utility
Does it select the specified target?

What was evaluated

On 102 prespecified RewardBench 2 items, configurations with zero non-tie clone preferences still had lower confidence bounds for target-selection utility below the declared threshold under the strict criterion. A separate fixed 485-item extension compared aggregate and task-subset results, showing a different pattern in precise instruction-following tasks.

Considering applications

For answer ranking and quality assessment, report identical-answer behavior alongside target-selection accuracy, including results for the intended task subsets. The public repository provides saved verdict records, analysis evidence, and verification and reproduction instructions.

Original research

LLM Judges Have Dark Current: Stratified Null and Utility Tests for LLM Judge Reliability

Usami, Hiroyasu, Hara, Keisuke, Tsuboi, Ayato, Matsuda, Naohiko

arXiv, 2026 · arXiv:2606.15610v2 [cs.CL]; revised October 13, 2026 (JST); 21 pages, 4 figures; code and supporting records available on GitHub

Papers citing this study

The following studies refer to the measurement framework discussed in the first version (v1).

A Calibrated Instrument for Measuring How Inference Optimizations Affect Output Quality

Jerry Kaplan

Stanford University

A study measuring how inference optimizations affect output quality. It cites Judge Datasheet as prior work characterizing an LLM evaluator as a measurement instrument.

Read the citing paper

Three Ways Classical Test Theory Misleads for LLM Judges

Louis Yiven Zhu

University of Oxford

A study examining measurement design when classical test theory is applied to LLM judges. It cites the Judge Datasheet framework among measurement-theoretic approaches to judge evaluation.

Read the citing paper

The First Token Is Not the Verdict: Hidden Costs of Reading LLM Judges Without Generating

Gnaneswar Villuri, Hashmath Shaik, Alex Doboli

Stony Brook University

In a paper submitted to JUDGe — Can We Trust the Judge? @ NeurIPS 2026, this work is cited as an example of a judge audit that generates text and parses the verdict. The paper examines how that procedure differs from first-token readout.

Read the citing paper

External coverage

· Jake Handy · AI Weekly Update

AI Research of the Week

The June 22, 2026 edition featured the paper and its authors in AI Research of the Week.

Read the article