LLM Evaluation & Reliability
Testing LLM judges as measuring instruments
We test LLM judges as measuring instruments: whether verdicts can be read as specified, whether identical answers receive non-tie judgments, and how accurately a target is selected under declared criteria.
LLM Judges Have Dark Current · arXiv preprint · 2026
Joint research with Mitsubishi Heavy Industries, Research & Innovation Center
Measurement stages
- Output validity
- Can the declared parser read the verdict?
- Dark current
- Does the judge prefer one exact duplicate?
- Target-selection utility
- Does it select the specified target?

