LLM-as-a-Judge Explainer
LLM-as-a-judge uses a language model to evaluate other outputs. The judge may assign scores, choose between responses, check criteria, or produce a critique using a rubric and sometimes a reference answer.
Automated judging can evaluate open-ended text more cheaply and quickly than large human studies. It is useful for development, regression testing, dataset filtering, and comparing many model variants.
A judge can favor its own style, longer answers, confident language, familiar model families, or the first or second position. It may also be fooled by content inside the response or make errors on tasks it cannot solve.
Reliable use includes separating system instructions from evaluated content, randomizing positions, calibrating against humans, using multiple judges or deterministic checks, measuring agreement, and auditing failures. Judge scores should be evidence, not an unquestioned ground truth.
LLM-as-a-judge uses a language model to score, rank, or critique outputs according to instructions or a rubric.
Key terms include judge model, rubric, reference answer, pairwise choice, pointwise score, position bias, self-preference, calibration, agreement and ground truth.
The judge receives the task, candidate output, and explicit criteria.
It returns a score, preference, or critique that can be aggregated across examples.
Style bias, prompt injection, weak task knowledge, and inconsistency can distort results.
Human calibration and independent checks define when automated judgments are trustworthy.
Explore more "Explainers"
Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.