Contrastive Search Explainer
Contrastive search is a decoding method for language generation that tries to preserve fluency while avoiding repetitive or bland continuations. It considers a small set of likely next tokens rather than sampling freely from the full distribution.
For each candidate, the method combines model confidence with a degeneration penalty. That penalty measures how similar the candidate’s hidden representation is to representations already in the generated context, discouraging repetitive states.
The result is a deterministic or nearly deterministic search that can produce coherent long-form text without the loops associated with some greedy methods. It does not require a second model when used in its standard form.
The quality depends on model representations, candidate-set size, and penalty weight. Too much contrast can avoid useful repetition or choose less natural wording. Comparisons should include diverse genres and both automatic and human evaluation.
Contrastive search balances a token's probability with a penalty for hidden-state similarity that can signal repetitive or degenerate continuation.
Key terms include contrastive search, candidate set, model confidence, degeneration penalty, hidden state, repetition, decoding and fluency.
The decoder starts from a small set of high-probability token candidates.
It favors candidates whose internal representations add novelty without sacrificing too much confidence.
Some repetition is necessary for names, definitions, structure, and stylistic consistency.
Candidate width and penalty strength should be tested across the intended writing tasks.
Explore more "Explainers"
Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.
