Contrastive Decoding Explainer
Contrastive decoding is a text-generation method that compares the predictions of a stronger expert language model with those of a weaker amateur model. It favors tokens the expert considers plausible but the amateur does not favor as strongly.
The intuition is that generic or undesirable tendencies may be shared by both models, while the expert’s additional capability appears in their disagreement. A plausibility constraint prevents the method from choosing tokens the expert itself considers unlikely.
This method changes inference rather than retraining the expert. Research has explored it as a way to improve open-ended generation, factuality, and reasoning by sharpening distinctions in next-token scores.
Performance depends on the relationship between the two models, the contrast strength, and the task. Running both models adds computation, and disagreement is not always evidence of quality. Evaluation should compare accuracy, fluency, latency, and failure patterns.
Contrastive decoding chooses tokens by comparing a stronger language model with a weaker model and favoring options that reflect the stronger model's expertise.
Key terms include contrastive decoding, expert model, amateur model, next token, probability, plausibility constraint, inference and contrast strength.
Candidate tokens are rewarded when the expert prefers them more strongly than the amateur.
A plausibility filter keeps the contrast within choices supported by the expert.
The choice of amateur model and contrast settings can help or harm different tasks.
Two-model inference also introduces extra latency and computation that must be measured.
Explore more "Explainers"
Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.
