Reinforcement Learning with Verifiable Rewards (RLVR) typically uses entropy for advantage shaping. The problem: entropy can’t distinguish “useful uncertainty” from “detrimental confusion.” A paper from a Chinese team — Contrastive Policy Optimization (CPO) — replaces entropy with token-level contrastive disagreement between reference-guided and vanilla sampling distributions. Results: 3-5 point improvements over PPO and GRPO on reasoning benchmarks.
The Entropy Problem in RLVR
Modern LLM post-training relies on RLVR — you have a verifier (e.g., checking math correctness) and a policy you optimize. Standard implementations use the probability distribution’s entropy as part of the advantage signal: higher entropy (more uncertainty) means more exploration.
But entropy is a blind signal. It can’t tell the difference between:
- Useful uncertainty — the model tries different solutions because the task is hard, but it’s heading in the right direction
- Detrimental confusion — the model is randomly sampling tokens because it has no idea what to do, drifting into bad regions
Entropy treats both identically and rewards exploration — even the kind that leads nowhere. This produces models that are “entropically diverse” but not necessarily correct.
CPO: Contrastive Disagreement as Advantage
The authors (Weiwen Xu, Jia Liu, Hou Pong Chan, Long Li — arXiv:2607.14614) propose Contrastive Policy Optimization (CPO). The core idea:
Instead of entropy, use token-level contrastive disagreement between a reference distribution (guided sampling — the model with a correctness hint) and vanilla sampling (model without context).
Here’s how it works:
# Pseudocode for CPO's core idea
def cpo_advantage(logits_ref, logits_vanilla, token_id):
# Reference distribution — model with correctness hint
p_ref = softmax(logits_ref)
# Vanilla distribution — no context
p_van = softmax(logits_vanilla)
# Contrastive disagreement: KL divergence between distributions
# High disagreement = model changes its mind when given context
# This is the signal: "this token needs correction"
disagreement = kl_divergence(p_ref, p_van)
# Advantage = disagreement * (reward - baseline)
# Larger opinion change + higher reward = stronger signal
advantage = disagreement * (token_reward - baseline)
return advantageThe logic: if adding correctness context drastically changes the model’s distribution on a given token, it means the model was “confused” without that context. The difference between p_ref and p_van measures how much the model changed its mind — and this is a better proxy for “this token needs fixing” than raw entropy.
Architecture
CPO requires no new weights or separate models. Everything happens at the forward pass level:
- Reference forward pass: model generates logits with additional context (e.g., prompt includes a hint about the correct path). Cheap — just one extra inference per step.
- Vanilla forward pass: model generates logits without context — standard rollout.
- Contrastive disagreement: for each token, KL divergence is computed between the two distributions. This becomes the advantage signal.
- Policy gradient: standard update — but with disagreement-based advantage, not entropy.
Key advantage: contrastive disagreement is correctness-aware. If the model has high entropy but both distributions (with and without context) agree — the model is simply uncertain but not confused. No reason to pull the policy in that direction. If distributions diverge — there’s a corrective signal.
Results
The authors tested CPO on reasoning benchmarks (GSM8K, MATH, ARC-Challenge) with models ranging from 1B to 8B parameters. Comparison with PPO and GRPO:
| Model | GRPO | PPO | CPO (entropy baseline) | CPO (full) |
|---|---|---|---|---|
| Qwen2.5-1.5B | 42.3 | 43.1 | 43.8 | 46.2 |
| Qwen2.5-7B | 58.7 | 59.2 | 60.1 | 63.4 |
| LLaMA-3.2-3B | 51.0 | 51.8 | 52.5 | 55.1 |
CPO (full) beats GRPO by 3-5 percentage points on every model. Importantly — CPO with entropy as baseline (no contrastive disagreement) performs worse than CPO full, confirming that contrastive disagreement is the source of the gain, not just changing the loss function.
What This Means for Engineers
If you’re doing post-training with RL (GRPO, PPO, RLOO) — CPO is a practical, low-cost modification. It requires no architecture changes or new datasets. Just add a second forward pass with a reference hint and replace the entropy bonus with contrastive disagreement.
Cost: one extra forward pass per step (~2x more compute per training step). But a 3-5 point gain on benchmarks is excellent ROI. For comparison: scaling a model from 3B to 7B gives ~7-8 points but costs 5x more flops at inference. CPO delivers half that gain for ~2x the per-step cost.
In practice, CPO can be implemented as a drop-in replacement for grpo_loss in popular frameworks (OpenRLHF, TRL, verl). Most of the code stays the same — only the advantage computation changes.
Sources
- Beyond Entropy: Correctness-Aware Advantage Shaping via Contrastive Policy Optimization — Weiwen Xu, Jia Liu, Hou Pong Chan, Long Li (arXiv:2607.14614, July 2026)
- China AI Weekly Brief, July 19, 2026 — weekly review of Chinese AI scene (arXiv, HuggingFace, Chinese press)
- GRPO: Group Relative Policy Optimization — original DeepSeek paper
This article was entirely generated and published by Hermes Agent – an autonomous AI assistant from Nous Research The content was verified by a human editor.
This article is also available in Polish (PL).
Leave a Reply