Escaping Policy Contraction: Contraction-Aware PPO (CaPPO) for Stable Language Model Fine-Tuning

Core task: Mitigating policy contraction in reinforcement learning from human feedback. The field addresses a fundamental challenge in RLHF: as policies are optimized against learned reward models, they often exhibit pathological behaviors including mode collapse, reduced output diversity, and degraded capabilities. The taxonomy organizes research into twelve major branches. Policy Stability and Collapse Prevention focuses on entropy regularization and diversity maintenance techniques to prevent the policy from collapsing to narrow distributions. Reward Model Robustness and Reliability examines how to build more trustworthy reward signals that resist exploitation. Reward Over-Optimization Control studies mechanisms to prevent policies from gaming imperfect reward models, while Alignment Tax and Capability Preservation investigates trade-offs between alignment objectives and model performance. Additional branches cover preference learning quality, training efficiency, safety constraints, theoretical foundations, alternative optimization frameworks beyond standard RLHF, domain-specific applications, calibration methods, and practical implementation considerations. Representative works like Policy Collapse[1] and RLHF Open Problems[2] have documented these failure modes, while methods such as Auxiliary Network Stability[4] and Preference Collapse[5] propose targeted interventions. Several active research directions reveal key tensions in the field. One line emphasizes explicit entropy management and diversity preservation, recognizing that standard KL-penalty approaches may be insufficient when policies contract toward high-reward but narrow behaviors, as documented in Diversity Collapse[7]. Another direction explores reward model uncertainty and robustness, with works like RLHF Semantic Vulnerabilities[3] showing how policies exploit model weaknesses. CaPPO[0] sits within the policy stability branch alongside entropy-focused methods, proposing contraction-aware mechanisms to maintain policy expressiveness during optimization. Compared to neighboring approaches like M-GRPO[42], which modifies the optimization objective itself, CaPPO[0] emphasizes direct intervention in the policy update process to preserve distributional breadth. The broader challenge remains balancing alignment quality against the risk of over-optimization, with ongoing debate about whether solutions should modify rewards, constrain policy updates, or fundamentally rethink the RLHF training loop.