Proximal Policy Optimization

Proximal Policy Optimisation (PPO) is the most widely deployed reinforcement learning algorithm in production systems, notable for powering both state-of-the-art robot control and the RLHF training pipeline behind ChatGPT, Claude, and most production LLMs. Its success stems from a simple insight: policy gradient updates can be destabilising when they change the policy too aggressively, and a conservative update mechanism produces more reliable training.

The core problem PPO addresses: standard policy gradient methods compute a gradient step and update policy parameters, but the correct step size is not obvious. Too small a step means slow learning; too large a step can "fall off a cliff" - destroy a well-performing policy in a single update, after which recovery may require many more update steps. TRPO (Trust Region Policy Optimisation) addressed this with a hard constraint on the KL divergence between old and new policy, but the constrained optimisation was expensive to solve.

PPO achieves a similar effect with a much simpler approach: the clipped surrogate objective. The policy ratio rt(theta) = pi_theta(a|s) / pi_theta_old(a|s) measures how much the new policy's action probabilities differ from the old policy's. The PPO objective clips this ratio at (1-epsilon, 1+epsilon) - typically with epsilon=0.2 - preventing the update from making the new policy more than 20% more or less likely to take any given action than the old policy. The objective takes the minimum of the clipped and unclipped values, creating a conservative pessimistic bound.

PPO is an on-policy algorithm: it collects rollouts from the current policy, performs several gradient updates on those rollouts, then discards the data and collects new rollouts. This limits data efficiency (off-policy methods like DQN can reuse experience many times), but PPO compensates with multiple epochs of mini-batch updates on each collected rollout - extracting more information from each batch than a single gradient step.

The Generalised Advantage Estimation (GAE) technique pairs with PPO in most implementations, computing advantage estimates that balance bias and variance using a hyperparameter lambda that interpolates between one-step TD advantages (low variance, high bias) and full Monte Carlo returns (high variance, low bias).

In RLHF for LLMs, PPO is used to optimise the language model's policy against a reward model trained from human preferences. The language model is the policy: its parameters determine the probability distribution over next tokens. The reward model assigns scalar rewards to complete responses. PPO updates the language model parameters to make high-reward response sequences more likely, with a KL penalty against the base model to prevent the policy from collapsing to reward hacking strategies.