VII · Reinforcement Learning & RoboticsAdvanced

Proximal Policy Optimization

The dominant policy gradient algorithm in modern RL and LLM fine-tuning - achieving stable, sample-efficient training by clipping the policy update ratio to prevent destructively large parameter changes.

Added May 18, 2026 · 3 min read

PPO is the algorithm most practitioners encounter when implementing or understanding RLHF for LLMs, and the dominant algorithm for robotic control. Understanding it explains the mechanics of RLHF training - why there is a clip hyperparameter, why the KL divergence penalty appears, why training with RL is more unstable than supervised training, and what is actually being optimised when a language model is fine-tuned with human feedback.

Proximal Policy Optimisation (PPO) is the most widely deployed reinforcement learning algorithm in production systems, notable for powering both state-of-the-art robot control and the RLHF training pipeline behind ChatGPT, Claude, and most production LLMs. Its success stems from a simple insight: policy gradient updates can be destabilising when they change the policy too aggressively, and a conservative update mechanism produces more reliable training.

The core problem PPO addresses: standard policy gradient methods compute a gradient step and update policy parameters, but the correct step size is not obvious. Too small a step means slow learning; too large a step can "fall off a cliff" - destroy a well-performing policy in a single update, after which recovery may require many more update steps. TRPO (Trust Region Policy Optimisation) addressed this with a hard constraint on the KL divergence between old and new policy, but the constrained optimisation was expensive to solve.

PPO achieves a similar effect with a much simpler approach: the clipped surrogate objective. The policy ratio rt(theta) = pi_theta(a|s) / pi_theta_old(a|s) measures how much the new policy's action probabilities differ from the old policy's. The PPO objective clips this ratio at (1-epsilon, 1+epsilon) - typically with epsilon=0.2 - preventing the update from making the new policy more than 20% more or less likely to take any given action than the old policy. The objective takes the minimum of the clipped and unclipped values, creating a conservative pessimistic bound.

PPO is an on-policy algorithm: it collects rollouts from the current policy, performs several gradient updates on those rollouts, then discards the data and collects new rollouts. This limits data efficiency (off-policy methods like DQN can reuse experience many times), but PPO compensates with multiple epochs of mini-batch updates on each collected rollout - extracting more information from each batch than a single gradient step.

The Generalised Advantage Estimation (GAE) technique pairs with PPO in most implementations, computing advantage estimates that balance bias and variance using a hyperparameter lambda that interpolates between one-step TD advantages (low variance, high bias) and full Monte Carlo returns (high variance, low bias).

In RLHF for LLMs, PPO is used to optimise the language model's policy against a reward model trained from human preferences. The language model is the policy: its parameters determine the probability distribution over next tokens. The reward model assigns scalar rewards to complete responses. PPO updates the language model parameters to make high-reward response sequences more likely, with a KL penalty against the base model to prevent the policy from collapsing to reward hacking strategies.

Analogy

Learning to ride a motorcycle by gradually leaning into curves. A novice who leans too aggressively crashes. A conservative rider who only leans slightly learns slowly. The ideal is to lean as much as possible while staying within the stable envelope - pushing capability with each ride but never so far that a crash resets progress. PPO applies the same principle to policy updates: update aggressively within a safe region defined by the clip ratio, then stop rather than risk destabilising the policy.

Real-world example

In RLHF fine-tuning of a language model: the policy (LLM) generates a response, the reward model scores it (e.g., +2.3 for a helpful response, -1.1 for a sycophantic one), and PPO computes an update. Without clipping, a large reward signal could push the policy to generate that exact response pattern extremely aggressively, collapsing diversity and causing reward hacking (the model learns to score high on the reward model without being genuinely helpful). The PPO clip limits how much any single update can shift the probability of any response, keeping the training stable across millions of RLHF updates.

Why it matters

PPO is the algorithm most practitioners encounter when implementing or understanding RLHF for LLMs, and the dominant algorithm for robotic control. Understanding it explains the mechanics of RLHF training - why there is a clip hyperparameter, why the KL divergence penalty appears, why training with RL is more unstable than supervised training, and what is actually being optimised when a language model is fine-tuned with human feedback.

In the news

Related concepts

Actor-Critic

A reinforcement learning architecture that combines a policy network (the actor, which decides which actions to take) with a value network (the critic, which evaluates how good the current state is) - reducing gradient variance and enabling more stable learning than pure policy gradient.

Policy Gradient

A family of reinforcement learning algorithms that directly optimise a parameterised policy by computing gradients of expected reward with respect to policy parameters - enabling RL on continuous action spaces where value-based methods struggle.

RLHF (Reinforcement Learning from Human Feedback)

A training technique that teaches AI to produce responses humans actually prefer, by having real people rate different outputs and using those ratings to improve the model.

← Back to concepts