Exploration vs Exploitation

Every reinforcement learning agent faces a fundamental tension. Exploitation means using what you know: take the action that currently appears best according to your learned Q-values or policy. This collects reward reliably but can leave you stuck in a suboptimal strategy if you never tried the actions that might be even better. Exploration means trying actions you are uncertain about, gathering information that might reveal better strategies at the cost of lower immediate reward.

This tension is not unique to RL - it appears throughout decision-making under uncertainty - but it is particularly acute in RL because the agent's learning depends entirely on the experience it generates. An agent that only exploits can only improve within the strategies it has already discovered. An agent that only explores wastes reward on random actions rather than applying what it has learned.

The multi-armed bandit problem is the canonical formalisation. An agent faces K slot machines (bandits), each with an unknown reward distribution. The agent must allocate a fixed number of pulls to maximise total reward. Pulling a machine it knows is good exploits that knowledge; pulling a less-tested machine explores. This simplified setting strips away sequential decision-making to isolate the exploration-exploitation tradeoff.

Epsilon-greedy exploration is the simplest practical strategy: take the greedy best action with probability 1-epsilon, and a uniformly random action with probability epsilon. Epsilon is often annealed over training - starting high (pure exploration) and decreasing toward zero (more exploitation) as the agent gains experience. Simple and effective, epsilon-greedy is the default in tabular RL and DQN.

More principled approaches use uncertainty estimates to guide exploration. Upper Confidence Bound (UCB) algorithms maintain optimistic value estimates and select actions based on the upper confidence bound of their value - actions that have been tried rarely receive a bonus that encourages their selection. Thompson Sampling maintains a full posterior distribution over action values and samples from it to select actions. Both achieve better theoretical regret bounds than epsilon-greedy.

Intrinsic motivation approaches reward the agent for visiting novel states, independent of the extrinsic task reward. Count-based exploration bonuses give rewards proportional to how rarely a state has been visited. Curiosity-driven exploration (using prediction error as an intrinsic reward signal) provides novelty bonuses without explicit state counting, enabling exploration of high-dimensional observation spaces.

For LLMs trained with RLHF, exploration is implicit in the stochasticity of token sampling. Temperature controls exploration-exploitation: high temperature generates diverse responses (exploration), low temperature generates predictable, near-greedy responses (exploitation). The PPO entropy bonus explicitly encourages the policy to maintain diversity.