Choosing and Designing Decay Types for Epsilon‑Greedy Exploration in Reinforcement Learning

One of the most widely used strategies for balancing exploration and exploitation in reinforcement learning (RL) is the epsilon‑greedy algorithm. At each decision step the agent chooses a random action with probability ε (exploration) and the best‑known action with probability 1 − ε (exploitation). The key to effective learning, however, lies not in the static value of ε, but in how ε is decayed over time.

Why Decay Matters

Early in training the environment is largely unknown, so a high ε encourages the agent to gather diverse experiences. As learning progresses, the policy becomes more reliable and the agent should exploit the knowledge it has acquired. A well‑designed decay schedule ensures that:

The agent explores enough to avoid local optima.
Training time is used efficiently, reducing wasted random actions.
The final policy is stable and performs well on unseen episodes.

Common Decay Functions

1. Linear Decay

$$\epsilon_t = \epsilon_{\text{start}} - \frac{t}{T}(\epsilon_{\text{start}}-\epsilon_{\text{end}})$$

Linear decay reduces ε at a constant rate until it reaches a predefined minimum ε_end. It is simple to implement and works well for environments with relatively uniform learning progress.

2. Exponential Decay

$$\epsilon_t = \epsilon_{\text{start}} \cdot \lambda^{\,t}$$

With a decay factor λ ∈ (0,1), ε drops quickly at first and then slows down, giving a long tail of low‑probability exploration. Exponential decay is useful when early exploration must be aggressive but you still want occasional random moves later.

3. Inverse (Hyperbolic) Decay

$$\epsilon_t = \frac{\epsilon_{\text{start}}}{1 + kt}$$

This schedule decays fast initially but asymptotically approaches zero more slowly than exponential. It can be tuned with the constant k to control the steepness of the fall‑off.

4. Cosine Annealing

$$\epsilon_t = \epsilon_{\text{end}} + \frac{1}{2}(\epsilon_{\text{start}}-\epsilon_{\text{end}})\bigl[1 + \cos(\pi t / T)\bigr]$$

Borrowed from learning‑rate schedules, cosine annealing provides a smooth, periodic reduction that can be restarted (warm‑restarts) to re‑inject exploration after a plateau.

5. Adaptive Decay (Performance‑Based)

Instead of tying decay to time steps, adaptive methods adjust ε based on recent performance metrics (e.g., moving average reward or TD‑error). A simple rule is:


if (reward_avg > threshold)   ε ← max(ε_min, ε * decay_factor);
else                           ε ← min(ε_max, ε / decay_factor);

This approach reacts to the actual learning dynamics, increasing exploration when performance stalls and decreasing it when progress accelerates.

Design Guidelines for Choosing a Decay Type

Environment Horizon: For short‑horizon tasks (few hundred steps) a rapid decay (linear or exponential) prevents wasted exploration. Long‑horizon problems benefit from slower schedules (inverse or cosine).
Reward Sparsity: Sparse rewards often demand sustained exploration. Consider a longer tail (inverse, cosine, or adaptive) to keep the agent probing the state space.
Computational Budget: Adaptive decay requires tracking additional statistics and may add overhead. If compute is constrained, a fixed schedule (linear/exponential) is preferable.
Policy Stability: When a near‑optimal policy is already reached, sharply decreasing ε can freeze learning. A gradual decay or a small constant ε_min (e.g., 0.01) preserves a minimal exploration rate.
Hyperparameter Interaction: Decay interacts with learning rate, replay buffer size, and target‑network update frequency. Conduct systematic grid or Bayesian searches that include decay parameters.

Practical Implementation Tips

Clip the Value: Always enforce ε = max(ε_min, min(ε_max, ε)) to avoid negative or excessively large probabilities.
Decay per Episode vs. per Step: Decaying per episode smooths the schedule, especially when episode lengths vary widely. Decay per step provides finer granularity for environments with many tiny timesteps.
Warm‑Start Exploration: Begin with ε_start = 1.0 or a very high value for the first few episodes to guarantee broad coverage.
Log and Visualize: Plot ε alongside reward curves. Misaligned trends often reveal a decay schedule that is too aggressive or too lax.
Combine with Other Exploration Strategies: Epsilon‑greedy can be hybridized with Bayesian posterior sampling, NoisyNets, or intrinsic curiosity modules to further enrich exploration.

Example Code Snippet (Python)

def get_epsilon(step, config):
    # Config dict contains schedule type and parameters
    if config['type'] == 'linear':
        slope = (config['start'] - config['end']) / config['steps']
        epsilon = max(config['end'], config['start'] - slope * step)
    elif config['type'] == 'exponential':
        epsilon = max(config['end'],
                      config['start'] * (config['decay'] ** step))
    elif config['type'] == 'inverse':
        epsilon = max(config['end'],
                      config['start'] / (1 + config['k'] * step))
    elif config['type'] == 'cosine':
        t = step / config['steps']
        epsilon = config['end'] + 0.5 * (config['start'] - config['end']) * (1 + 
                  math.cos(math.pi * t))
    else:
        raise ValueError('Unsupported decay type')
    return epsilon

Conclusion

The epsilon‑greedy algorithm remains a cornerstone of RL exploration, but its performance hinges on a thoughtfully designed decay schedule. By aligning the decay type with the problem’s horizon, reward structure, and computational constraints, you can dramatically improve sample efficiency and final policy quality. Experiment with linear, exponential, inverse, cosine, and adaptive decays—track their impact, and let empirical evidence guide the final choice.

Remember: exploration is not a one‑size‑fits‑all component. Tailoring ε decay is an art that blends theoretical insight with practical experimentation, and mastering it is a key step toward building robust, high‑performing reinforcement‑learning agents.

Search This Blog

ChippyTech Blog