Choosing and Designing Decay Types for Epsilon‑Greedy Exploration in Reinforcement Learning

In reinforcement learning (RL), the epsilon‑greedy policy is one of the most popular strategies for balancing exploration and exploitation. The key to its success lies not only in the initial value of ε, but also in how ε decays over time. A well‑designed decay schedule can dramatically accelerate learning, improve final performance, and reduce sensitivity to hyper‑parameters.

Why Decay Matters

Early exploration: A high ε encourages the agent to sample diverse state‑action pairs, preventing premature convergence to sub‑optimal policies.
Gradual exploitation: As the agent gathers knowledge, lowering ε allows it to exploit learned value estimates more often.
Stability: An abrupt transition from exploration to exploitation can destabilize learning, especially in non‑stationary environments.

Common Decay Functions

1. Linear Decay

$$\epsilon_t = \epsilon_{0} - \frac{t}{T}\cdot(\epsilon_{0} - \epsilon_{\text{min}})$$

Simple to implement; reduces ε uniformly over a predefined number of steps T. Works well when the learning progress is roughly linear.

2. Exponential Decay

$$\epsilon_t = \epsilon_{\text{min}} + (\epsilon_{0} - \epsilon_{\text{min}})\cdot \exp(-kt)$$

The decay rate k controls how fast ε approaches the minimum. This schedule is useful when early exploration should be intense but quickly give way to exploitation.

3. Inverse‑Time Decay

$$\epsilon_t = \frac{\epsilon_{0}}{1 + kt}$$

Provides a slower, asymptotic reduction. It is less aggressive than exponential decay and can be advantageous in environments with delayed rewards.

4. Piecewise (Step) Decay

Define checkpoints t₁, t₂, … at which ε is multiplied by a factor γ (0 < γ < 1). Example:

if t >= t₁: ε = ε·γ
if t >= t₂: ε = ε·γ
...

This mimics a “cool‑down” schedule, giving the agent periods of high exploration followed by rapid exploitation phases.

5. Adaptive Decay

Instead of a predetermined function, adjust ε based on learning signals such as:

Moving average of the reward R̄ – decrease ε when R̄ improves.
Temporal‑difference (TD) error – reduce ε when TD error stabilizes.
Visit count per state – lower ε for well‑visited states while keeping it higher for rarely visited ones.

Adaptive schemes can automatically tailor exploration to the problem’s difficulty.

Design Guidelines

Set sensible bounds: Choose ε₀ (often between 0.9 and 1.0) and ε_min (commonly 0.01–0.1). Extremely low ε_min may cause the agent to miss late‑stage improvements.
Match decay speed to environment dynamics:
- Fast‑changing environments → slower decay (e.g., inverse‑time).
- Simple, deterministic tasks → faster decay (e.g., exponential or step).
Synchronize with training milestones: Align decay checkpoints with episodes, frames, or policy‑update cycles to keep the schedule interpretable.
Monitor performance metrics: Plot ε vs. average reward. If reward plateaus while ε is still high, consider accelerating decay.
Combine schedules: A common practical approach is to start with linear decay for the first N steps, then switch to a slower exponential tail.

Implementation Example (Python‑like Pseudocode)

def epsilon_greedy(state, Q, step, config):
    # config contains: eps_start, eps_end, decay_type, decay_params
    eps = compute_epsilon(step, config)
    if random.random() < eps:
        return random_action()
    else:
        return argmax(Q[state])

def compute_epsilon(t, cfg):
    if cfg.decay_type == 'linear':
        return max(cfg.eps_end,
                   cfg.eps_start - (t / cfg.decay_steps)*(cfg.eps_start-cfg.eps_end))
    elif cfg.decay_type == 'exp':
        return cfg.eps_end + (cfg.eps_start-cfg.eps_end)*math.exp(-cfg.k*t)
    elif cfg.decay_type == 'inverse':
        return cfg.eps_start / (1 + cfg.k*t)
    elif cfg.decay_type == 'step':
        eps = cfg.eps_start
        for pt in cfg.steps:
            if t >= pt:
                eps *= cfg.gamma
        return max(eps, cfg.eps_end)
    elif cfg.decay_type == 'adaptive':
        # example using moving average reward
        if cfg.reward_avg > cfg.reward_target:
            cfg.k *= 1.05   # slow down decay
        return max(cfg.eps_end, cfg.eps_start * math.exp(-cfg.k*t))

Empirical Tips

Run a short hyper‑parameter sweep over decay rates (e.g., k in exponential decay) before committing to long training runs.
Log both ε and performance to spot over‑exploration or premature exploitation.
Combine with other exploration methods (e.g., Boltzmann, noisy nets) when epsilon‑greedy alone is insufficient.

Conclusion

The choice and design of an epsilon‑greedy decay schedule is a decisive factor in RL success. Simple linear or exponential decays work for many benchmark tasks, but more sophisticated piecewise or adaptive schemes can unlock higher performance in complex or non‑stationary environments. By aligning decay dynamics with the problem’s learning curve, monitoring metrics, and iteratively tuning hyper‑parameters, practitioners can achieve faster convergence and more robust policies.

Search This Blog

ChippyTech Blog