Choosing and Designing Decay Types for Epsilon‑Greedy Exploration in Reinforcement Learning

In reinforcement learning (RL), the epsilon‑greedy policy is one of the most widely used strategies for balancing exploration and exploitation. The key hyper‑parameter, ε, determines the probability of taking a random action versus selecting the current best‑known action. While a fixed ε can work for simple problems, most practical scenarios benefit from decaying ε over time so that the agent explores heavily early on and gradually shifts toward exploitation.

Why Decay Matters

• Early‑stage discovery: A high ε encourages the agent to sample diverse state‑action pairs, helping it uncover rewarding regions of the environment.
• Late‑stage refinement: As learning progresses, a lower ε reduces stochasticity, allowing the policy to converge on the optimal behavior.
• Stability: Proper decay mitigates the risk of “over‑exploration” that can destabilize value estimates, especially in function‑approximation settings.

Common Decay Functions

1. Linear Decay

Linear decay reduces ε at a constant rate until it reaches a predefined minimum.

ε_t = max(ε_min, ε_start - decay_rate * t)

Pros: Simple to implement and intuitive.

Cons: May drop exploration too quickly or too slowly depending on the chosen rate; lacks flexibility for non‑uniform learning dynamics.

2. Exponential Decay

Exponential decay follows a multiplicative schedule:

ε_t = ε_min + (ε_start - ε_min) * exp(-k * t)

Pros: Provides a smooth, rapid reduction early on while preserving a long tail of small exploration.

Cons: Choosing the decay constant k can be tricky; too large a k may lead to premature exploitation.

3. Inverse‑Time Decay

Often used in theoretical analyses, this schedule decays proportionally to the inverse of the timestep:

ε_t = ε_start / (1 + decay_factor * t)

Pros: Guarantees ∑ε_t diverges, satisfying certain convergence proofs.

Cons: Decay can be overly aggressive for deep RL where learning spans millions of steps.

4. Stepwise (Piecewise) Decay

In stepwise decay, ε is held constant for a number of episodes, then abruptly reduced:

if episode % step_interval == 0:
    ε = max(ε_min, ε * decay_factor)

Pros: Aligns well with curriculum‑learning setups; easy to tune per training phase.

Cons: Abrupt changes may cause temporary performance drops as the policy adapts.

5. Adaptive Decay (Performance‑Based)

Adaptive methods adjust ε based on observed performance metrics, such as moving‑average reward or TD‑error magnitude:

if recent_reward_improvement < threshold:
    ε = max(ε_min, ε * decay_factor)

Pros: Directly ties exploration to learning progress, often yielding more efficient training.

Cons: Requires careful selection of metrics and thresholds; can introduce additional hyper‑parameter noise.

Design Guidelines for Selecting a Decay Type

1. Consider the problem horizon. For short episodes (e.g., classic control), a fast linear or exponential decay often suffices. For long‑horizon tasks (e.g., Atari or robotics), slower decays like inverse‑time or adaptive schedules are preferable.
2. Align decay with the learning algorithm. Value‑based methods (DQN, SARSA) benefit from a smoother decay to maintain stable Q‑updates, while policy‑gradient methods (A2C, PPO) can tolerate more aggressive early decay.
3. Set meaningful bounds. Always define ε_start (commonly 1.0) and ε_min (often 0.01–0.1). The minimum ensures the agent never becomes completely deterministic, preserving a safety net against local optima.
4. Validate decay hyper‑parameters with a small grid search. Even simple schedules have two crucial knobs (rate/constant and minimum). A quick sweep across 3–5 values can reveal whether the agent is under‑ or over‑exploring.
5. Monitor exploration diagnostics. Track the proportion of random actions, state‑visitation heatmaps, and entropy of the policy. Sudden drops may indicate overly aggressive decay.

Practical Implementation Example (Python‑style Pseudocode)

class EpsilonScheduler:
    def __init__(self, start=1.0, min_eps=0.05, decay_type='exponential', **kwargs):
        self.eps = start
        self.min_eps = min_eps
        self.decay_type = decay_type
        self.kwargs = kwargs

    def step(self, t):
        if self.decay_type == 'linear':
            self.eps = max(self.min_eps,
                           self.kwargs['start'] - self.kwargs['rate'] * t)
        elif self.decay_type == 'exponential':
            self.eps = self.min_eps + (self.kwargs['start'] - self.min_eps) \
                       * math.exp(-self.kwargs['k'] * t)
        elif self.decay_type == 'inverse_time':
            self.eps = self.kwargs['start'] / (1 + self.kwargs['factor'] * t)
        elif self.decay_type == 'stepwise':
            if t % self.kwargs['interval'] == 0:
                self.eps = max(self.min_eps, self.eps * self.kwargs['factor'])
        elif self.decay_type == 'adaptive':
            # pseudo‑logic based on recent reward trend
            if self.kwargs['reward_improvement'] < self.kwargs['threshold']:
                self.eps = max(self.min_eps, self.eps * self.kwargs['factor'])
        return self.eps

Conclusion

Choosing and designing the decay type for epsilon‑greedy exploration is not a “set‑and‑forget” decision. The decay schedule directly shapes the exploration‑exploitation trade‑off, influences convergence speed, and can determine whether an RL agent discovers the optimal policy at all. By understanding the characteristics of linear, exponential, inverse‑time, stepwise, and adaptive decays—and by aligning them with the problem horizon, algorithmic properties, and performance diagnostics—practitioners can craft more robust and efficient learning pipelines.

Remember: explore wisely, decay deliberately, and let the agent’s performance guide the schedule.