The usual Bellman backup has a tiny courtroom inside it:

bring me every action
score each future
keep the largest

For a discounted Markov decision process, that courtroom is usually written as the Bellman optimality backup:1

\[V(s) = \max_a \left\{ r(s,a) + \gamma \mathbb{E}[V(s') \mid s,a] \right\}.\]

That max is decisive. It is also brittle. If two actions are nearly tied, the backup still returns one winner. If the model is a little wrong, the winner can flip. If the policy needs to keep several competent behaviors alive, the hard maximum has already thrown most of them away.

Entropy-regularized control changes the courtroom rule:

\[V_\alpha(s) = \alpha \log \sum_a \exp(Q(s,a)/\alpha).\]

The policy implied by the same backup is

\[\pi_\alpha(a \mid s) = \frac{\exp(Q(s,a)/\alpha)} {\sum_b \exp(Q(s,b)/\alpha)}.\]

The max has a temperature.

As \(\alpha \to 0\), log-sum-exp becomes the hard maximum. At larger \(\alpha\), near-winners keep probability mass. That is not epsilon-greedy exploration bolted on after planning. It is a different dynamic program.

The Temperature Is Paid In Reward Units

The most important practical detail is also the easiest one to miss:

temperature and reward scale are the same knob in different clothes

If every reward is multiplied by ten and \(\alpha\) is left alone, the policy gets sharper. If \(\alpha\) is multiplied by ten and rewards are left alone, the policy gets softer. Soft Actor-Critic made this operational in deep RL: the maximum-entropy objective optimizes expected reward plus an entropy term, and the temperature controls how much stochasticity survives in the optimal policy.2

This is why “turn up exploration” is not a clean sentence. The agent is not sprinkling noise over a fixed plan. It is solving a planning problem where randomness itself earns or costs value.

Geist, Scherrer, and Pietquin put the modern abstraction cleanly: regularized MDPs can be built by replacing the ordinary Bellman operator with a regularized Bellman operator, and the Legendre-Fenchel transform supplies the smoothed maximum.3 For negative entropy, the smoothed maximum is the familiar log-sum-exp.

So the soft backup is not a trick for neural networks. It is a tabular object before any neural network appears.

The Lab

The little world below has one start state and two terminal goals:

A reward: 10.00
B reward:  9.55
step cost: -0.04
slip:       8%
gamma:      0.96

The hard optimal policy almost always ends at A, because A is slightly better. The soft policy is allowed to keep some probability on the near-tie route to B.

At the default setting, with \(\alpha = 0.15\):

top goal A          82.3%
bottom goal B       17.7%
hard-max A rate     99.9%
start entropy        1.25 nats
extrinsic return     5.27
entropy term         1.30
soft value bonus     0.56

The start-state soft policy is not a single arrow:

up     34.8%
right  40.1%
down   11.5%
left   13.7%

That is the point. The policy remembers the alternatives because the backup remembered them.

Goal A / occupancy Goal B Soft policy Hard max Timeout

Deterministic tabular solver plus seeded Monte Carlo rollouts. The solver runs hard and entropy-regularized value iteration on the same small MDP, then samples trajectories from the soft policy. Reward scale multiplies the rewards used by the Bellman backup; reported return is unscaled extrinsic reward.

Try four moves.

First, lower Temperature alpha. The policy sharpens, and goal A dominates. At alpha = 0.05, the sweep shows A around 96% of rollouts. This is the hard-max world returning through the side door.

Second, set Goal gap A-B to zero. Now the two terminal rewards are tied. The hard policy’s tie-breaking is arbitrary, but the soft policy can represent the symmetry as probability mass.

Third, raise Reward scale. The displayed temperature did not move, but the effective temperature fell because the action-value gaps got larger in reward units. This is the same reason SAC implementations care about reward scaling: it changes the stochasticity of the energy-based policy.2

Fourth, raise Temperature alpha too far. The entropy line saturates near log(4), and the top-goal probability collapses. That is not a bug. A maximum-entropy objective can make dithering valuable if the temperature is large relative to the task rewards and horizon. Randomness is not free; it has been placed inside the objective.

Soft Is Not Always Better

The soft backup has a seductive story:

keep near-optimal modes alive
avoid brittle argmax flips
explore without bolting noise onto a deterministic policy

All three are real. None is a license to forget the objective.

The hard maximum answers:

what is the best action if the model and reward are exactly right?

The soft maximum answers:

what distribution trades reward against entropy at this temperature?

Those are different questions. A temperature that is wonderful during learning can be too random for deployment. A reward scale that looks harmless in a code cleanup can silently change the effective policy. A stochastic policy that covers multiple modes can also keep revisiting bad states because entropy is being paid every step.

This is why I prefer to write the backup explicitly in design docs. If the algorithm is doing

\[\alpha \log \sum_a \exp(Q(s,a)/\alpha)\]

then the review should ask for \(\alpha\), reward units, discount, horizon, and the deployment policy. The “exploration knob” is part of the theorem.

Control As Inference, In One Sentence

There is a deeper reason this form keeps appearing. Exponentials turn sums of rewards into weights over trajectories.

Ziebart, Maas, Bagnell, and Dey’s maximum entropy inverse reinforcement learning paper used a globally normalized distribution over decision sequences to model noisy route behavior.4 Instead of declaring one path optimal, the model keeps a probability distribution over paths, with higher-reward paths receiving more mass.

Todorov’s linearly-solvable MDPs take a related control-theoretic route: control rescales passive dynamics, and a KL-divergence control cost makes a transformed Bellman equation linear.5 That is not the same algorithm as the toy lab above, but it shares the same smell: reward, probability, and relative entropy are not separate bookkeeping systems. They can be one planning language.

The compact memory hook is:

hard Bellman: choose the best future
soft Bellman: price a distribution over futures

That one change explains why soft Q-learning, SAC, KL-control, path-integral control, and maximum-entropy imitation learning keep talking to one another.

The Audit I Want

Before trusting a maximum-entropy controller, I want the following written down:

What units are rewards in?
What value of alpha or target entropy is being used?
Was reward scaling changed during preprocessing or environment wrapping?
Is the learned policy deployed stochastically, greedily, or with another temperature?
Does entropy reward continue after the task is effectively over?
Are there near-tie modes that should remain alive for robustness or diversity?
Are there loops where entropy can be collected without task progress?

The last line is the uncomfortable one. Entropy regularization can be a principled way to avoid premature commitment. It can also turn indecision into income.

The max has a temperature. That makes it smoother, more expressive, and easier to misuse.

  1. Richard Bellman, “On the Theory of Dynamic Programming”, Proceedings of the National Academy of Sciences 38(8):716-719, 1952. DOI: 10.1073/pnas.38.8.716. ↩

  2. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine, “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor”, ICML 2018. ↩ ↩2

  3. Matthieu Geist, Bruno Scherrer, and Olivier Pietquin, “A Theory of Regularized Markov Decision Processes”, ICML 2019. ↩

  4. Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey, “Maximum Entropy Inverse Reinforcement Learning”, AAAI 2008. ↩

  5. Emanuel Todorov, “Linearly-solvable Markov decision problems”, NeurIPS 2006. ↩