A value function in reinforcement learning estimates the expected cumulative future reward obtainable from a given state (state-value) or state-action pair (action-value) under a particular policy. It captures the long-term desirability of situations rather than immediate reward, and satisfies the recursive Bellman equation that relates the value of a state to the values of its successors. Value functions are central to dynamic programming, temporal-difference learning, and actor-critic methods, providing the signal that guides an agent toward reward-maximising behaviour.
Content
- The value function answers the question an agent must repeatedly confront: not “what reward do I get now?” but “how good is it to be here, given everything that follows?” By summing discounted future rewards in expectation, it converts the sequential, delayed-consequence nature of decision problems into a scalar measure of state desirability, allowing an agent to prefer actions that lead to good situations even when immediate reward is low.
- Its defining mathematical property is recursion through the Bellman equation, which states that the value of a state equals the immediate reward plus the discounted value of the successor state. This self-consistency turns value estimation into a fixed-point problem solvable by iteration: dynamic programming sweeps the equation to convergence when the model is known, while temporal-difference learning bootstraps estimates from sampled experience when it is not, updating each estimate toward the observed reward plus the current estimate of the next state.
- Two flavours recur throughout the field. The state-value function evaluates how good a state is under a policy; the action-value (Q) function evaluates how good taking a specific action in a state is, which is more directly useful for control because the agent can simply choose the highest-valued action. Q-learning and its deep-network descendants learn the optimal action-value function directly, and value estimates also serve as the critic that reduces variance in policy-gradient and actor-critic methods.
- In large or continuous state spaces, the value function cannot be tabulated and must be approximated, classically with linear features and now predominantly with deep neural networks. This combination — value functions represented by neural approximators and trained with temporal-difference targets — powered landmark results in game playing and robotics, while also introducing instabilities (the deadly triad of bootstrapping, off-policy learning, and function approximation) that motivate techniques such as target networks and experience replay to keep learning stable.