An RL researcher describes a method that updates a state's estimated value immediately after each single step, using the observed reward plus the current estimated value of the next state, rather than waiting for an entire episode to finish. What learning approach is being described?
Select an answer to reveal the explanation.
Short Explanation and Infographic
That one-step, update-as-you-go approach is temporal-difference, or TD, learning. The key trick is bootstrapping: instead of waiting around for the whole episode to finish to know the true return, TD learning updates its value estimate right after a single step using the immediate reward plus its current guess about the next state's value. It's model-free too, no need to know the environment's transition probabilities ahead of time. Monte Carlo learning is the one that waits for the full episode to end before updating anything, using the actual total return, so that's the opposite of what's described here. Supervised regression on a fixed labeled dataset doesn't fit either, there's no interactive bootstrapping or ongoing environment feedback loop. And dynamic programming needs a full, known model of the environment's dynamics upfront, which TD learning explicitly does not require.
Full explanation below image
Full Explanation
Temporal-difference (TD) learning is a family of model-free reinforcement learning methods that update value estimates incrementally, after each individual time step, rather than waiting for an entire episode to conclude. The defining characteristic of TD learning is bootstrapping: the value update for the current state uses the observed immediate reward plus the current (possibly still-inaccurate) estimated value of the successor state, rather than the actual full return that would only be known once the episode terminates. This is expressed in the simplest form, TD(0), as an update rule where the value of the current state is nudged toward the sum of the immediate reward and the discounted estimated value of the next state, scaled by a learning rate. Because TD methods bootstrap from estimates rather than waiting for complete outcomes, they can learn online, from incomplete episodes, and even in continuing (non-episodic) tasks, and they are foundational to algorithms like SARSA and Q-learning, both of which are specific instances of TD-style updates applied to action-value functions.
The Monte Carlo distractor is incorrect because Monte Carlo methods, by definition, wait until an entire episode has finished and use the actual observed cumulative return from that episode to update value estimates, with no bootstrapping from intermediate value estimates at all; this makes Monte Carlo methods unbiased but typically higher variance and unusable in non-episodic (continuing) tasks, which is precisely the opposite tradeoff profile from TD learning. The supervised-regression distractor is incorrect because that describes a static, offline learning setup where a model is fit to a fixed, pre-existing dataset of labeled input-output pairs using standard loss minimization; it lacks the online, incremental, self-referential bootstrapping structure central to TD learning, where the 'label' (the target value) is itself derived from the model's own evolving estimates rather than being fixed ground truth. The dynamic-programming distractor is incorrect because classical dynamic programming methods, such as policy iteration and value iteration, require complete and accurate knowledge of the environment's transition probabilities and reward function to perform full backups across the entire state space; TD learning, by contrast, is explicitly model-free, learning purely from sampled experience without ever needing or using an explicit model of the environment's dynamics.
A useful memory aid: TD learning is like updating your commute-time estimate the moment you hit the next landmark, using how long that leg took plus your current guess of the remaining trip, rather than waiting until you're all the way home to figure out if your original estimate was right.