A robotics team's RL agent first learns to predict how the environment will transition given a state and action, then uses that learned model to simulate and plan ahead before acting in the real world. What kind of RL approach is this?
Select an answer to reveal the explanation.
Short Explanation and Infographic
That's model-based reinforcement learning, and the giveaway is right there in the description: the agent isn't just reacting to experience, it's building an internal model that predicts 'if I'm in this state and take this action, here's roughly what happens next.' Once it has that model, it can mentally simulate rollouts and plan ahead, like a chess player thinking several moves ahead, instead of needing to try everything in the real, possibly expensive or dangerous, environment. Tabular Q-learning is model-free, it learns values directly from experience with no explicit dynamics model. Pure policy-gradient methods are also typically model-free, learning a policy straight from trial and error. And imitation learning skips the reward-and-planning loop entirely, just copying expert behavior without any model of dynamics or reward-driven planning.
Full explanation below image
Full Explanation
Model-based reinforcement learning refers to an approach in which the agent explicitly learns, or is given, a model of the environment's transition dynamics, typically a function approximating the probability distribution over next states (and sometimes rewards) given a current state and action. Once this dynamics model is learned, the agent can use it to simulate hypothetical trajectories internally, without needing to interact with the real environment for every rollout, enabling planning algorithms such as model predictive control, Monte Carlo tree search, or trajectory optimization to search over possible action sequences and select ones predicted to yield high cumulative reward. This can dramatically improve sample efficiency compared to model-free methods, since the agent can 'imagine' many outcomes from a relatively small amount of real-world interaction, which is especially valuable in robotics, where real-world trials are slow, costly, or physically risky.
The tabular Q-learning distractor is incorrect because Q-learning is a canonical model-free, value-based method: it learns action-value estimates directly from sampled transitions and rewards without ever constructing or using an explicit model of how the environment transitions between states, meaning it cannot simulate hypothetical futures the way a model-based approach can. The pure policy-gradient distractor is incorrect for a similar reason: policy-gradient methods like REINFORCE or PPO are also generally model-free, directly optimizing policy parameters based on sampled trajectory returns from real (or simulated-as-real, in the case of standard simulators) environment interaction, without learning an internal predictive dynamics model that the agent itself constructs and plans with. The imitation-learning distractor is incorrect because imitation learning, such as behavior cloning, trains a policy to directly mimic expert-provided state-action pairs using supervised learning techniques, bypassing the reward-driven, model-and-plan loop altogether; there is no dynamics model being learned or leveraged for internal simulation and planning in this paradigm.
A helpful memory aid: model-based RL is like a chess player who mentally simulates several moves ahead using their understanding of how pieces move, before committing to an actual move on the board, whereas model-free methods are more like learning purely through played-out games and trial and error, gradually shaping behavior or value estimates from real outcomes without ever building an internal simulation of the game's rules.