In reinforcement learning, what does it mean to use a 'policy gradient' method to train an agent?
Select an answer to reveal the explanation.
Short Explanation
Policy gradient just means you skip the middleman. Instead of learning Q-values and deriving a policy from them, you parameterize the policy itself, usually as a neural network mapping states to action probabilities, and nudge its parameters directly using gradient ascent so expected reward goes up. That's why the third option is correct. Gradient clipping is a training-stability trick used everywhere in deep learning, it has nothing specifically to do with policies. Computing gradients of the environment's dynamics doesn't make sense in model-free settings, and often the dynamics aren't even differentiable or known. And evolutionary search is deliberately gradient-free, the opposite of what 'policy gradient' means. The name tells you exactly what's happening: gradients, applied to the policy.
Full Explanation
Policy gradient methods are a class of reinforcement learning algorithms that directly parameterize the policy, typically as a neural network with parameters theta that outputs a probability distribution over actions given a state, and optimize those parameters by ascending the gradient of an expected-return objective function, J(theta). Using the policy gradient theorem, the gradient of this objective can be estimated from sampled trajectories without needing a model of the environment, allowing the policy's parameters to be updated via standard gradient ascent (or a variant like Adam) so that actions leading to higher returns become more probable over time. Algorithms such as REINFORCE, Proximal Policy Optimization (PPO), and Trust Region Policy Optimization (TRPO) all fall under this umbrella.
The distractor describing gradient clipping is incorrect because clipping is a general optimization stabilization technique applied to prevent exploding gradients during backpropagation in any deep network, unrelated to the specific mechanism of policy parameterization; it is often used alongside policy gradient methods but does not define them. The distractor about computing the gradient of the environment's transition dynamics is incorrect because most policy gradient methods are model-free, meaning they never need to know or differentiate through the environment's dynamics; the policy gradient theorem cleverly sidesteps this by reformulating the objective's gradient in terms of the policy's own log-probabilities weighted by observed returns. The distractor describing gradient-free evolutionary search is incorrect because it describes an entirely separate family of optimization techniques, such as evolution strategies or genetic algorithms, which perturb and select among candidate policies without computing any gradients at all; this is explicitly the opposite approach from policy gradient methods.
A useful way to remember this: policy gradient methods answer the question 'which way should I nudge my policy's dials to make good outcomes more likely,' directly optimizing behavior rather than first learning value estimates and extracting a policy afterward, as value-based methods like Q-learning do.