In a standard PyTorch training loop, after loss.backward() has computed gradients for every parameter, what is the job of the optimizer object (for example, an instance of torch.optim.SGD)?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Here's the deal: by the time you call loss.backward(), the hard math is already done — every parameter in your model now has a .grad attribute telling it which direction to move to reduce the loss. The optimizer's whole job is to take those gradients and actually adjust the parameters, using whatever update rule it implements, like plain SGD or something fancier like Adam. Call optimizer.step() and the weights shift; call optimizer.zero_grad() and the old gradients get cleared out for the next round. That's why 'update parameters using computed gradients' is correct. Computing the loss is the loss function's job, the forward pass is the model's job, and batching data is DataLoader's job — the optimizer only touches parameters, after gradients already exist.
Full explanation below image
Full Explanation
In PyTorch, the optimizer (an instance of a class in torch.optim, such as SGD, Adam, or RMSprop) is responsible for updating a model's learnable parameters based on the gradients computed during backpropagation. The typical training loop is: run a forward pass to get predictions, compute a loss by comparing predictions to ground truth, call loss.backward() to populate the .grad attribute on every parameter tensor via automatic differentiation, and then call optimizer.step() to apply an update rule — such as gradient descent with a learning rate, or a more adaptive rule like Adam's momentum and per-parameter scaling — that adjusts each parameter in the direction that reduces the loss. The optimizer is initialized with a reference to the model's parameters (via model.parameters()) precisely so it knows which tensors to update, and it also handles bookkeeping like clearing gradients via optimizer.zero_grad() so gradients from one step don't accumulate into the next.
Computing the loss value is not the optimizer's role; that is handled by a loss function object (such as nn.CrossEntropyLoss or nn.MSELoss), which quantifies how far predictions are from targets and produces the scalar that backward() differentiates.
Performing the forward pass is the responsibility of the model itself — calling model(inputs) runs the input through the defined layers to produce output predictions. The optimizer never touches this step and has no knowledge of the network's architecture beyond the parameter tensors it was given.
Loading and shuffling training data into mini-batches is the job of a DataLoader wrapping a Dataset. This happens before the forward pass even begins and is entirely separate from parameter updates.
Understanding this separation of concerns — Dataset/DataLoader for data, the model for the forward pass, the loss function for measuring error, autograd for computing gradients, and the optimizer strictly for applying updates — is fundamental to reading and debugging any PyTorch training loop, since each piece can be swapped independently.