A student asks what makes a value inside a neural network count as a 'trainable parameter.' What is the best definition?
Select an answer to reveal the explanation.
Short Explanation
A trainable parameter is one of the values inside the model — a weight or a bias — that the optimizer is actively allowed to change during training. Each step, gradients get computed showing how much that value contributed to the error, and the optimizer nudges it in a direction that should reduce that error. Do that over and over, thousands of times, and that's how the network 'learns.' That automatic, gradient-driven updating is exactly what defines a trainable parameter, so that's the answer. A fixed setting like batch size is a hyperparameter, chosen by you up front and never touched by the optimizer. A raw input feature value is just data flowing into the model, not something the model adjusts. And a reported metric like accuracy or loss is a measurement of performance, not a value living inside the network that gets updated by gradient descent.
Full Explanation
A trainable parameter is a value within a neural network — most commonly a weight or a bias term associated with a layer — that is automatically adjusted during training based on gradients computed through backpropagation. During each training step, the loss function measures how far the model's predictions are from the target values, autograd computes the gradient of that loss with respect to each trainable parameter, and the optimizer (such as SGD or Adam) uses those gradients to update the parameter's value, typically moving it in the direction that reduces the loss. Over many iterations across the training data, this process is what allows the network to 'learn' patterns from data rather than relying on hand-crafted rules.
A fixed configuration value chosen by the engineer before training starts, such as batch size, learning rate, or number of layers, describes a hyperparameter, not a trainable parameter. Hyperparameters govern how the training process itself is configured and are set by the practitioner (sometimes via a tuning search), but they are not updated by the optimizer during training; they remain constant (or follow an externally defined schedule) throughout a given training run.
The raw numeric value of a single input feature before preprocessing is simply data, not a parameter of the model. Input features flow into the network and influence the computations performed using the trainable parameters, but the input values themselves are not adjusted by the optimizer; they are treated as fixed observations for a given sample.
A metric like accuracy or loss computed only for reporting purposes is a measurement used to monitor and evaluate model performance, not a parameter that lives inside the model's architecture. While the loss value does drive the computation of gradients (which in turn update trainable parameters), the loss itself is not a weight or bias being learned — it is the quantity being minimized, and metrics like accuracy are typically not even differentiable or directly involved in the gradient computation at all.
Memory aid: if a value is inside the model's architecture (a weight or bias) and gets updated by the optimizer based on gradients each training step, it is a trainable parameter. If it is set beforehand and left alone during training (like batch size or learning rate), it is a hyperparameter. Keeping this weight/bias-vs-hyperparameter distinction clear is fundamental to understanding how training actually modifies a model.