What defines a Dense (fully connected) layer in a neural network?
Select an answer to reveal the explanation.
Short Explanation
The name gives it away — 'Dense' means densely connected. Every single neuron in a Dense layer takes input from every single neuron (or every input feature) in the layer before it, each with its own learned weight, then sums it all up plus a bias. That's the classic fully connected layer you see in Keras as layers.Dense(units) or in PyTorch as nn.Linear. So 'each neuron connects to every neuron in the previous layer' is correct. Connecting only to a small local patch is describing a convolutional layer instead, which deliberately restricts connections to exploit local spatial structure. Randomly disabling neurons during training is dropout, a totally separate regularization layer. And normalizing activations to zero mean and unit variance is batch normalization, also a distinct layer type with a different job.
Full Explanation
A Dense layer, also called a fully connected layer, is defined by the property that every neuron in the layer receives input from every neuron (or every element of the input vector) in the preceding layer. Each of these connections has its own independently learned weight, and each neuron additionally has a learnable bias term. Mathematically, the output of a Dense layer is computed as a matrix multiplication between the input vector and a weight matrix, plus a bias vector, typically followed by a non-linear activation function (such as ReLU or sigmoid). This full connectivity means a Dense layer with n input features and m neurons has n × m weight parameters plus m bias parameters, and it makes no assumptions about spatial or sequential structure in the input — it treats the input purely as a flat vector of features. Dense layers are the classic building block of the original multilayer perceptron and remain common as the final classification or regression layers in many modern architectures, including CNNs and Transformers, after feature extraction has already reduced the data to a manageable, flattened representation.
Connecting each neuron only to a small local patch of the previous layer's output describes a convolutional layer instead, not a Dense layer. Convolutional layers use small, shared filters (kernels) that slide across the input, deliberately restricting each output neuron's receptive field to a local region, which exploits spatial locality and dramatically reduces the number of parameters compared to full connectivity — the opposite design philosophy from a Dense layer.
Randomly disabling a fraction of neurons during training is the behavior of a Dropout layer, a regularization technique used to reduce overfitting by preventing neurons from co-adapting too strongly; Dropout is typically inserted between other layers (including Dense layers) but is a distinct layer type with its own, separate purpose.
Normalizing activations to zero mean and unit variance describes Batch Normalization, a technique that stabilizes and speeds up training by rescaling layer activations using batch statistics, along with learnable scale and shift parameters; it is also a separate layer type that can be inserted alongside Dense or convolutional layers, but it does not itself define full connectivity.
Understanding that 'Dense' specifically denotes the full-connectivity pattern, as distinct from the local, weight-shared connectivity of convolutions or the stochastic masking of dropout, is essential for correctly reading and designing neural network architectures.