An operations engineer runs the nvidia-smi command during a deep learning training run and notices that the memory usage is hovering near 95%, while the volatile GPU utilization is at 45%. What does the 95% metric specifically represent?
Select an answer to reveal the explanation.
Short Explanation and Infographic
Check this out: a common trap on the exam is confusing GPU memory usage with GPU utilization. Think of it like this: your GPU memory is like a parking lot. If it's 95% full, it means almost all the parking spaces (or VRAM bytes) have cars parked in them. But that doesn't mean those cars are actually driving around! That's why your GPU cores might only be 45% active. When you look at nvidia-smi, the memory metric is purely telling you how much of that high-speed VRAM has been allocated by your ML processes, not how hard the execution engines are working. Pay close attention here, because this one bites people in production all the time!
Full explanation below image
Full Explanation
In GPU monitoring, it is critical to distinguish between GPU utilization (often called volatile GPU utilization) and GPU memory utilization. The nvidia-smi tool reports both metrics, but they measure completely different aspects of hardware activity. GPU memory utilization refers to the amount of physical video RAM (VRAM) that is currently allocated to and occupied by active processes (such as model weights, batch data, and intermediate activations) relative to the total physical memory capacity of the GPU. If a model with a batch size of 32 consumes 15 GB of a 16 GB GPU, the memory utilization will show as approximately 93.75%, regardless of whether the GPU is actively calculating gradients or sitting idle between steps.
Let's look at why the other options are incorrect: Option A is incorrect because the percentage of active GPU processing cores is tracked by volatile GPU utilization (compute utilization). This represents the percentage of time over the past sample period during which one or more kernels were executing on the GPU. Option C is incorrect because PCIe bandwidth usage measures the rate of data transfer between the host system memory and the GPU memory over the PCI Express bus. While this is crucial for detecting bottlenecks during data ingestion, it is not what GPU memory utilization measures. Option D is incorrect because GPU operating temperature and power consumption are separate physical and electrical metrics reported independently by nvidia-smi (e.g., in Celsius and Watts) to help monitor thermal throttling and energy efficiency.