A team needs to run a trained image classifier on a low-power embedded sensor that has no GPU. What is the most effective way to speed up inference in this environment?
Select an answer to reveal the explanation.
Short Explanation
You've got a model that runs fine on your dev machine but crawls on a tiny sensor with no GPU — this is a classic deployment problem, not a training problem. The fix is to convert the model into a format built for fast, lightweight inference, like TensorFlow Lite or ONNX. These formats strip out training-only overhead, apply optimizations like operator fusion and quantization, and are built to run efficiently on CPUs and constrained hardware. Bigger batch size only helps during training throughput, not single-sample inference speed on a tiny device. Adding more layers makes the model heavier, which is the opposite of what you want here. And swapping the loss function affects how the model learns, not how fast it predicts. When someone says 'no GPU, needs to be fast,' think conversion and optimization, not retraining.
Full Explanation
When deploying a trained model to hardware without a GPU — embedded sensors, microcontrollers, or edge devices — the primary bottleneck is inference efficiency, not model accuracy. Converting the trained model to a format such as TensorFlow Lite or ONNX is the correct approach because these runtimes are purpose-built for efficient execution on CPUs and specialized edge accelerators. The conversion process typically applies graph optimizations (operator fusion, constant folding), removes training-only nodes, and enables quantization (e.g., converting 32-bit floats to 8-bit integers), all of which dramatically reduce compute and memory demands while preserving most of the model's accuracy.
Increasing the batch size is incorrect because larger batches primarily improve throughput during training on hardware with parallel compute capacity (typically GPUs); at inference time on a single-sample, GPU-less device, batch size has little to no bearing on latency, and a larger batch can actually increase memory pressure.
Adding more layers to improve accuracy is counterproductive in this scenario: deeper networks require more parameters, more multiply-accumulate operations, and more memory, all of which further slow down inference on constrained hardware. The goal here is efficiency, not marginal accuracy gains, and this choice moves in the wrong direction entirely.
Changing the loss function is unrelated to inference speed. The loss function only governs how gradients are computed and parameters are updated during training; it has no bearing on the computational cost of a forward pass once training is complete.
The underlying principle: deployment optimization and training optimization are different problems. When compute is constrained at inference time, the standard toolkit is model conversion (TFLite, ONNX, TensorRT), quantization, and pruning — not architectural changes aimed at accuracy or training hyperparameter changes.