Housing LoRA has only two GPUs and needs a larger effective batch. Which technique mimics that batch?
Select an answer to reveal the explanation.
Short Explanation
Two GPUs, big dreams: accumulate sixteen micro-steps before the optimizer step and housing LoRA behaves like a larger batch without buying more cards.
Full Explanation
Housing LoRA on only two GPUs that needs a larger effective batch should use gradient accumulation—for example accumulation of 16—so microbatches run forward and backward without an optimizer step until enough gradients accumulate to mimic the larger target batch under limited memory. Effective batch grows without requiring each step’s microbatch to fit the full global batch in memory at once. Cutting accumulation to one and hoping memory expands does not create a larger effective batch and leaves the two-GPU limit unchanged. Raising per-step batch until both GPUs OOM immediately fails the job instead of simulating a larger batch through accumulation. Disabling gradients so no accumulation math applies stops learning altogether and cannot mimic any batch size. Exam caveat: accumulation changes how often the optimizer steps, not the instantaneous microbatch memory footprint—in limited-GPU civic settings it is the standard way to hit a target global batch. Operational check: set microbatch to what two GPUs stably hold, set accumulation steps so microbatch × accumulation × data-parallel degree equals the desired global batch with no remainder, verify that effective-batch math in logs, and compare housing LoRA loss stability to a larger-GPU baseline when available.