A public-health Run:ai training job requests more GPUs than the project quota allows. What should operators expect?
Select an answer to reveal the explanation.
Short Explanation
You can expect the public-health training job to stay pending until reclaim frees quota within the project limit; that’s fair-share scheduling, not a failure. You won’t get an automatic unlimited GPU grant, and nvidia-smi can’t change project policy.
Full Explanation
When a public-health Run:ai training job requests more GPUs than its project quota allows, expect it to stay pending until reclaim frees quota within the project limit. Project quotas are admission and scheduling controls, so an over-quota request does not silently expand capacity; pending is the visible signal that demand exceeds the department’s allocation. Capacity may become available as work finishes or is reclaimed, while an administrator can raise the quota through policy. Automatically overriding the quota and assigning every requested GPU without approval would defeat multi-tenancy: quotas prevent one project from capturing unbounded resources. Running nvidia-smi on a node cannot permanently raise the project quota; it inspects devices but does not change Run:ai control-plane policy. Quota checks are not limited to inference: training submits also consume project GPU quotas. Exam caveat: pending until reclaim does not mean the job is broken or that operators must delete peer projects. Separate a quota wait from misconfiguration, such as the wrong project, zero quota, or a request larger than the cluster’s entire capacity; reclaim cannot make an impossible request runnable. Practical check: compare the requested GPUs with the project quota and current allocation in Run:ai, confirm the pending reason indicates quota or resources, and free or reclaim capacity within policy before expecting the training job to start.