Setting the file. One moment.
Subchapter 2.2
references/gpu-observability.mdMarkdown3 KBView on GitHub
Bind telemetry to the affected cluster, namespace, pod, container, GPU, and incident window. Keep host/container-memory OOM and CUDA device-allocator OOM as separate live hypotheses. A Kubernetes OOM symptom alone does not prove GPU memory exhaustion, and a CUDA allocator error does not prove which workload condition exhausted the device. Exclude model-size calculations and GPU SKU-sizing tables, not container or host memory evidence.
kubectl get pod -n <namespace> <pod> -o json
kubectl logs -n <namespace> <pod> -c <container> --previous --timestamps
kubectl get node <gpu-node> -o json
az aks nodepool show --subscription <subscription> -g <resource-group> \
--cluster-name <cluster> -n <pool> --query gpuProfile -o jsonFor an observed managed profile (managementMode: Managed,
driver: Install), AKS exposes its managed DCGM exporter on port 19400 and
labels enabled nodes kubernetes.azure.com/dcgm-exporter=enabled. For an
unmanaged or unknown profile, inspect the installed exporter’s declared port;
do not copy the managed port assumption.
Correlate the model-load timestamp and container termination with:
| Signal | Use |
|---|---|
DCGM_FI_DEV_FB_USED | Frame-buffer memory consumed before failure |
DCGM_FI_DEV_FB_FREE | Remaining frame-buffer memory before failure |
DCGM_FI_DEV_GPU_UTIL | Kernel-active time; not an efficiency score |
Termination reason OOMKilled (often exit 137) | Supports a container-memory path; exit 137 alone is insufficient |
torch.cuda.OutOfMemoryError or CUDA allocation failure in model logs | Supports a device-allocation path; correlate with per-GPU incident-window telemetry |
If FB_USED rises while FB_FREE approaches exhaustion in the same GPU and
incident window, the evidence supports VRAM pressure. Healthy aggregate
frame-buffer values outside that GPU and incident window do not establish a
cause. If historical telemetry is absent, keep device exhaustion unconfirmed.
If the container was OOMKilled, inspect its memory limit/usage and node
memory conditions rather than discarding the host-memory path.
When the user supplied evidence and prohibited commands, state that no reads ran. Require explicit authorization before monitoring mutation, restart, scaling, cordon/drain, or node-pool changes. Propose a target-bound telemetry collection plan instead.
Sources: managed exporter port (opens in a new tab), GPU observability (opens in a new tab), PyTorch CUDA OOM (opens in a new tab).