Setting the file. One moment.
Skill 05 · AKS Troubleshooting
Subchapter 5.17
pod-failures.mdMarkdown8 KBView on GitHub
Use the shipped collector instead of streaming raw logs into model context. It verifies the kube context against the named AKS resource before any Kubernetes API read, captures current and previous logs across all containers, and keeps the raw log files in a caller-selected artifact directory.
AKS_SUBSCRIPTION_ID=<subscription-id> \
scripts/pod-deep-dive.sh \
<namespace> <pod-name> <resource-group> <cluster-name> <kube-context> \
<artifacts-directory>The model-visible projection redacts credential-shaped values and is capped at 50 lines per current/previous log stream. Inspect expanded raw evidence only outside model context.
Pod starts, crashes, restarts with exponential backoff (10s, 20s, 40s… up to 5m).
Diagnostics:
kubectl describe pod <pod-name> -n <namespace>
# Check: Exit Code, Reason, Last State, EventsUse the Previous Logs section from pod-deep-dive.sh; it preserves the last crashed container output without exposing the raw stream.
Decision tree:
| Exit Code | Meaning | Fix Path |
|---|---|---|
0 | App exited successfully (unexpected for long-running) | Check if entrypoint/command is correct; app may be a one-shot |
1 | Application error | Read logs - unhandled exception, missing config, bad startup |
137 | OOMKilled (SIGKILL) | Increase resources.limits.memory; check for memory leaks |
139 | Segfault (SIGSEGV) | Binary compatibility issue or native code bug |
143 | SIGTERM - graceful shutdown | Pod was terminated; check if liveness probe killed it |
OOMKilled specifically:
kubectl describe pod <pod-name> -n <namespace> | grep -A2 "Last State"
# Reason: OOMKilled -> container exceeded memory limitFix: increase resources.limits.memory or optimize application memory usage. Check kubectl top pod <pod-name> -n <namespace> for actual usage.
OOM kill tracing with Inspektor Gadget: Use trace_oomkill (timeout 30) with --k8s-namespace <namespace> --k8s-podname <pod-name> to see which process was killed and memory at kill time. See references/inspektor-gadget.md.
Deep diagnostics with Inspektor Gadget (when logs and describe are inconclusive):
Use the IG base command pattern with --k8s-namespace <namespace> --k8s-podname <pod-name> and these gadgets:
trace_exec (timeout 30) — see what the container executes at startuptrace_open (timeout 30) — find missing configs/secrets (retval -2 = ENOENT, -13 = EACCES)snapshot_process (timeout 5) — list running processes in the podPod can’t pull the container image.
Diagnostics:
kubectl describe pod <pod-name> -n <namespace>
# Events section shows the exact pull error| Error Message | Cause | Fix |
|---|---|---|
ErrImagePull / ImagePullBackOff | Image name or tag is wrong | Verify image name and tag exist in the registry |
unauthorized: authentication required | Missing or wrong pull secret | Create/update imagePullSecrets on the pod or service account |
manifest unknown | Tag doesn’t exist | Check available tags in the registry |
context deadline exceeded | Registry unreachable | Check network/firewall; for ACR, verify AKS -> ACR integration |
ACR integration check:
# Verify AKS is attached to ACR
az aks check-acr -g <rg> -n <cluster> --acr <acr-name>.azurecr.ioPod stays in Pending - scheduler can’t place it.
Diagnostics:
kubectl describe pod <pod-name> -n <namespace>
# Events section shows why scheduling failed| Event Message | Cause | Fix |
|---|---|---|
Insufficient cpu / Insufficient memory | No node has enough resources | Scale node pool; reduce resource requests; check for overcommit |
node(s) had taint ... that the pod didn't tolerate | Taint/toleration mismatch | Add matching toleration or use a different node pool |
node(s) didn't match Pod's node affinity/selector | Affinity rule unsatisfiable | Check nodeSelector or nodeAffinity rules |
persistentvolumeclaim ... not found / unbound | PVC not ready | Check PVC status; verify storage class exists |
0/N nodes are available: N node(s) had volume node affinity conflict | Zonal disk vs pod in different zone | Use ZRS storage class or ensure same zone |
Readiness probe failure -> pod removed from Service endpoints (no traffic). Liveness probe failure -> pod killed and restarted.
Diagnostics:
kubectl describe pod <pod-name> -n <namespace>
# Look for: "Readiness probe failed" or "Liveness probe failed" in Events
# Check the pod's READY column - must show n/n
kubectl get pod <pod-name> -n <namespace>| Symptom | Cause | Fix |
|---|---|---|
READY shows 0/1 but pod is Running | Readiness probe failing | Check probe path, port, and app health endpoint |
| Pod restarts repeatedly | Liveness probe failing | Increase initialDelaySeconds; check if app starts slowly |
| Probe timeout errors | App responds too slowly | Increase timeoutSeconds; check app performance |
💡 Tip: Set
initialDelaySecondson liveness probes to be longer than your app’s startup time. A common mistake is killing pods before they finish initializing.
Check actual usage vs limits:
kubectl top pod <pod-name> -n <namespace>
kubectl top pod -n <namespace> --sort-by=memory
# Compare with requests/limits
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[*].resources}'| Symptom | Cause | Fix |
|---|---|---|
| OOMKilled (exit code 137) | Container exceeded memory limit | Increase limits.memory or fix memory leak |
| CPU throttling (slow responses) | Container hitting CPU limit | Increase limits.cpu or remove CPU limits |
| Pending - insufficient resources | Requests exceed available node capacity | Lower requests, scale nodes, or use larger VM sizes |
⚠️ Warning: Setting CPU limits can cause unnecessary throttling even when the node has spare capacity. Many teams set CPU requests but not limits. Memory limits should always be set.