Node Issues · AKS Troubleshooting · microsoft/azure-skills · Skills Docs

Node Issues · AKS Troubleshooting · microsoft/azure-skills · Skills Docs

az aks list \
  --subscription <authorized-subscription-id> \
  --resource-group <authorized-resource-group> \
  --query "[].{id:id,name:name,resourceGroup:resourceGroup,fqdn:fqdn,privateFqdn:privateFqdn}" \
  -o json

AKS_SUBSCRIPTION_ID="<authorized-subscription-id>"
AKS_RG="<cluster-resource-group>"
AKS_NAME="<cluster-name>"
KUBE_CONTEXT="<verified-context>"
NODE="<node-name>"

# Kubernetes description, conditions, and node-scoped events
kubectl --context "$KUBE_CONTEXT" describe node "$NODE"
kubectl --context "$KUBE_CONTEXT" get node "$NODE" \
  -o jsonpath='{range .status.conditions[*]}{.lastTransitionTime}{"\t"}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'
kubectl --context "$KUBE_CONTEXT" get events --all-namespaces \
  --field-selector involvedObject.kind=Node,involvedObject.name="$NODE" \
  --sort-by='.metadata.creationTimestamp'

# AKS node resource group and Kubernetes node to VMSS instance mapping
NODE_RG=$(az aks show \
  --subscription "$AKS_SUBSCRIPTION_ID" \
  --resource-group "$AKS_RG" \
  --name "$AKS_NAME" \
  --query nodeResourceGroup -o tsv)
AGENT_POOL=$(kubectl --context "$KUBE_CONTEXT" get node "$NODE" \
  -o jsonpath='{.metadata.labels.agentpool}')
PROVIDER_ID=$(kubectl --context "$KUBE_CONTEXT" get node "$NODE" \
  -o jsonpath='{.spec.providerID}')
VMSS=$(printf '%s\n' "$PROVIDER_ID" |
  awk -F'/virtualMachineScaleSets/' '{print $2}' | cut -d/ -f1)
INSTANCE_ID=${PROVIDER_ID##*/}

printf 'nodeResourceGroup=%s\nagentPool=%s\nvmss=%s\ninstanceId=%s\n' \
  "$NODE_RG" "$AGENT_POOL" "$VMSS" "$INSTANCE_ID"

# AKS pool state
az aks nodepool show \
  --subscription "$AKS_SUBSCRIPTION_ID" \
  --resource-group "$AKS_RG" \
  --cluster-name "$AKS_NAME" \
  --name "$AGENT_POOL" \
  --query '{provisioningState:provisioningState,powerState:powerState.code,nodeImageVersion:nodeImageVersion,orchestratorVersion:orchestratorVersion}' \
  -o yaml

# Exact VMSS instance view and extension statuses
az vmss get-instance-view \
  --subscription "$AKS_SUBSCRIPTION_ID" \
  --resource-group "$NODE_RG" \
  --name "$VMSS" \
  --instance-id "$INSTANCE_ID" \
  -o json
az vmss get-instance-view \
  --subscription "$AKS_SUBSCRIPTION_ID" \
  --resource-group "$NODE_RG" \
  --name "$VMSS" \
  --instance-id "$INSTANCE_ID" \
  --query 'extensions[].{name:name,statuses:statuses,substatuses:substatuses}' \
  -o json

Condition	Value	Evidence boundary	Next investigation
`Ready`	`False`	Compare the transition time and reason with VMSS instance and extension statuses	Determine whether the failure is kubelet, host, provisioning, or network related before requesting node access
`MemoryPressure`	`True`	Review allocated resources, pod requests/limits, eviction events, and metrics	Identify the workload or node-pool capacity constraint
`DiskPressure`	`True`	Review eviction events, pod ephemeral-storage requests/limits, and node image/OS state	Determine whether workload storage use or node storage capacity is responsible
`PIDPressure`	`True`	Correlate condition transitions with workload placement and process evidence	Use IG `snapshot_process` if process-level evidence is required
`NetworkUnavailable`	`True`	Review CNI pod state and logs plus node NIC routes and NSG evidence	Continue with Networking Troubleshooting

# 1. Operation record first. `az aks operation` is part of the aks-preview extension;
#    the CLI auto-installs it on first use, which may be disallowed on managed hosts.
az aks operation show-latest -g <rg> -n <cluster>
az aks operation show-latest -g <rg> -n <cluster> --nodepool-name <pool>
az aks operation show -g <rg> -n <cluster> --operation-id <id>

# 2. Activity-log fallback (documented Learn path; the only evidence if aks-preview is unavailable)
az monitor activity-log list -g <rg> --resource <cluster-resource-id> --status Failed --offset 24h -o table

# 3. Pool, scale set, and instance provisioning state
az aks nodepool show -g <rg> --cluster-name <cluster> -n <pool> --query '{state:provisioningState,power:powerState.code}'
az vmss show -g <node-rg> -n <vmss> --query provisioningState -o tsv
az vmss list-instances -g <node-rg> -n <vmss> --query '[].{id:instanceId,state:provisioningState}' -o table

# Autoscaler logs
kubectl logs -n kube-system -l app=cluster-autoscaler --tail=100

# Autoscaler status
kubectl get configmap cluster-autoscaler-status -n kube-system -o yaml

# Verify autoscaler is enabled on the node pool
az aks nodepool show -g <rg> --cluster-name <cluster> -n <nodepool> \
  --query "{autoscaleEnabled:enableAutoScaling, min:minCount, max:maxCount}"

# HPA ceiling versus autoscaler: are any pods actually unschedulable?
kubectl get hpa -A
kubectl get pods -A --field-selector=status.phase=Pending

kubectl top node <node>
kubectl top pod -A --containers --sort-by=memory
kubectl get pods -A --field-selector spec.nodeName=<node> \
  -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,CPU_LIMIT:.spec.containers[*].resources.limits.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory,MEM_LIMIT:.spec.containers[*].resources.limits.memory'
kubectl describe node <node>

Signal	Meaning
`MemoryPressure=True` plus eviction events	Node-wide memory pressure; identify consumers and reservation/capacity before changing pod limits
Container `OOMKilled` without node pressure	Container crossed its cgroup memory limit; compare the affected container, sidecars, and previous logs
CPU usage near a container limit with throttling telemetry	Limit-induced throttling is plausible; CPU usage alone does not prove throttling
Requests near allocatable but low live usage	Scheduler capacity constraint, not runtime pressure

# Correlate node pressure with workload requests, limits, placement, and events
kubectl describe node <node>
kubectl get pods -A --field-selector spec.nodeName=<node> \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,EPHEMERAL_REQUESTS:.spec.containers[*].resources.requests.ephemeral-storage,EPHEMERAL_LIMITS:.spec.containers[*].resources.limits.ephemeral-storage'
kubectl get events --all-namespaces \
  --field-selector involvedObject.kind=Node,involvedObject.name=<node> \
  --sort-by='.metadata.creationTimestamp'