Setting the file. One moment.
Skill 05 · AKS Troubleshooting
Subchapter 5.1
references/api-server-webhooks-tunnel.mdMarkdown6 KBView on GitHub
Bind every read to the named AKS cluster, affected object, nodes, and incident window. Use supplied evidence and approved host capabilities; unavailable telemetry is unknown, not absent.
For API latency, timeouts, HTTP 429, or “unable to handle the request,” inspect Resource Health and AKS control-plane diagnostics when available, then correlate the affected command, recent bulk operations, and high-volume list/watch clients. Distinguish customer-driven request load from a platform condition. If the remaining evidence requires service-side control-plane access, stop and escalate with the cluster, timestamps, symptoms, correlation IDs, and completed read-only checks.
kubectl get flowschemas
kubectl get prioritylevelconfigurations
kubectl get flowschema aks-managed-apiserver-guard -o yaml
kubectl get events -n kube-system --field-selector type=Warning --sort-by=.lastTimestampFlowSchema and PriorityLevelConfiguration named
aks-managed-apiserver-guard mean AKS is throttling non-system client
requests as a last-resort protection after repeated API-server OOM events. Its
presence explains that requests are being throttled; it does not by itself
identify the root cause, which is still excessive client load, oversized
list/watch patterns, or etcd object growth. Report it as the throttling
mechanism, keep reading evidence, and never delete or relabel the guard
objects: that is a mutation the owner must approve after the load source is
fixed.# Metric name varies by Kubernetes version; the grep matches all three
kubectl get --raw /metrics | grep -E "etcd_db_total_size_in_bytes|apiserver_storage_size_bytes|apiserver_storage_db_total_size_in_bytes"
kubectl get --raw /metrics | grep -E '^apiserver_storage_objects' | sort -t' ' -k2 -nr | head -20
kubectl get events -A --sort-by=.lastTimestamp | tail -50ownerReferences,
managedFields manager, or a creating controller’s logs). Name the kind and
the suspected owner, quantify the growth, and stop. Do not delete objects;
cleanup, retention, and ResourceQuota object quotas are owner decisions.kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations -o yaml
kubectl get service,endpoints,endpointslice -n <webhook-namespace>
kubectl get pods -n <webhook-namespace> -o wideFor the named webhook, inspect failurePolicy, timeoutSeconds,
namespaceSelector, backing Service, endpoints, and ready pods. A failing
webhook with failurePolicy: Fail can block matching operations, but an error
string alone does not prove the backend cause or blast radius. Never delete,
patch, or weaken a webhook until the backend state is proven and the owner
approves the remediation.
If an admission denial or an unexpected pod mutation (CPU/memory requests or
limits set or raised, pod anti-affinity or topology spread constraints added)
does not match any customer ValidatingWebhookConfiguration or
MutatingWebhookConfiguration, check whether AKS Deployment Safeguards is
enabled at the Enforce level before attributing it to a customer webhook.
Safeguards is enabled and set to Enforce by default on AKS Automatic clusters
(namespaces can be excluded, but the level cannot be changed to Warn) and is
optional Warn or Enforce on AKS Standard, so require the cluster’s actual
Safeguards state rather than partitioning on Automatic versus Standard:
az aks safeguards show -g <rg> -n <cluster> # aks-preview extension: level, excluded namespaces, PSS level
kubectl get constraints # Gatekeeper constraints active on the clusterReport the matching Safeguards policy or mutator by name and leave the
enablement level, excluded namespaces, and workload manifest changes to the
owner. Policy-by-policy manifest compatibility for AKS Automatic belongs to the
nested azure-kubernetes Automatic readiness reference; hand off there when the
question becomes “will this manifest be admitted.”
If kubectl logs, exec, port-forward, or webhook calls fail only for pods
on some nodes while ordinary API reads work, test the node-placement pattern
and inspect the konnectivity-agent pods in kube-system. Port 10250 errors
can indicate the API-server-to-kubelet tunnel path rather than pod networking.
Inspect the affected nodes’ NSG, firewall, UDR, and tunnel-component evidence
read-only. Do not change node firewall or iptables state; escalate service-side
or unsupported node conditions with the collected evidence.
Exact nested VMExtensionError_K8SAPIServerConnFail / CSE exit 51 belongs to
aks-known-issues. Explicit packet capture belongs to aks-network-capture.
Sources: API server and etcd troubleshooting (opens in a new tab), tunnel connectivity (opens in a new tab), admission-controller triage (opens in a new tab), Deployment Safeguards (opens in a new tab), and AKS support policies (opens in a new tab).