Networking

First define the failing flow: source pod, destination (Service, pod, or host), port and protocol, and direction. Walk DNS → Service → Endpoints → pod → network policy/CNI → Azure NSG/UDR/egress, and stop at the first layer that blocks. After any fix, re-test the same flow from the same source.

Service Unreachable / Connection Refused

Diagnostics - always start here:

Decision tree:

Pods that are running but not Ready are removed from Endpoints. Check kubectl get pod <pod> -n <ns>.

Deep diagnostics with Inspektor Gadget (when the above checks are inconclusive):

Use the IG base command pattern with --k8s-namespace <ns> --k8s-podname <pod-name> and these gadgets:

• snapshot_socket (timeout 5) — check what ports the pod is listening on
• trace_tcp (timeout 30) — trace connect/accept/close events
• trace_tcpretrans (timeout 30) — packet retransmissions

See references/inspektor-gadget.md.

DNS Resolution Failures

Use the affected pod; do not create a test pod or change DNS, NetworkPolicy, NSG, route, or firewall configuration without explicit approval.

Select the resolver path first

Inspect the affected pod’s resolver inputs and determine whether its node pool uses LocalDNS before assuming every query goes directly through kube-dns and CoreDNS. AKS Automatic preconfigures LocalDNS; AKS Standard can enable it per node pool.

If LocalDNS is enabled, scope the failure before changing anything: all pools or one pool, all nodes or one node, UDP or TCP, and cluster zones or external zones. Inspect the effective LocalDNS forwarding configuration and logs on the affected node using an approved node-access method. LocalDNS forwards different zones to CoreDNS or VNet DNS, so a node-, pool-, protocol-, or zone-specific failure can be local even when CoreDNS is healthy. Query logging has resource cost and requires approval. Updating a node pool’s LocalDNS configuration can reimage nodes; disclose that impact before proposing it.

If LocalDNS is disabled, or once its forwarding path reaches CoreDNS, continue with the CoreDNS branch:

Repeat these checks for each node hosting a CoreDNS pod. For Azure CNI Pod Subnet, COREDNS_SOURCE_SUBNET_ID is the pod subnet; otherwise it is the CoreDNS node NIC subnet. Record an NSG or route table as absent only when its association lookup succeeds with an empty ID; record a failed lookup as inaccessible instead of running its dependent show command. Evaluate outbound rules and routes from the CoreDNS pod/node source to every configured upstream on both UDP and TCP destination port 53.

If the selected route has a VirtualAppliance next hop, run the exact policy or classic network-rule collection commands in Mandatory Firewall or NVA branch when traversed, then query the matching DNS decisions:

For a custom NVA, collect equivalent network-rule and log evidence filtered to the same CoreDNS source, upstream destination, incident window, and UDP/TCP 53.

If CoreDNS imports a custom ConfigMap, retrieve the named ConfigMap shown by the inventory before evaluating its forwarding rules. A successful direct query to a custom forwarder with a failed query through kube-dns points to CoreDNS configuration or service-path evidence; failure to reach the forwarder points to routing, NSG, firewall, or NetworkPolicy evidence.

Do not name UDP source-port reuse as the cause of CoreDNS imbalance until logs show the same client source IP and port repeatedly reaching one replica.

DNS failure patterns:

Changing CoreDNS replicas/configuration, custom forwarders, NetworkPolicy, NSGs, routes, or firewall rules is remediation and requires explicit approval.

Sources: LocalDNS troubleshooting (opens in a new tab), LocalDNS configuration (opens in a new tab), and CoreDNS troubleshooting (opens in a new tab).

Deep diagnostics with Inspektor Gadget (when the above checks are inconclusive):

Use the IG base command pattern with --k8s-namespace <ns> --k8s-podname <pod-name> and trace_dns (timeout 30). Key signals: rcode=3 (NXDOMAIN), rcode=2 (SERVFAIL), high latency values, queries going to unexpected destinations.

See references/inspektor-gadget.md.

AKS to an External Azure Service

Use references/azure-network-path.md to derive the source node/VMSS NIC, prove the effective NSG and route, and inspect the selected UDR, NAT gateway, load-balancer outbound rule, firewall/NVA, private endpoint, service endpoint, or peering path. It includes current Network Watcher show-next-hop and test-ip-flow syntax plus their AKS VMSS limitations.

Before concluding, require:

1. DNS and TCP evidence from the affected pod using tools already present;
2. the cluster network profile and outbound type;
3. effective node-NIC NSG and route evidence;
4. the target service’s public-access, firewall/VNet-rule, or private-endpoint configuration; and
5. matching firewall/NVA policy and incident-window logs when the selected route traverses a stateful appliance.

Record a skipped branch and why. A successful empty association means absent; a failed lookup means inaccessible, not absent. Do not change DNS, endpoint policy, NSGs, UDRs, peering, NetworkPolicy, or firewall/NVA rules until the first blocking layer is proven and remediation is explicitly approved.

Detailed Networking Guides

• Load Balancer And Ingress Troubleshooting for pending services, ingress controller state, backend routing, and TLS failures.
• Network Policy Troubleshooting for default-deny checks, Azure NPM or Calico validation, and ingress or egress rule audits.
• AKS Azure Network Path Evidence for node NIC/VMSS mapping, effective routes and NSGs, Network Watcher, egress/SNAT, private endpoints, service endpoints, and peering.