Setting the file. One moment.
Skill 05 · AKS Troubleshooting
Subchapter 5.6
references/inspektor-gadget.mdMarkdown8 KBView on GitHub
Use Inspektor Gadget for real-time, low-level node/pod diagnostics when kubectl is insufficient.
Linux nodes only; kernel 5.10+ with BTF; not Kata-sandboxed pods. Admission policy can deny privileged pods. If eBPF fails to load or a pod is denied, record the evidence as unavailable and do not relax policy.
kubectl get ds gadget -n gadget -o jsonpath='{.spec.template.spec.containers[0].image}'. The AKS extension microsoft.inspektorgadget also creates this DaemonSet.kubectl gadget present (preferred; no privileged pod): require kubectl auth can-i create pods/portforward -n gadget. Match the gadget tag to kubectl gadget version, because a mismatched version fails to load. Run kubectl --context <kube-context> gadget run <gadget>:<server-version> -n <ns> [-p <pod>] --timeout <s> -o json > <raw-artifact>.json. Never use --detach, which creates persistent instances.kubectl auth can-i create pods -n <debug-ns> and explicit approval.kubectl gadget deploy, Helm, or az k8s-extension create). Propose installation to the owner as a separate change.If a check fails, name the identity and the missing verb, then stop that path.
mcr.microsoft.com/oss/v2/inspektor-gadget/ig:v0.51.0@sha256:6610863f6d8cae28800f9331756434639bca44be065719cbcfe76e34c91dffa4
Do not replace the digest with a tag-only reference. Re-review the upstream release and platform manifests before changing either the version or digest.
The incident owner must explicitly approve privileged debug-pod creation and choose a finite outer deadline before this command is run:
IG_IMAGE='mcr.microsoft.com/oss/v2/inspektor-gadget/ig:v0.51.0@sha256:6610863f6d8cae28800f9331756434639bca44be065719cbcfe76e34c91dffa4'
timeout <approved-deadline> \
kubectl --context <kube-context> debug --profile=sysadmin \
node/<node-name> --attach --quiet --image="$IG_IMAGE" -- \
ig run <gadget>:v0.51.0 -o json --timeout <gadget-seconds> [filters...] \
> <raw-artifact>.jsonUse both bounds: the outer deadline caps the whole kubectl debug operation, and the inner IG timeout caps the gadget. Use --timeout 5 for snapshot/top and --timeout 30 for trace/profile. Record the generated debug-pod name and delete that exact pod within the same approved operation.
Note: The gadget observes read-only state, but creating and deleting a
--profile=sysadmindebug pod are privileged cluster mutations. Approval and appropriate RBAC are mandatory.
Required: Resolve the node name first:
kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.nodeName}'| Filter | Description |
|---|---|
--k8s-namespace <ns> | Scope to a Kubernetes namespace |
--k8s-podname <pod> | Scope to a specific pod |
--k8s-containername <ctr> | Scope to a specific container |
--timeout <seconds> | Cap streaming duration for trace/profile gadgets |
--max-entries <n> | Max entries per batch for top/profile gadgets |
--map-fetch-interval <dur> | Map fetch interval for top (except top_process) and profile gadgets (default 1000ms) |
--interval <dur> | Reporting interval for top_process only (e.g. 5s) |
--syscall-filters <list> | Comma-separated syscalls for traceloop (e.g. open,connect,accept). Always specify to limit data volume |
Tip: For top/profile, set
--map-fetch-interval≤ half of--timeoutto collect at least one batch. E.g.--timeout 2 --map-fetch-interval 1000ms --max-entries 20.Note:
top_processuses--intervalinstead of--map-fetch-interval. E.g.--timeout 10 --interval 5s --max-entries 20.
| Gadget | Type | What It Does | When To Use |
|---|---|---|---|
trace_dns | trace | Trace DNS queries and responses with latency | DNS failures, NXDOMAIN, SERVFAIL, slow resolution, intermittent DNS |
trace_tcp | trace | Trace TCP connect/accept/close events | Connection refused, timeouts, unexpected drops, mapping pod connectivity |
trace_tcpretrans | trace | Trace TCP retransmissions | Network congestion, lossy links, high latency between pods/services |
trace_bind | trace | Trace socket bind calls | Port conflicts, address-already-in-use errors |
trace_sni | trace | Trace TLS SNI (Server Name Indication) values | HTTPS routing issues, ingress TLS debugging, mTLS problems |
snapshot_socket | snapshot | List open sockets (TCP/UDP/Unix) | Port conflicts, listening ports, connection leaks, ECONNREFUSED |
tcpdump | special | Capture raw packets in pcap-ng format | Deep packet inspection, protocol-level debugging, reproducing network issues |
Keep raw pcap-ng outside model context:
IG_IMAGE='mcr.microsoft.com/oss/v2/inspektor-gadget/ig:v0.51.0@sha256:6610863f6d8cae28800f9331756434639bca44be065719cbcfe76e34c91dffa4'
timeout <approved-deadline> \
kubectl --context <kube-context> debug --profile=sysadmin \
node/<node-name> --attach --quiet --image="$IG_IMAGE" -- \
ig run tcpdump:v0.51.0 -o pcap-ng \
--k8s-namespace <ns> --k8s-podname <pod> \
--timeout 30 --pf "port 80" \
> <raw-artifact>.pcapngUse --pf "<expr>" for a narrow tcpdump filter (for example, port 80 or host 10.0.0.1). Output must be -o pcap-ng (not -o json). Inspect the artifact with an approved offline packet-analysis tool rather than pasting packet payloads into model context.
| Gadget | Type | What It Does | When To Use |
|---|---|---|---|
snapshot_process | snapshot | List running processes in pod/node | PID pressure, unknown processes, verifying entrypoint, CrashLoopBackOff |
trace_exec | trace | Trace process execution (execve calls) | CrashLoopBackOff (what actually runs), unexpected child processes, security audit |
trace_oomkill | trace | Trace OOM kill events with victim details | OOMKilled pods — see which process was killed, memory usage at kill time |
trace_signal | trace | Trace signals delivered to processes | Unexpected SIGKILL/SIGTERM, liveness probe kills, graceful shutdown issues |
top_process | top | Rank processes by CPU/memory usage | Identifying resource-hungry processes inside a pod or across a node |
profile_cpu | profile | CPU profiling via stack sampling | High CPU usage investigation, finding hot code paths |
traceloop | trace | Record syscalls as a flight recorder | Catch-all for intermittent issues. Always use --syscall-filters (e.g., open,connect,accept) to limit data volume |
| Gadget | Type | What It Does | When To Use |
|---|---|---|---|
trace_open | trace | Trace openat syscall | Missing config/secret files (ENOENT), permission denied (EACCES), startup failures |
trace_fsslower | trace | Trace slow filesystem operations | Slow disk I/O, PVC performance issues, NFS/Azure Disk latency |
top_file | top | Rank files by read/write activity | Identifying I/O-heavy files, noisy log writers, disk pressure diagnosis |
| Gadget | Type | What It Does | When To Use |
|---|---|---|---|
trace_capabilities | trace | Trace Linux capability checks | Permission denied from dropped capabilities, SecurityContext debugging |