k8s-debug

intent

use this skill to systematically diagnose and fix kubernetes cluster issues across pod lifecycles, networking, storage, and node health. run it when pods are stuck in crashloopbackoff, pending, or imagepullbackoff states; when services fail dns resolution; when pvcs won't bind; when deployments won't rollout; or when cluster health degrades after a change. the skill provides deterministic, read-first workflows with safety gates before destructive operations.

inputs

required

• kubectl installed and in PATH
• active cluster context (verify: kubectl config current-context)
• RBAC permissions: get pods, get events, get services, get nodes, get pvc, get pv, get deployments across namespaces
• preflight check: run kubectl auth can-i get pods -A and kubectl auth can-i get events -A

optional but strongly recommended

• jq installed for json filtering in cluster_health.sh (script continues without it; output is less readable)
• metrics-server running (for kubectl top commands; scripts fall back to kubectl describe and events if unavailable)
• in-pod debug tools: nslookup, getent, curl, wget, ip (required only for deep network tests; scripts note if missing)
• python3 for pod_diagnostics.py (required for deep pod diagnostics)

environment variables (optional)

• K8S_REQUEST_TIMEOUT: timeout in seconds for kubectl calls (default: 30s). set to higher value for slow clusters or high latency.

procedure

step 1: preflight and scope

inputs: cluster context, target namespace

outputs: confirmed access level, scope narrowed to namespace or pod

1. verify cluster context: kubectl config current-context
2. list available namespaces: kubectl get ns
3. confirm read access to target namespace: kubectl auth can-i get pods -n <namespace>
4. if access check fails, stop here. fix kubeconfig, rbac, or context before proceeding.
5. if scope is single pod, get pod status: kubectl get pod <pod-name> -n <namespace> -o wide

step 2: identify the problem layer

inputs: symptom description, pod/service/cluster status output

outputs: layer identified (application, pod, service, node, cluster, storage, configuration)

categorize the issue by symptoms:

• application layer: app crashes, throws errors, exits with non-zero code
• pod layer: pod not starting, restarting (restarts > 0), or stuck in pending
• service layer: endpoints not registered, service dns not resolving, service ips unreachable
• node layer: node not ready, resource exhaustion (memory/cpu), disk pressure
• cluster layer: control plane unhealthy, api server slow/unavailable
• storage layer: pvc pending, pv not provisioned, mount failures in logs
• configuration layer: configmap/secret missing, rbac denies, resource limits misconfigured

step 3a: pod-level diagnostics (if issue is pod-scoped)

inputs: pod name, namespace (optional, defaults to "default")

outputs: detailed pod report (status, events, logs, resource usage, node context)

1. run deep diagnostics: python3 ./scripts/pod_diagnostics.py <pod-name> -n <namespace>
2. optionally save to file: python3 ./scripts/pod_diagnostics.py <pod-name> -n <namespace> -o diagnostics-<pod-name>.txt
3. script gathers:
   • pod status and description
   • pod creation events (newest first)
   • container logs (current and previous if available)
   • resource usage (memory, cpu if metrics-server present)
   • node name and node description
   • full pod yaml configuration
4. script exits with code 0 if all data collected, 1 if warnings occur, 2 if access denied or pod not found
5. review logs for stack traces, error messages, or segfaults
6. review events for imagepull failures, crashloopbackoff reasons, or pending triggers

step 3b: cluster-level health check (if issue is cluster-wide or scope unclear)

inputs: none required, optional K8S_REQUEST_TIMEOUT environment variable

outputs: cluster health report (nodes, workloads, failed pods, events, component status)

1. run cluster health: ./scripts/cluster_health.sh > cluster-health-$(date +%Y%m%d-%H%M%S).txt
2. optional: run in strict mode (treat warnings as failures): ./scripts/cluster_health.sh --strict
3. script checks:
   • kubernetes version and cluster info
   • node status, capacity, and allocatable resources
   • pods in all namespaces (group by phase: running, pending, failed, unknown)
   • recent events sorted by timestamp
   • deployment, statefulset, daemonset status
   • pvc and pv status
   • control plane component health (if visible)
   • common error states (e.g., how many pods in crashloopbackoff)
4. script exits with code 0 if all checks passed (warnings ok unless --strict), 1 if check failures, 2 if blocked (missing kubectl, no context)
5. identify any nodes marked notready, pods pending > 5 minutes, or control plane components down

step 3c: network diagnostics (if issue involves service connectivity or dns)

inputs: pod name, namespace (optional, defaults to "default")

outputs: network diagnostics report (dns resolution, service endpoints, network policies, connectivity tests)

1. run network debug: ./scripts/network_debug.sh <pod-name> -n <namespace>
2. optional: strict mode (warnings are failures): ./scripts/network_debug.sh --strict <pod-name> -n <namespace>
3. optional: insecure TLS (skip cert validation): ./scripts/network_debug.sh --insecure <pod-name> -n <namespace>
4. script analyzes:
   • pod network namespace and interfaces
   • dns server configuration in pod (/etc/resolv.conf)
   • dns resolution of kubernetes.default and custom service names
   • service endpoints (verify they match pod ips)
   • networkpolicies that apply to this pod
   • connectivity tests to service ips, pod ips, external endpoints
   • coredns logs (if dns resolution failed)
5. script exits with code 0 if all probes succeed, 1 if any probe fails or warnings in --strict mode, 2 if access denied or pod unreachable
6. note any failed name resolution, endpoint mismatches, or networkpolicy blocks

step 4: follow issue-specific reference workflow

inputs: identified layer and symptoms, diagnostics output

outputs: targeted troubleshooting steps, candidate root causes

1. consult ./references/troubleshooting_workflow.md for your issue type:
   • pod pending: scheduling workflow (node affinity, resource requests, taints)
   • crashloopbackoff: application crash workflow (logs, liveness probe, dependencies)
   • imagepullbackoff: image pull workflow (registry credentials, image name, network)
   • service issues: network connectivity workflow (endpoints, service ips, ingress)
   • dns failures: dns workflow (coredns health, resolv.conf, service names)
   • resource exhaustion: performance investigation workflow (top pods/nodes, limits)
   • storage issues: pvc binding workflow (storage class, provisioner, node selectors)
   • deployment stuck: rollout workflow (replica status, pod template errors, selector mismatch)
2. follow the numbered steps in the matching section
3. cross-reference with ./references/common_issues.md for known fixes

step 5: apply targeted fixes

inputs: root cause identified, reference workflow completed

outputs: issue resolved or escalated with clear next steps

1. review candidate fixes in ./references/common_issues.md under your symptom heading
2. before any disruptive operation (delete, drain, restart, rollout undo): a. snapshot current state: kubectl get deploy,rs,pod,svc -n <namespace> -o wide > before-snapshot.txt b. save pod yaml: kubectl get pod <pod-name> -n <namespace> -o yaml > before-<pod-name>.yaml c. save events: kubectl get events -n <namespace> --sort-by='.lastTimestamp' > before-events.txt d. note incident context (what changed, when, who, why)
3. execute the fix (see decision points below for confirmation requirements)
4. proceed to step 6 for verification

step 6: verify and close

inputs: fix applied, cluster state

outputs: confirmation that issue is resolved, rollback plan discarded

1. check pod status: kubectl get pods -n <namespace> -o wide | grep <pod-name>
2. verify pod is running and not restarting: restarts count should not increase
3. check recent events: kubectl get events -n <namespace> --sort-by='.lastTimestamp' | tail -20 should show no new warnings or errors
4. if deployment, verify rollout: kubectl rollout status deployment/<name> -n <namespace>
5. verify service connectivity (if network issue): kubectl exec <test-pod> -n <namespace> -- curl <service-name>.<namespace>.svc.cluster.local
6. if all checks pass, session is complete. if not, loop back to step 3 (new diagnostics) or step 4 (different workflow)

decision points

preflight access denied (step 1)

if kubectl auth can-i get pods -A returns "no", stop. the user cannot run this skill without rbac permissions. action: request cluster admin to grant read access to pods, events, services, nodes, pvc, pv across all namespaces, or narrow to specific namespace.

scope: single pod vs. cluster-wide (step 2)

if user reports symptoms tied to one pod (e.g., "pod is crashing"), follow step 3a (pod diagnostics). if symptoms are cluster-wide or unclear (e.g., "cluster is slow"), follow step 3b (cluster health). if unsure, run step 3b first; it identifies problem pods and narrows scope for step 3a.

network issue suspected (step 2)

if symptom involves service connectivity, dns resolution, or "pod cannot reach service", include step 3c (network diagnostics) in addition to pod or cluster diagnostics.

metrics-server unavailable (step 3a/3b)

if kubectl top pods fails (metrics-server not running), scripts print warning and continue using kubectl describe and events. resource usage data will be less precise but diagnosis continues. action: note in report that cpu/memory metrics are unavailable; if needed, install metrics-server or use external monitoring (prometheus, etc.).

pod logs empty or previous logs unavailable (step 3a)

if pod logs are empty or no previous logs exist, pod may have never started or started cleanly then crashed before logging. action: check pod events for imagepull failures or pending reasons; inspect describe output for init container status; if still unclear, exec into pod and check application startup behavior.

disruptive fix required (step 5)

before running delete, drain, rollout restart, rollout undo, or debug --copy-to commands, user must confirm understanding of blast radius (which pods/services affected, how long outage, rollback window). action: always snapshot state first (step 5.2); confirm user acknowledges risk; proceed only after confirmation. if uncertain, escalate to sre or cluster owner.

fix fails or issue recurs (step 6)

if verification fails (pod still crashing, service still unreachable), do not retry same fix. action: loop back to step 3 with fresh diagnostics; root cause may differ from assumed; re-run pod_diagnostics.py or network_debug.sh to gather new data; consult different section of troubleshooting_workflow.md.

cluster api server unavailable (step 1)

if kubectl config current-context or any kubectl command times out or hangs, cluster api is likely down. action: verify cluster network connectivity, check control plane node status (if accessible), contact cluster owner. skill cannot proceed without api access.

pod not found or already deleted (step 3a)

if pod_diagnostics.py exits with code 2 and reports "pod not found", pod may have been deleted or is in a different namespace. action: confirm correct namespace, list pods in namespace, check recent events for pod deletion, or ask user to clarify which pod to debug.

network policy blocks diagnostics (step 3c)

if network_debug.sh reports all connectivity tests fail and networkpolicies exist, policy may be too strict (blocking pod-to-pod, pod-to-dns, or pod-to-api-server traffic). action: review networkpolicies in namespace, verify they allow egress to dns (port 53) and kubernetes api (port 443); if policy is correct, issue is elsewhere; if policy is wrong, update it (requires network admin).

output contract

cluster_health.sh output

• format: plain text sections (cluster info, node status, pod summary, events, component status, error count)
• file location: stdout or redirect to cluster-health-<timestamp>.txt
• exit code: 0 (success), 1 (checks failed or --strict + warnings), 2 (blocked preconditions)
• content guaranteed: node status, pod phase summary, recent events, error state counts

network_debug.sh output

• format: plain text sections (pod network, dns resolution results, service endpoints, networkpolicy, connectivity test results, coredns logs if dns failed)
• file location: stdout
• exit code: 0 (all probes pass), 1 (probe failures or --strict + warnings), 2 (access denied, pod unreachable)
• content guaranteed: dns resolution test, service endpoint check, networkpolicy listing

pod_diagnostics.py output

• format: plain text sections (pod status, events, logs, resource usage, node info, yaml)
• file location: stdout or -o <filename> (saves to file, still prints to stdout unless redirected)
• exit code: 0 (all data collected), 1 (warnings, e.g., metrics unavailable), 2 (access denied, pod not found)
• content guaranteed: pod status and description, events, logs (current), yaml configuration

snapshots (before fix)

• format: plain text or yaml
• file location: user specified, conventional names: before-snapshot.txt, before-<pod-name>.yaml, before-events.txt
• content: full output of kubectl get, kubectl describe, or kubectl get events at time of snapshot

outcome signal

the skill has succeeded when all are true:

1. cluster context confirmed: kubectl config current-context returns expected context and matches namespace in diagnostic output
2. diagnostics captured: at least one script (pod_diagnostics.py, cluster_health.sh, or network_debug.sh) ran successfully and produced readable output; exit code 0 or 1 (not 2)
3. root cause identified: diagnostics and reference workflow narrowed issue to specific layer (pod, network, node, storage, configuration) with supporting evidence from logs, events, describe output, or networkpolicies
4. before snapshot exists (if fix applied): before-snapshot.txt, before-pod-yaml.yaml, and before-events.txt exist and match actual state at time of snapshot
5. fix applied and verified:
   • pod now running and not restarting (restarts count stable or reset)
   • service endpoints now registered and reachable (if service issue)
   • pvc now bound to pv (if storage issue)
   • deployment rollout complete (if rollout issue)
   • no new error events in kubectl get events output
6. session documented: incident notes include symptom, root cause, fix applied, and reference path used (troubleshooting_workflow.md section name)

if any of these are false, session is incomplete. loop back to step 3 or 4 and gather more data, or escalate to sre/cluster owner if diagnostics cannot pinpoint root cause.