Kubernetes Debugging Skill

Overview

Systematic toolkit for debugging Kubernetes clusters, workloads, networking, and storage with a deterministic, safety-first workflow.

Trigger Phrases

Use this skill when requests resemble: - "My pod is in CrashLoopBackOff; help me find the root cause." - "Service DNS works in one pod but not another." - "Deployment rollout is stuck." - "Pods are Pending and not scheduling." - "Cluster health looks degraded after a change." - "PVC is pending and pods cannot mount storage."

Prerequisites

Run from the skill directory (devops-skills-plugin/skills/k8s-debug) so relative script paths work as written.

Required

• kubectl installed and configured.
• An active cluster context.
• Read access to namespaces, pods, events, services, and nodes.

Quick preflight:

kubectl config current-context
kubectl auth can-i get pods -A
kubectl auth can-i get events -A
kubectl get ns

Optional but Recommended

• jq for more precise filtering in ./scripts/cluster_health.sh.
• Metrics API (metrics-server) for kubectl top.
• In-container debug tools (nslookup, getent, curl, wget, ip) for deep network tests.

Fallback behavior: - If optional tools are missing, scripts continue and print warnings with reduced output. - If kubectl top is unavailable, continue with kubectl describe and events.

When to Use This Skill

Use this skill for: - Pod failures (CrashLoopBackOff, ImagePullBackOff, Pending, OOMKilled) - Service connectivity or DNS resolution issues - Network policy or ingress problems - Volume and storage mount failures - Deployment rollout issues - Cluster health or performance degradation - Resource exhaustion (CPU/memory) - Configuration problems (ConfigMaps, Secrets, RBAC)

Safety Rules for Disruptive Commands

Default mode is read-only diagnosis first. Only execute disruptive commands after confirming blast radius and rollback.

Commands requiring explicit confirmation: - kubectl delete pod ... --force --grace-period=0 - kubectl drain ... - kubectl rollout restart ... - kubectl rollout undo ... - kubectl debug ... --copy-to=...

Before disruptive actions:

# Snapshot current state for rollback and incident notes
kubectl get deploy,rs,pod,svc -n <namespace> -o wide
kubectl get pod <pod-name> -n <namespace> -o yaml > before-<pod-name>.yaml
kubectl get events -n <namespace> --sort-by='.lastTimestamp' > before-events.txt

Reference Navigation Map

Load only the section needed for the observed symptom.

Symptom / Need	Open	Start section
You need an end-to-end diagnosis path	./references/troubleshooting_workflow.md	General Debugging Workflow
Pod state is Pending, CrashLoopBackOff, or ImagePullBackOff	./references/troubleshooting_workflow.md	Pod Lifecycle Troubleshooting
Service reachability or DNS failure	./references/troubleshooting_workflow.md	Network Troubleshooting Workflow
Node pressure or performance regression	./references/troubleshooting_workflow.md	Resource and Performance Workflow
PVC / PV / storage class issues	./references/troubleshooting_workflow.md	Storage Troubleshooting Workflow
Quick symptom-to-fix lookup	./references/common_issues.md	matching issue heading
Post-mortem fix options for known issues	./references/common_issues.md	Solutions sections

Scripts Overview

Script	Purpose	Required args	Optional args	Output	Fallback behavior
./scripts/cluster_health.sh	Cluster-wide health snapshot (nodes, workloads, events, common failure states)	None	--strict, K8S_REQUEST_TIMEOUT env var	Sectioned report to stdout	Continues on check failures, tracks them in summary and exit code
./scripts/network_debug.sh	Pod-centric network and DNS diagnostics	<pod-name> (<namespace> defaults to default)	--strict, --insecure, K8S_REQUEST_TIMEOUT env var	Sectioned report to stdout	Uses secure API probe by default; insecure TLS requires explicit --insecure
./scripts/pod_diagnostics.py	Deep pod diagnostics (status, describe, YAML, events, per-container logs, node context)	<pod-name>	-n/--namespace, -o/--output	Sectioned report to stdout or file	Fails fast on missing access; skips optional metrics/log blocks with clear messages

Script Exit Codes

./scripts/cluster_health.sh and ./scripts/network_debug.sh share the same contract:

• 0: checks completed with no check failures (warnings allowed unless --strict is set).
• 1: one or more checks failed, or warnings occurred in --strict mode.
• 2: blocked preconditions (for example: missing kubectl, no active context, inaccessible namespace/pod).

Deterministic Debugging Workflow

Follow this systematic approach for any Kubernetes issue:

1. Preflight and Scope

kubectl config current-context
kubectl get ns
kubectl auth can-i get pods -n <namespace>

If preflight fails, stop and fix access/context first.

2. Identify the Problem Layer

Categorize the issue: - Application Layer: Application crashes, errors, bugs - Pod Layer: Pod not starting, restarting, or pending - Service Layer: Network connectivity, DNS issues - Node Layer: Node not ready, resource exhaustion - Cluster Layer: Control plane issues, API problems - Storage Layer: Volume mount failures, PVC issues - Configuration Layer: ConfigMap, Secret, RBAC issues

3. Gather Diagnostics with the Right Script

Use the appropriate diagnostic script based on scope:

Pod-Level Diagnostics

Use ./scripts/pod_diagnostics.py for comprehensive pod analysis:

python3 ./scripts/pod_diagnostics.py <pod-name> -n <namespace>

This script gathers: - Pod status and description - Pod events - Container logs (current and previous) - Resource usage - Node information - YAML configuration

Output can be saved for analysis:

python3 ./scripts/pod_diagnostics.py <pod-name> -n <namespace> -o diagnostics.txt

Cluster-Level Health Check

Use ./scripts/cluster_health.sh for overall cluster diagnostics:

./scripts/cluster_health.sh > cluster-health-$(date +%Y%m%d-%H%M%S).txt

This script checks: - Cluster info and version - Node status and resources - Pods across all namespaces - Failed/pending pods - Recent events - Deployments, services, statefulsets, daemonsets - PVCs and PVs - Component health - Common error states (CrashLoopBackOff, ImagePullBackOff)

Network Diagnostics

Use ./scripts/network_debug.sh for connectivity issues:

./scripts/network_debug.sh <namespace> <pod-name>
# or force warning sensitivity / insecure TLS only when explicitly needed:
./scripts/network_debug.sh --strict <namespace> <pod-name>
./scripts/network_debug.sh --insecure <namespace> <pod-name>

This script analyzes: - Pod network configuration - DNS setup and resolution - Service endpoints - Network policies - Connectivity tests - CoreDNS logs

4. Follow Issue-Specific Reference Workflow

Based on the identified issue, consult ./references/troubleshooting_workflow.md:

• Pod Pending: Resource/scheduling workflow
• CrashLoopBackOff: Application crash workflow
• ImagePullBackOff: Image pull workflow
• Service issues: Network connectivity workflow
• DNS failures: DNS troubleshooting workflow
• Resource exhaustion: Performance investigation workflow
• Storage issues: PVC binding workflow
• Deployment stuck: Rollout workflow

5. Apply Targeted Fixes

Refer to ./references/common_issues.md for symptom-specific fixes.

6. Verify and Close

Run final verification:

kubectl get pods -n <namespace> -o wide
kubectl get events -n <namespace> --sort-by='.lastTimestamp' | tail -20
kubectl rollout status deployment/<name> -n <namespace>

Issue is done when user-visible behavior is healthy and no new critical warning events appear.

Example Flows

Example 1: CrashLoopBackOff in payments Namespace

python3 ./scripts/pod_diagnostics.py payments-api-7c97f95dfb-q9l7k -n payments -o payments-diagnostics.txt
kubectl logs payments-api-7c97f95dfb-q9l7k -n payments --previous --tail=100
kubectl get deploy payments-api -n payments -o yaml | grep -A 8 livenessProbe

Then open ./references/common_issues.md and apply the CrashLoopBackOff solutions.

Example 2: Service DNS/Connectivity Failure

./scripts/network_debug.sh checkout checkout-api-75f49c9d8f-z6qtm
kubectl get svc checkout-api -n checkout
kubectl get endpoints checkout-api -n checkout
kubectl get networkpolicies -n checkout

Then follow Service Connectivity Workflow in ./references/troubleshooting_workflow.md.

Essential Manual Commands

Pod Debugging

# View pod status
kubectl get pods -n <namespace> -o wide

# Detailed pod information
kubectl describe pod <pod-name> -n <namespace>

# View logs
kubectl logs <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --previous  # Previous container
kubectl logs <pod-name> -n <namespace> -c <container>  # Specific container

# Execute commands in pod
kubectl exec <pod-name> -n <namespace> -it -- /bin/sh

# Get pod YAML
kubectl get pod <pod-name> -n <namespace> -o yaml

Service and Network Debugging

# Check services
kubectl get svc -n <namespace>
kubectl describe svc <service-name> -n <namespace>

# Check endpoints
kubectl get endpoints -n <namespace>

# Test DNS
kubectl exec <pod-name> -n <namespace> -- nslookup kubernetes.default

# View events
kubectl get events -n <namespace> --sort-by='.lastTimestamp'

Resource Monitoring

# Node resources
kubectl top nodes
kubectl describe nodes

# Pod resources
kubectl top pods -n <namespace>
kubectl top pod <pod-name> -n <namespace> --containers

Emergency Operations

# Restart deployment
kubectl rollout restart deployment/<name> -n <namespace>

# Rollback deployment
kubectl rollout undo deployment/<name> -n <namespace>

# Force delete stuck pod
kubectl delete pod <pod-name> -n <namespace> --force --grace-period=0

# Drain node (maintenance)
kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

# Cordon node (prevent scheduling)
kubectl cordon <node-name>

Completion Criteria

Troubleshooting session is complete when all are true: - [ ] Cluster context and namespace are confirmed. - [ ] Relevant diagnostic script output is captured. - [ ] Root cause is identified and tied to evidence (events/logs/config/state). - [ ] Any disruptive action was preceded by snapshot and rollback plan. - [ ] Fix verification commands show healthy state. - [ ] Reference path used (./references/troubleshooting_workflow.md or ./references/common_issues.md) is documented in notes.

Related Tools

Useful additional tools for Kubernetes debugging: - kubectl-debug: Advanced debugging plugin - stern: Multi-pod log tailing - kubectx/kubens: Context and namespace switching - k9s: Terminal UI for Kubernetes - lens: Desktop IDE for Kubernetes - Prometheus/Grafana: Monitoring and alerting - Jaeger/Zipkin: Distributed tracing