Skip to content

Kubernetes Troubleshooting

Broader, cluster-level version of Local → Troubleshooting - applies to any cluster (local k3d, real staging/production), not just local.

Diagnostic sequence (read-only - safe to run against any environment)

kubectl get pods -n <namespace> -o wide
kubectl get deployment -n <namespace>
kubectl get svc -n <namespace>
kubectl get endpoints -n <namespace>
kubectl get ingress -n <namespace>
kubectl describe ingress -n <namespace> <name>
kubectl get networkpolicy -n <namespace>
kubectl logs -n <namespace> <pod> --tail=200
kubectl describe pod -n <namespace> <pod>
helm list -n <namespace>
helm history <release> -n <namespace>

Every command above is read-only - see Operations → Troubleshooting for the full read-only vs. mutating command reference.

Pod stuck Pending

kubectl describe pod -n <namespace> <pod>
Check the Events section at the bottom - almost always insufficient node resources (Insufficient cpu/memory) or an unsatisfiable scheduling constraint.

Pod CrashLoopBackOff

kubectl logs -n <namespace> <pod> --previous --tail=200
--previous is essential - a crash-looping pod's current log is often just the startup of the newest, still-crashing attempt; --previous shows what the last attempt actually failed on.

CreateContainerConfigError

Two known causes in this chart specifically: 1. Missing Secret - envFrom.secretRef references a Secret that doesn't exist yet in this namespace. kubectl get secret -n <namespace> to check; see Helm → Secrets. 2. Non-numeric image user - the image's Dockerfile USER is a name, and securityContext.runAsNonRoot: true has no matching runAsUser. Both this chart's images pin numeric UIDs already (backend 1000, frontend 101) - this only recurs if a Dockerfile change regresses that.

Service has 0 endpoints

kubectl get endpoints -n <namespace> <service-name>
Selector/pod label mismatch, or no pod is Ready yet - cross-reference kubectl get pods -n <namespace> -l <same-selector>.

Migration Job BackoffLimitExceeded

kubectl get pods -n <namespace> -l job-name=<release>-backend-migrate
kubectl logs -n <namespace> <pod> --all-containers --tail=300
See Helm → Migrations and, for the specific NetworkPolicy-caused variant, Network Policy → the Helm hook-ordering deadlock.

Backend reachable in-cluster but Ingress returns nothing

Split the problem in two - test the Service directly first (rules out the app itself), then the Ingress/Traefik path: 

kubectl exec -n <namespace> deploy/<release>-frontend -- wget -qO- http://<release>-backend:8080/health/live
kubectl get ingress -n <namespace>
kubectl get pods -n kube-system -l app.kubernetes.io/name=traefik
See Ingress → Troubleshooting.

helm upgrade hangs or times out

kubectl get events -n <namespace> --sort-by='.lastTimestamp' | Select-Object -Last 30
--atomic deploys (staging/production, and scripts/deploy-staging.ps1) auto-rollback on timeout - check helm history to confirm the rollback completed: 
helm history <release> -n <namespace>