Operations Troubleshooting¶

The full command reference for diagnosing a running environment - every command below is labeled [READ-ONLY] or [MUTATING]. When in doubt, run the read-only command first.

Health verification¶

.\scripts\verify-staging.ps1                                    # [READ-ONLY]
kubectl get pods -n <namespace>                                 # [READ-ONLY]
kubectl get svc -n <namespace>                                  # [READ-ONLY]
kubectl get ingress -n <namespace>                               # [READ-ONLY]
kubectl describe ingress -n <namespace> <name>                   # [READ-ONLY]
helm list -n <namespace>                                         # [READ-ONLY]

Full detail: Operations → Verification.

Logs¶

kubectl logs -n <namespace> <pod> --tail=200                     # [READ-ONLY]
kubectl logs -n <namespace> <pod> --previous --tail=200          # [READ-ONLY] - the last CRASHED attempt, not the current restart
kubectl logs -n <namespace> <pod> --all-containers --tail=300    # [READ-ONLY]
kubectl logs -n <namespace> -l job-name=<release>-backend-migrate --all-containers  # [READ-ONLY] - migration Job

Pod troubleshooting¶

kubectl get pods -n <namespace> -o wide                          # [READ-ONLY]
kubectl describe pod -n <namespace> <pod>                        # [READ-ONLY] - check Events at the bottom
kubectl get events -n <namespace> --sort-by='.lastTimestamp'     # [READ-ONLY]

kubectl delete pod -n <namespace> <pod>                          # [MUTATING] - Deployment recreates it; use restart-staging.ps1 instead where possible for a controlled rolling restart
.\scripts\restart-staging.ps1                                    # [MUTATING] - rolling restart, no downtime

Ingress troubleshooting¶

kubectl get ingress -n <namespace>                                # [READ-ONLY]
kubectl describe ingress -n <namespace> <name>                    # [READ-ONLY]
kubectl get pods -n kube-system -l app.kubernetes.io/name=traefik # [READ-ONLY]
docker port k3d-akshaya-staging-serverlb                          # [READ-ONLY] - local k3d port mapping

Full detail: Kubernetes → Ingress.

DNS / hosts troubleshooting¶

Get-Content "$env:SystemRoot\System32\drivers\etc\hosts" | Select-String "staging.akshaya.local"   # [READ-ONLY]
Resolve-DnsName staging.akshayathreadworld.in                     # [READ-ONLY] - real staging, via Cloudflare
nslookup staging.akshaya.local                                    # [READ-ONLY]

Editing the hosts file (C:\Windows\System32\drivers\etc\hosts) requires an elevated Administrator PowerShell - see Local → First-time setup. This portal never edits it automatically.

Database / Redis checks¶

kubectl exec -n <namespace> deploy/<release>-frontend -- wget -qO- http://<release>-backend:8080/health/ready   # [READ-ONLY] - gates on both DB and Redis

No script or page in this portal ever reads/prints a database or Redis credential directly - every connectivity check goes through the backend's own /health/ready endpoint instead. See Operations → Verification "How each layer is actually checked".

NetworkPolicy troubleshooting¶

kubectl get networkpolicy -n <namespace>                                                          # [READ-ONLY]
kubectl get networkpolicy -n <namespace> <release>-default-deny-ingress -o jsonpath='{.spec.podSelector}'  # [READ-ONLY]

Full detail: Kubernetes → Network Policy.

Rollback / disaster recovery basics¶

helm history <release> -n <namespace>                            # [READ-ONLY]
.\scripts\rollback-staging.ps1                                    # [READ-ONLY] (dry run, no -Yes)
.\scripts\rollback-staging.ps1 -Yes                                # [MUTATING]

Full detail: Operations → Rollback. Database-level disaster recovery: Operations → Backup.

Terraform-level diagnostics (safe subset only)¶

docker run --rm -v "$(pwd)/terraform:/tf" -w /tf hashicorp/terraform:1.9 fmt -recursive -check -diff   # [READ-ONLY]
docker run --rm -v "$(pwd)/terraform:/tf" -w "/tf/environments/staging" hashicorp/terraform:1.9 validate  # [READ-ONLY]

terraform plan/apply/destroy are never run from this portal - see Terraform → Execution.

Escalation - when read-only diagnostics aren't enough¶

If diagnosis points at needing a mutating action against staging, follow the specific runbook for that action (Kubernetes → Troubleshooting, Operations → Rollback). If it points at needing a mutating action against production, stop and follow Deployment → Production - never improvise a production fix from a general troubleshooting command.