Verification¶

Purpose¶

Prove a deployment is actually working end-to-end - not just that helm upgrade returned success. A Deployment can be "successfully applied" and still have 0 Service endpoints, a NetworkPolicy silently blocking traffic, or a healthy-looking pod that can't reach its database.

Prerequisites¶

kubectl on PATH, pointed at the target cluster (-Context, or the current context / $env:KUBECONFIG). docker optional - only WARNs, never FAILs, if absent.

The check pyramid¶

Docker
   |
Kubernetes Cluster
   |
Namespace
   |
Pods
   |
Deployments
   |
Services (+ Endpoints)
   |
Ingress
   |
Frontend (external HTTP)
   |
Backend/API Health (in-cluster)
   |
Database connectivity
   |
Redis connectivity

Commands¶

# Real staging/production - defaults already point at the real namespace/release/host
.\scripts\verify-staging.ps1

# Local k3d
.\scripts\verify-staging.ps1 `
  -Context k3d-akshaya-staging `
  -Namespace staging `
  -ReleaseName akshaya-local `
  -IngressHost staging.akshaya.local `
  -IngressPort 80

# Production explicitly
.\scripts\verify-staging.ps1 -Namespace akshayabazaar-production -ReleaseName akshayabazaar -IngressHost akshayathreadworld.in

This is a read-only command - it makes no cluster changes.

Expected result¶

Summary: 14 PASS, 0 WARNING, 0 FAIL

Every check PASS → exit code 0. A WARN (e.g. DNS for a local hostname not resolvable from the machine running the script) doesn't fail the run; any FAIL sets exit code 1.

Live run (local k3d cluster, port 80)¶

AkshayaBazaar staging verification
Namespace: staging | Release: akshaya-local | Context: k3d-akshaya-staging | Host: staging.akshaya.local:80
-----------------------------------------------------------------------
[PASS] Docker                           daemon reachable
[PASS] Kubernetes cluster               reachable
[PASS] Namespace                        'staging' exists
[PASS] Pod: akshaya-local-backend-...   Running, all containers ready
[PASS] Pod: akshaya-local-frontend-...  Running, all containers ready
[PASS] Deployment: akshaya-local-backend 1/1 ready
[PASS] Deployment: akshaya-local-frontend 1/1 ready
[PASS] Service: akshaya-local-backend   1 endpoint(s)
[PASS] Service: akshaya-local-frontend  1 endpoint(s)
[PASS] Ingress: akshaya-local           host(s): staging.akshaya.local
[PASS] Frontend (external)              http://staging.akshaya.local:80 -> 200
[PASS] Backend/API health (live)        in-cluster akshaya-local-backend:8080/health/live -> Healthy
[PASS] Database connectivity            backend /health/ready reports Healthy (gates on AppDbContext.CanConnectAsync)
[PASS] Redis connectivity               backend /health/ready reports Healthy (gates on Redis, if configured)
-----------------------------------------------------------------------
Summary: 14 PASS, 0 WARNING, 0 FAIL

How each layer is actually checked¶

Layer	Method
Docker	`docker version` - `WARN`, not `FAIL`, if unavailable (verifying an already-deployed remote cluster doesn't need a local Docker daemon)
Kubernetes cluster	`kubectl cluster-info`
Namespace	`kubectl get namespace <ns>`
Pods	`kubectl get pods -l app.kubernetes.io/instance=<release> -o json` - `phase=Running` and every container `ready`
Deployments	`kubectl get deployment ... -o json` - `status.readyReplicas == spec.replicas`
Services + Endpoints	`kubectl get service`/`kubectl get endpoints` per Service - `FAIL`s on 0 endpoints
Ingress	`kubectl get ingress -l app.kubernetes.io/instance=<release>`
Frontend (external)	`Invoke-WebRequest http://<IngressHost>:<IngressPort>` - the only check that leaves the cluster
Backend/API health	`kubectl exec` into the real frontend pod, `wget` the backend Service's `/health/live` - deliberately not an ad-hoc unlabeled pod, which the backend's NetworkPolicy would deny regardless of app health (see Kubernetes → Network Policy)
Database connectivity	Same in-cluster request, to `/health/ready` - gates on `AppDbContext.CanConnectAsync()`. No DB credentials ever touch this script.
Redis connectivity	Same `/health/ready` response - ASP.NET Core's default health-check writer returns one combined `Healthy`/`Unhealthy`, so this script reports DB and Redis together, not independently (documented here rather than claimed otherwise)

Troubleshooting¶

FAIL on Services with "0 endpoints" - selector/pod label mismatch, or no pod Ready yet; cross-reference the Pods/Deployments results just above it.
FAIL on Backend/API health, but Pods/Deployments/Services all PASS - almost always a NetworkPolicy blocking frontend→backend - see Kubernetes → Network Policy.
WARN on Frontend (external) - DNS/hosts-file resolution, or (local k3d) the wrong -IngressPort - k3d commonly remaps the load balancer's port 80 to a different host port; check with docker port k3d-<cluster>-serverlb and see Kubernetes → k3d.

Rollback¶

Not applicable - this script makes no changes.