Skip to content

Verification

scripts/verify-staging.ps1 is the primary post-deploy validation command - read-only, checks the full requested pyramid, and prints PASS/WARN/FAIL per check.

Docker
  |
Kubernetes Cluster
  |
Namespace
  |
Pods
  |
Deployments
  |
Services (+ Endpoints)
  |
Ingress
  |
Frontend (external HTTP)
  |
Backend/API Health (in-cluster)
  |
Database connectivity
  |
Redis connectivity
  • Purpose: prove a deployment is actually working end-to-end, not just that helm upgrade returned success - a Deployment can be "successfully applied" and still have 0 Service endpoints, a NetworkPolicy silently blocking traffic, or a healthy-looking pod that can't reach its database.
  • Prerequisites: kubectl on PATH and pointed at the target cluster (-Context, or the current context / $env:KUBECONFIG); docker optional (only WARNs, never FAILs, if absent - see "Docker" below).
  • Files involved: none changed - purely reads live cluster state.
  • Command: 
    .\scripts\verify-staging.ps1                     # real staging defaults
.\scripts\verify-staging.ps1 -Context k3d-akshaya-staging -Namespace staging -ReleaseName akshaya-local -IngressHost staging.akshaya.local -IngressPort 8080   # local
  • Expected result: every check PASS, exit code 0. A WARN doesn't fail the run (e.g. DNS for a local hostname not resolvable from the machine running the script); any FAIL sets exit code 1.
  • How to verify this script itself: it was executed live against the local k3d cluster while building it - 14 PASS, 0 WARNING, 0 FAIL, transcript below.
  • Rollback: not applicable - the script makes no changes.

Live run (local k3d cluster)

AkshayaBazaar staging verification
Namespace: staging | Release: akshaya-local | Context: k3d-akshaya-staging | Host: staging.akshaya.local:8080
-----------------------------------------------------------------------
[PASS] Docker                           daemon reachable
[PASS] Kubernetes cluster               reachable
[PASS] Namespace                        'staging' exists
[PASS] Pod: akshaya-local-backend-85b74f9bb4-x8th4 Running, all containers ready
[PASS] Pod: akshaya-local-frontend-f7f6c89c8-827dz Running, all containers ready
[PASS] Deployment: akshaya-local-backend 1/1 ready
[PASS] Deployment: akshaya-local-frontend 1/1 ready
[PASS] Service: akshaya-local-backend   1 endpoint(s)
[PASS] Service: akshaya-local-frontend  1 endpoint(s)
[PASS] Ingress: akshaya-local           host(s): staging.akshaya.local
[PASS] Frontend (external)              http://staging.akshaya.local:8080 -> 200
[PASS] Backend/API health (live)        in-cluster akshaya-local-backend:8080/health/live -> Healthy
[PASS] Database connectivity            backend /health/ready reports Healthy (gates on AppDbContext.CanConnectAsync)
[PASS] Redis connectivity               backend /health/ready reports Healthy (gates on Redis, if configured)
-----------------------------------------------------------------------
Summary: 14 PASS, 0 WARNING, 0 FAIL

How each layer is actually checked

Layer Method
Docker docker version - WARN (not FAIL) if unavailable, since verifying an already-deployed remote cluster doesn't need a local Docker daemon.
Kubernetes cluster kubectl cluster-info
Namespace kubectl get namespace <ns>
Pods kubectl get pods -l app.kubernetes.io/instance=<release> -o json, checked per-pod for phase=Running and every container ready.
Deployments kubectl get deployment -l app.kubernetes.io/instance=<release> -o json, status.readyReplicas == spec.replicas.
Services + Endpoints kubectl get service ... -o json then kubectl get endpoints <name> per Service - FAILs on 0 endpoints (the exact failure mode diagnosed live during this branch's NetworkPolicy debugging).
Ingress kubectl get ingress -l app.kubernetes.io/instance=<release> - reports configured host(s).
Frontend (external) Invoke-WebRequest http://<IngressHost>:<IngressPort> - the only check that leaves the cluster.
Backend/API health kubectl exec into the real frontend pod, wget the backend Service's /health/live - reuses the exact in-cluster-exec technique used to diagnose this chart's NetworkPolicy live (see docs/devops/README.md "Repository audit findings"). Deliberately does not use an ad-hoc unlabeled debug pod, which would be denied by <release>-backend-allow-ingress's NetworkPolicy regardless of whether the app itself is healthy.
Database connectivity Same in-cluster request, to /health/ready - gates on AppDbContext.CanConnectAsync() (see BoundedDbContextHealthCheck.cs). No DB credentials touch this script.
Redis connectivity Same /health/ready response - also gates on Redis when ConnectionStrings__Redis is configured for that environment. /health/ready returns a single combined Healthy/Unhealthy (ASP.NET Core's default health check response writer, no custom JSON body configured in Program.cs), so this script reports DB and Redis together, not independently - documented here rather than claimed otherwise.

Common failure scenarios

  • FAIL on Services with "0 endpoints": selector/pod label mismatch, or no pod is Ready yet. Cross-reference with the Pods/Deployments results just above it.
  • FAIL on Backend/API health, but Pods/Deployments/Services all PASS: almost always a NetworkPolicy blocking the frontend→backend path - see environments/local/README.md "Common failure scenarios" for the exact live-diagnosed example (an ad-hoc debug pod without the right labels gets denied even though the real app is healthy; this is why the script execs into the real frontend pod rather than a throwaway one).
  • WARN on Frontend (external): usually DNS/hosts-file resolution for the hostname from the machine running the script, or (local k3d) the wrong -IngressPort - k3d commonly remaps the load balancer's port 80 to a host port like 8080; check with docker port k3d-<cluster>-serverlb.