Skip to content

End-to-End Deployment Runbook

This runbook ties together Terraform, Docker, Kubernetes, Helm, CI/CD, verification, rollback, and troubleshooting. Use the environment-specific pages for exact values; this page provides the execution order and safety gates.

AkshayaBazaar staging/production application deployment is currently blocked

This is a generic, environment-agnostic procedure. For the AkshayaBazaar application specifically, steps 3 onward (Terraform apply, container deploy, Kubernetes/Helm deploy) are not currently executable against real staging or production - AkshayaBazaar's own public domain has not been registered yet, so there is no real, reachable hostname to deploy ingress/CORS/TLS/media configuration against. See Deployment → Staging "Deployment is currently blocked" and Deployment → Production for the full explanation and what must happen first. This blocker does not apply to local development (step 4's Docker/local examples) or to publishing this documentation portal itself (step 12, Cloudflare Pages - a separately hosted, already-live Cloudflare Pages site, unrelated to the marketplace application's future domain).

1. Identify the change

Record:

Repository: <repository>
Commit/tag: <sha-or-tag>
Environment: <local|dev|staging|production>
Namespace: <namespace-or-N/A>
Change type: <application|infrastructure|configuration|documentation>
Rollback target: <previous-known-good-version>

Do not continue until the target environment is unambiguous.

2. Validate source and CI

For AkshayaBazaar, required CI is defined in .github/workflows/ci.yml:

backend: restore → build → tests
frontend: npm ci → lint → build
docs: install requirements → mkdocs build --strict

Production/staging deployment must use a reviewed commit that passed the required checks.

3. Infrastructure changes — Terraform first

Skip this section when the release contains no infrastructure change.

For local/dev disposable environments only:

cd terraform/environments/dev
terraform init
terraform fmt -check -recursive
terraform validate
terraform plan -out=tfplan
terraform apply tfplan

For staging and production environments, raw un-gated terraform apply is strictly prohibited. All staging and production infrastructure changes must execute through the approved gated entrypoints:

  1. Jenkins Gated Pipeline (ACTION=Infrastructure or ACTION=Deploy):
  2. Select ACTION=Infrastructure and INFRA_ENV=<staging|production>.
  3. The pipeline runs scripts/terraform-ci.sh plan <environment>, archives the plan for review and approval, structurally executes the authoritative pre-deployment privacy gate (scripts/pre-deploy-gate.sh), and applies only upon exit code 0.
  4. Canonical Gated Automation (scripts/terraform-ci.sh): 
    scripts/terraform-ci.sh plan <environment>
# Review the generated plan: artifacts/terraform/<environment>-plan.txt
# When approved, apply through the authoritative privacy gate:
scripts/terraform-ci.sh apply <environment>
    scripts/terraform-ci.sh apply <environment> automatically invokes scripts/pre-deploy-gate.sh akshayabazaar-<environment> immediately before Terraform mutation and halts if the gate fails or times out.

CRITICAL MUTATION GATE INVARIANT: Direct, raw terraform apply commands against real staging or production environments are strictly prohibited. Staging and production infrastructure mutations must always pass the authoritative pre-deployment privacy gate (scripts/pre-deploy-gate.sh).

For production, follow the production approval process. Never reuse a staging plan file in production.

4. Container validation

Before Kubernetes deployment, verify the application can be built as containers.

Local/read-only validation examples:

docker compose config
docker compose build
docker compose ps

When images are published by CI, record immutable image tags/digests used by the deployment. Avoid relying on an ambiguous latest tag for rollback-critical releases.

5. Kubernetes pre-flight

kubectl config current-context
kubectl get ns
kubectl get pods -n <namespace>
kubectl get secret akshayabazaar-<env>-secrets -n <namespace>

Confirm the namespace matches the environment inventory.

For production, capture the current state before upgrade:

helm list -n <namespace>
helm history <release> -n <namespace>
kubectl get deploy,pod,svc,ingress -n <namespace>

6. Helm render and diff/review

Use the environment-specific values file.

helm lint helm/akshayabazaar
helm template <release> helm/akshayabazaar \
  --namespace <namespace> \
  -f helm/akshayabazaar/values-<environment>.yaml > rendered.yaml

Review images, ingress hosts, service ports, environment references, Secret names, replica counts, probes, resources, and namespace before upgrade.

Do not put secret values in Helm values committed to Git.

7. Deploy

All deployments to staging or production MUST execute via the repository's canonical gated automation:

  • Jenkins CI/CD: ACTION=Deploy
  • Staging (Linux/Bash): scripts/deploy-staging.sh <image-tag>
  • Staging (PowerShell): .\scripts\deploy-staging.ps1 -ImageTag <image-tag>
  • Production (Linux/Bash): scripts/deploy-production.sh <image-tag>

CRITICAL SECURITY INVARIANT: Direct, raw helm upgrade --install or manual cluster mutations against staging or production are strictly prohibited. All mutations must first satisfy the authoritative private-storage privacy gate (scripts/pre-deploy-gate.sh / scripts/pre-deploy-gate.ps1).

8. Immediate verification

kubectl rollout status deployment/<frontend-deployment> -n <namespace>
kubectl rollout status deployment/<backend-deployment> -n <namespace>
kubectl get pods -n <namespace>
kubectl get events -n <namespace> --sort-by=.lastTimestamp

Then perform application verification using Operations → Verification:

  • frontend responds,
  • API health responds,
  • authentication path works,
  • database/Redis connectivity is healthy,
  • critical marketplace smoke tests pass,
  • ingress/TLS/DNS point to the intended environment,
  • no unexpected error spike appears in logs.

9. Release acceptance

A release is complete only when:

CI                 PASS
Deployment         PASS
Pods/rollout       HEALTHY
Smoke tests        PASS
Monitoring/logs    STABLE
Version/commit     CONFIRMED

Record the deployed commit/image and verification result.

10. Rollback decision

Rollback immediately when there is a material regression such as:

  • failed startup/readiness,
  • authentication outage,
  • checkout/payment regression,
  • database migration incompatibility,
  • severe error-rate increase,
  • wrong environment configuration,
  • data-integrity risk.

Follow Operations → Rollback. Do not improvise database rollback: schema/data rollback needs the specific migration/backup plan for that release.

Rollback entrypoints (all gated):

  • Jenkins CI/CD: ACTION=Rollback, ROLLBACK_ENV=<staging|production>, ROLLBACK_REVISION=<rev>
  • Staging (PowerShell): .\scripts\rollback-staging.ps1 -Revision <rev> -Yes

CRITICAL SECURITY INVARIANT: Never run bare helm rollback or manual kubectl mutations against staging or production. There is no "emergency manual rollback" bypass: every rollback operation must execute through the canonical pre-deployment privacy gate.

Verify again after rollback.

11. Troubleshooting order

When deployment fails, investigate from outside in:

DNS / Cloudflare
      ↓
Ingress / TLS
      ↓
Kubernetes Service
      ↓
Pods / readiness / logs
      ↓
Application configuration
      ↓
Database / Redis / external dependencies

Use read-only commands first. See Operations → Troubleshooting, Kubernetes Troubleshooting, and Docker Troubleshooting.

12. Documentation-only deployment

For MkDocs portal changes, do not run Terraform/Helm. The path is:

branch → PR → GitHub docs CI → merge master → Cloudflare Pages → portal verification

See Cloudflare Pages.