Skip to content

Deployment Runbook: Production

No production deployment is executed as part of this documentation portal

This page documents the existing, real production deployment process for reference. Building, validating, or updating this documentation never runs any command on this page against real production infrastructure.

Preconditions

  • Staging has already been deployed with the exact same image tag and its smoke tests passed (see Staging) - production only ever receives an image staging already validated, never a separate build.
  • terraform/environments/production/ already applied.
  • akshayabazaar-production-secrets Secret already applied in akshayabazaar-production, with live (not test-mode) provider credentials where required (Razorpay live key, real notification provider credentials - values-production.yaml's own guard refuses to render if a notification channel is left in Simulated mode while aspnetcoreEnvironment: Production, unless explicitly overridden).
  • A named approver from the release-managers group is available - the Jenkins pipeline blocks on this for up to 24h before timing out.

Backup/checkpoint

Mandatory, not optional, before any production release - see Operations → Backup for the automated nightly backup, and take a manual pre-release checkpoint on top of it for anything non-trivial: 

# Linux/Bash, from a host that can reach production's data VM
mysqldump -h <production-data-vm-private-ip> -u <DB_USER> -p ecommerce_db | gzip > production-pre-release-$(date +%Y%m%d-%H%M).sql.gz

Build

Same image staging just validated - never rebuilt. See CI/CD → Jenkins "Promote, don't rebuild".

Deploy

Exclusively through Jenkins' Promote to production stage, gated on manual approval: 

input message: "Promote <tag> to PRODUCTION? Staging smoke tests passed.", submitter: 'release-managers'
// then:
helm upgrade --install akshayabazaar helm/akshayabazaar -f values-production.yaml \
    --set backend.image.tag=<tag> --set frontend.image.tag=<tag> \
    --namespace akshayabazaar-production --wait --timeout 10m --atomic
There is deliberately no PowerShell production deploy script in this repository - production deploys go through Jenkins' approval gate only, never an ad-hoc local command. See Operations → Scripts reference for why.

Database migration

Runs automatically as the same Helm pre-upgrade hook - against production's own data VM. Because this project's migrations are additive-only by convention, an older application version continues to work against a newer schema, which is what makes --atomic's automatic rollback-on-timeout safe even after a migration has already applied - see Operations → Rollback "What Helm rollback does NOT do".

Verification

.\scripts\verify-staging.ps1 -Namespace akshayabazaar-production -ReleaseName akshayabazaar -IngressHost akshayathreadworld.in

Expected result

Summary: 14 PASS, 0 WARNING, 0 FAIL against production's own namespace/host.

Smoke testing

Jenkins runs scripts/smoke-test.sh against production automatically, immediately after deploy - a failure here means production is serving something broken despite --atomic's own rollback attempt (e.g. the rollout reported Ready but the app is still misbehaving). Operator response is the explicit rollback path below, not a second deploy attempt.

Monitoring

See Operations → Monitoring.

Rollback

# Via Jenkins (ACTION=Rollback, ROLLBACK_ENV=production) - the standard path
or manually, only by someone authorized against production: 
helm history akshayabazaar -n akshayabazaar-production
helm rollback akshayabazaar <revision> -n akshayabazaar-production --wait --timeout 5m
Full detail: Operations → Rollback.

Security considerations

  • Production's Secret, DNS, and Cloudflare zone settings are never read, written, or referenced by this documentation portal.
  • Every command on this page that mutates production is a Jenkins pipeline stage behind a named- group manual approval - there is no "quick" unattended path to production in this repository.