Skip to content

Rollback

Two layers can need a rollback independently: the Helm release (application + config) and Terraform (infrastructure). This project never rolls back Terraform automatically - infrastructure changes are rare, deliberate, and reviewed via terraform plan before any apply, so "rollback" there means "apply a corrected configuration," not an automated revert.

Helm release rollback

Purpose: revert a Helm release (chart version, values, image tags) to a previous revision.

Prerequisites: helm on PATH, pointed at the right cluster.

helm history akshaya-local -n staging          # read-only - see what's available first
.\scripts\rollback-staging.ps1                  # dry run - prints `helm history` only
.\scripts\rollback-staging.ps1 -Yes             # MUTATING - rolls back one revision
.\scripts\rollback-staging.ps1 -Revision 4 -Yes # MUTATING - rolls back to revision 4 specifically
Local cluster: add -Context k3d-akshaya-staging -Namespace staging -ReleaseName akshaya-local. Production: add -Namespace akshayabazaar-production -ReleaseName akshayabazaar - only run this against production with explicit authorization, per Deployment → Production.

Expected result: the release's Deployments/ConfigMaps/etc. revert to the target revision's manifests. The pre-upgrade migration hook does not re-run on a rollback (Helm only re-runs hooks on install/upgrade, not rollback).

Verify: .\scripts\verify-staging.ps1 (or with the relevant overrides).

Common failure scenarios: target revision no longer in history (Helm keeps the last 10 by default); rolling back to a revision whose image tag no longer exists in the registry (ImagePullBackOff - check kubectl get pods after rollback).

Rollback of a rollback: roll forward again using the gated rollback script .\scripts\rollback-staging.ps1 -Revision <REVISION> -Yes or Jenkins ACTION=Rollback. Never run bare helm rollback commands directly.

CRITICAL SECURITY INVARIANT: All rollback operations against staging or production enforce the canonical pre-deployment privacy gate (scripts/pre-deploy-gate.ps1 / scripts/pre-deploy-gate.sh). Direct, raw helm rollback or manual cluster mutations are strictly prohibited without passing the privacy gate first. There is no emergency manual bypass.

What Helm rollback does not do

  • Does not undo a database migration. This chart's migrations are additive-only by convention specifically so an older application version continues to work against a newer schema - verify this holds for the specific migration involved before rolling back across one.
  • Does not revert Terraform-provisioned infrastructure (VM size, DNS records, R2 buckets) - see "Terraform rollback" below.
  • Does not un-rotate a Secret - Secrets are managed outside this chart; rolling back the Helm release does not roll back a Secret's contents.

Terraform rollback

Not automated - "rolling back" infrastructure means:

  1. git revert (or manually re-edit) the .tf files to the previous state.
  2. terraform plan against the reverted configuration - review carefully; some changes (a VM's server_type or private_ip) can be destructive (replace, not update) even when "reverting."
  3. Only after reviewing the plan, apply the reverted state strictly through the approved gated entrypoint:
  4. Staging / Production: Execute via the canonical gated automation scripts/terraform-ci.sh apply <environment> (which structurally invokes the authoritative pre-deployment privacy gate scripts/pre-deploy-gate.sh before mutation) or via the gated Jenkins infrastructure action (ACTION=Infrastructure INFRA_ENV=<environment>). Raw un-gated terraform apply is strictly prohibited on real staging or production environments.
  5. Local / Disposable Dev: Direct terraform apply is permitted only when clearly scoped to disposable local dev environments (environments/dev). See Terraform → Execution.

If the issue is transient (a stuck pod, a stale connection pool) rather than a bad release/revision:

.\scripts\restart-staging.ps1     # MUTATING - rolling restart, no downtime (maxUnavailable: 0)
or a full stop/redeploy: 
.\scripts\stop-staging.ps1 -Yes   # MUTATING - scales to 0, does not uninstall or touch data
# then re-run the matching deploy script to bring it back

Database restore (last resort, not a "rollback")

See Operations → Backup "Rollback / recovery (restore)" - a deliberate, manual, human-run operation, never automated, never attempted casually.