Skip to content

Rollback procedures

Consolidated in one place since multiple pages above reference it. Two layers can need a rollback independently: the Helm release (application + config) and Terraform (infrastructure). This repository's automation never rolls back Terraform automatically - infrastructure changes are rare, deliberate, and reviewed via terraform plan before any apply, so "rollback" there means "apply a corrected configuration," not an automated revert.

Helm release rollback

  • Purpose: revert a Helm release (chart version, values, image tags) to a previous revision.
  • Prerequisites: helm on PATH, pointed at the right cluster (-Context/$env:KUBECONFIG).
  • Files involved: none - operates entirely on Helm's own release history stored in the cluster.
  • Command: 
    .\scripts\rollback-staging.ps1                 # dry run - prints `helm history` only
.\scripts\rollback-staging.ps1 -Yes             # rolls back one revision
.\scripts\rollback-staging.ps1 -Revision 4 -Yes # rolls back to revision 4 specifically
    Local cluster: add -Context k3d-akshaya-staging -Namespace staging -ReleaseName akshaya-local.
  • Expected result: the release's Deployments/ConfigMaps/etc. revert to the target revision's manifests; the pre-upgrade migration hook does not re-run on a rollback (Helm only re-runs hooks on install/upgrade, not rollback) - a rollback that needs a database schema reverted as well is not something helm rollback alone handles; see "What Helm rollback does NOT do" below.
  • How to verify: .\scripts\verify-staging.ps1
  • Common failure scenarios: target revision no longer in history (Helm keeps the last 10 by default - helm history shows what's actually available); rolling back to a revision whose image tag no longer exists in the registry (pods will ImagePullBackOff - check kubectl get pods after rollback).
  • Rollback of a rollback: roll forward again with helm rollback <release> <REVISION> pointed at whichever revision you actually want, or re-run the normal deploy script.

What Helm rollback does NOT do

  • Does not undo a database migration. If the revision you're rolling back from included an EF Core migration that's already been applied, helm rollback reverts the application to code that may not match the now-migrated schema. This chart's migrations are additive-only by convention (see backend Migrations/ history) specifically so that an older application version continues to work against a newer schema - verify this holds for the specific migration involved before rolling back across one.
  • Does not revert Terraform-provisioned infrastructure (VM size, DNS records, R2 buckets) - see "Terraform rollback" below.
  • Does not un-rotate a Secret - if a Secret was updated (e.g. rotated JWT key) between the two revisions, rolling back the Helm release does not roll back the Secret's contents (Secrets are managed outside this chart - see helm/akshayabazaar/examples/secret.example.yaml's own header).

Terraform rollback

Not automated by any script in this pass - deliberately. "Rolling back" infrastructure means:

  1. git revert (or manually re-edit) the .tf files to the previous state.
  2. terraform plan against the reverted configuration - review the diff carefully; some changes (e.g. a VM's server_type or private_ip) can be destructive (replace, not update) even when "reverting."
  3. Only after reviewing the plan, a human runs terraform apply - never this repository's automation, and never unattended.

No terraform apply/destroy was run against any environment during this pass - see docs/devops/README.md "Safety guardrails observed". If a Terraform change genuinely needs rolling back, that's an "items requiring approval" situation, not something to script around.

If the issue is transient (a stuck pod, a stale connection pool) rather than a bad release/revision, restart-staging.ps1 (rolling restart, no downtime) or stop-staging.ps1 + re-deploy (full stop/start) may resolve it without needing a rollback at all - see scripts.md.