Skip to content

Database Backup

Purpose

Nightly encrypted MySQL backup to Cloudflare R2, per environment - and how to run one manually before a risky migration.

Status: disabled by default

# helm/akshayabazaar/values.yaml
backup:
  enabled: false
Enable per-environment in that environment's values-<env>.yaml.

Architecture/context

mysqldump (read-only against the external DB)
   |
   v
gzip
   |
   v
AES-256 encrypt
   |
   v
upload to R2 (WRITE-ONLY - this script never deletes anything, anywhere)

Retention (daily/weekly/monthly expiry) is enforced entirely by R2's own lifecycle rules (scripts/r2-lifecycle-policy.json), not by the backup script - a bug in the script can waste storage but can never delete a backup out from under a rotation still in progress.

Files involved

helm/akshayabazaar/files/mysql-backup.sh, helm/akshayabazaar/templates/backup/mysql-backup-cronjob.yaml, helm/akshayabazaar/templates/backup/mysql-backup-scripts-configmap.yaml, helm/akshayabazaar/examples/mysql-backup-secret.example.yaml, scripts/mysql-backup.sh, scripts/r2-lifecycle-policy.json, scripts/apply-r2-lifecycle.sh.

Configuration (Secret, never a literal value in any values file)

DB_HOST: "<DB_HOST>"
DB_PORT: "3306"
DB_NAME: "ecommerce_db"
DB_USER: "<READ_ONLY_DB_USER>"
DB_PASSWORD: "<MYSQL_PASSWORD>"
BACKUP_ENCRYPTION_PASSPHRASE: "<AES_256_PASSPHRASE>"   # NOT the same secret as any DB/API credential
R2_ACCOUNT_ID: "<CLOUDFLARE_ACCOUNT_ID>"
R2_ACCESS_KEY_ID: "<R2_ACCESS_KEY_ID>"
R2_SECRET_ACCESS_KEY: "<R2_SECRET_ACCESS_KEY>"
R2_BACKUP_BUCKET: "akshayabazaar-backups-<env>"

A separate R2 bucket per environment, and a separate bucket entirely from the media bucket - blast-radius isolation, so a compromise of backup credentials never also exposes the live application's own DB/JWT/payment secrets, and vice versa. ENVIRONMENT_LABEL (optional) prefixes every uploaded key with <label>/ as a second, defense-in-depth layer even if a bucket were ever pointed at by two environments by mistake.

Commands

# Confirm the CronJob is scheduled (read-only)
kubectl get cronjob -n staging

# Trigger a manual run right now, without waiting for the schedule (mutating - creates a Job)
kubectl create job -n staging --from=cronjob/akshaya-local-mysql-backup manual-backup-$(Get-Date -Format yyyyMMdd-HHmmss)

# Watch it
kubectl get jobs -n staging -w
kubectl logs -n staging -l job-name=manual-backup-... --tail=100

# Linux/Bash - a one-off manual backup run directly (e.g. right before a risky migration),
# outside Kubernetes entirely, using the same script
DB_HOST=<DB_HOST> DB_USER=<DB_USER> DB_PASSWORD=<MYSQL_PASSWORD> \
BACKUP_ENCRYPTION_PASSPHRASE=<PASSPHRASE> \
R2_ACCOUNT_ID=<ACCOUNT_ID> R2_ACCESS_KEY_ID=<KEY> R2_SECRET_ACCESS_KEY=<SECRET> R2_BACKUP_BUCKET=<BUCKET> \
  ./helm/akshayabazaar/files/mysql-backup.sh

Validation / expected result

Backup pipeline validated end to end (mysqldump → gzip → openssl encrypt → decrypt → restore → row/table counts matched) against a disposable database - see repository BACKUP_RESTORE.md "Restore validation" for the full transcript.

Troubleshooting

  • CronJob never runs - confirm backup.enabled: true for this environment's values file, and that the Secret referenced by backup.existingSecretName actually exists in the namespace.
  • Job fails at the mysqldump step - almost always credentials (a read-only DB user is strongly recommended, never the application's own DB_USER) or network reachability to the data VM.
  • Job fails at the upload step - R2 access key/secret or bucket name mismatch; confirm against the Secret, never by guessing a new value.

Rollback / recovery (restore)

Restoring from a backup is a deliberate, manual, human-run operation - never automated by this script (write-only, by design). See repository BACKUP_RESTORE.md for the full restore procedure. Never restore into staging/production without confirming this is the intended recovery action first - see Operations → Rollback for when a Helm/application rollback is sufficient instead of a database restore.

Security considerations

  • Read-only DB credentials recommended for the backup user - it only ever needs mysqldump --single-transaction (SELECT, SHOW VIEW, TRIGGER, LOCK TABLES, EVENT), never write access.
  • Backups are AES-256 encrypted before upload - losing the encryption passphrase makes every existing backup permanently unrecoverable (by design - store it in a password manager / secret manager with restricted access, in addition to wherever the Kubernetes Secret lives).
  • Separate R2 bucket, separate Secret, from the application's own media/DB credentials.