Skip to content

Deployment Runbook: Staging (VPS — Current)

This is the canonical, currently-used staging runbook. The real staging environment is https://stg.vayzomart.com, running on a shared VPS via Docker Compose - not Kubernetes/Helm (see Staging (legacy — Kubernetes/Helm) for the separate, on-hold future architecture; do not mix commands between the two pages).

Purpose

Deploy an already-built, immutable backend/frontend image pair to the shared-VPS staging stack, with a mandatory pre-migration backup + restore rehearsal, containerized schema migration, policy registry bootstrap, health verification, and external smoke test - all in one script that fails closed and automatically rolls back on any step's failure.

Topology

Value
Hostname stg.vayzomart.com
VPS directory /opt/vayzomart/staging
Compose project vayzomart-staging
Network vayzomart-staging-net (bridge, isolated from any other stack on the same VPS)
Frontend 127.0.0.1:${STAGING_FRONTEND_PORT} (canonical live value: 3200)
API 127.0.0.1:${STAGING_API_PORT} (canonical live value: 3201)
Database vayzomart_staging (user vayzomart_stg_user) - MySQL 8.2, never exposed to a host port
Cache Redis 7.2, never exposed to a host port
Data Protection key ring named volume vayzomart_staging_data_protection_keys mounted at /app/data-protection-keys (staging-only; see "Data Protection key ring" below)
Media (public) https://media.stg.vayzomart.com (FileStorage:R2:PublicBaseUrl)
R2 buckets stg-vayzomart (public), stg-vayzomart-private (invoices/grievance evidence)
Payments TEST mode only for either provider. Default provider Razorpay (rzp_test_*); PayU-only (the Production V1 model, B-28) via the "PayU-only staging profile" below - see "Safety guards"

3100/5100 are this stack's generic defaults (used only for an ad-hoc local run of the deploy script outside the VPS); 3300/3301 are reserved for a future production stack on the same shared VPS and are actively rejected by the deploy script if either port variable ever resolves to them - see "Safety guards" below.

PayU-only staging profile

Production V1 launches PayU-only with Razorpay disabled (business decision B-28). Switching staging to that model is a separately authorized step (PayU TEST certification); do not set these on the live VPS without that authorization. The .env.next for it:

STAGING_PAYMENT_PROVIDER=PayU
STAGING_PAYU_ENABLED=true
STAGING_PAYU_WEBHOOK_ENABLED=true
STAGING_PAYU_MERCHANT_KEY=<PayU TEST key, out of band>
STAGING_PAYU_MERCHANT_SALT=<PayU TEST salt, out of band>
STAGING_PAYU_RECONCILIATION_ENABLED=true        # explicit decision required (true/false)
STAGING_PAYU_REFUND_EXECUTION_ENABLED=false     # stays false unless separately authorized
STAGING_RAZORPAY_REFUND_EXECUTION_ENABLED=false # must be false while PayU is active
STAGING_RAZORPAY_WEBHOOK_ENABLED=false          # Razorpay webhook inactive
STAGING_API_HOST=stg.vayzomart.com              # api-stg.vayzomart.com has no DNS
# STAGING_RAZORPAY_KEY_ID / STAGING_RAZORPAY_KEY_SECRET: leave empty (not required, not used)

deploy-staging-vps.sh refuses the deploy unless every line above holds and the Compose file still pins PayU__Environment: Test and the test.payu.in endpoints. The API independently refuses to start without the PayU requirements, and with Razorpay__RefundExecutionEnabled=true. In this mode the shipped mobile app cannot pay (it is offered no provider it can launch) - see B-29.

Data Protection key ring

The API persists its ASP.NET Core Data Protection key ring to the staging-only named volume vayzomart_staging_data_protection_keys (DataProtection__KeysDirectory=/app/data-protection-keys), so recreating staging_api - every deploy and every rollback - keeps the same key ring. Startup fails closed if that directory is not a mounted volume or not writable by the container user (UID 1000): check docker logs staging_api for DataProtection:KeysDirectory if the API refuses to start. The first deploy that introduces the volume creates it empty and the API writes a new key ring once (payloads protected by the previous, ephemeral in-container key ring become unreadable - the same effect every container recreate had before). Never copy key files between staging and production. Re-certify with scripts/test-data-protection-persistence.sh <backend-image> (disposable; never touches the real volumes).

Kubernetes/Helm remains on hold. It is not compatible with the new production Data Protection persistence requirement until a durable PVC/shared key-store design is implemented and independently certified. The Helm chart is intentionally not wired for it (no temporary emptyDir workaround): outside Development the API refuses to start without a mounted, writable key volume.

Preconditions

  • An immutable backend/frontend image pair already pushed to GHCR, tagged with the exact release SHA (never latest - the deploy script refuses that tag outright). See End-to-End Deployment Runbook for how those images get built.
  • /opt/vayzomart/staging/.env already populated on the VPS with real values for (never committed, never logged): MYSQL_ROOT_PASSWORD, MYSQL_PASSWORD, REDIS_PASSWORD, JWT_SECRET_KEY, MFA_ENCRYPTION_KEY, STAGING_R2_ENDPOINT_URL, STAGING_R2_PUBLIC_BASE_URL, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, STAGING_RAZORPAY_KEY_ID (rzp_test_*), STAGING_RAZORPAY_KEY_SECRET. Optionally STAGING_FRONTEND_PORT=3200 / STAGING_API_PORT=3201 (the canonical override - see "Topology" above).
  • GHCR pull authentication available on the VPS - either GHCR_PULL_USER/GHCR_PULL_TOKEN supplied to the deploy invocation, or a persistent docker login ghcr.io already done on that host.
  • A valid TLS certificate for stg.vayzomart.com (Let's Encrypt/certbot, per infra/nginx/nginx-vps-staging.conf.template) already provisioned on the VPS host nginx, and DNS for stg.vayzomart.com already pointing at the VPS. This runbook does not provision either - both are one-time VPS/DNS operator setup, out of scope for the deploy script itself.
  • scripts/smoke-test.sh and scripts/generate-staging-vps-evidence.sh present alongside the deploy script on the VPS (both are mandatory - the deploy fails closed if either is missing).

Deploy

STAGING_VPS_DIR=/opt/vayzomart/staging \
IMAGE_TAG=<exact-release-sha> \
BACKEND_IMAGE=ghcr.io/thirunk/akshayabazaar-backend:<exact-release-sha> \
FRONTEND_IMAGE=ghcr.io/thirunk/akshayabazaar-frontend:<exact-release-sha> \
./scripts/deploy-staging-vps.sh <exact-release-sha>

The script runs nine steps, in order, each fail-closed:

  1. Pre-migration backup + restore rehearsal - if staging_mysql already has data, dumps it, generates a schema/migration/row-count manifest, restores the dump into a throwaway verify-DB, and byte-compares its manifest against the original before proceeding. A corrupt or non-restorable backup halts the deploy before anything else is touched.
  2. Atomically preserve release state & activate candidate config - snapshots the current .env/current_release.env as the rollback target, then promotes .env.next to .env.
  3. Safety guards (all fail-closed, all before any image pull or container mutation):
    • Port contract: STAGING_FRONTEND_PORT/STAGING_API_PORT must each be a valid TCP port, must differ from each other, and must never resolve to 3300/3301 (production-reserved).
    • Provider-aware payment TEST-mode gate: with STAGING_PAYMENT_PROVIDER=Razorpay (default), STAGING_RAZORPAY_KEY_ID must be set and start rzp_test_ (and PayU must not be enabled); with STAGING_PAYMENT_PROVIDER=PayU, the PayU-only profile below is enforced and Razorpay credentials are neither required nor used. Result recorded as PAYMENT_TEST_MODE_RESULT.
    • Database identity guard: MYSQL_DATABASE/MYSQL_USER/ConnectionStrings__DefaultConnection must not reference anything that looks like production.
    • R2 configuration preflight: public/private bucket names present and different from each other, R2 endpoint + access/secret keys present, and STAGING_R2_PUBLIC_BASE_URL is a real https:// URL - not localhost, not .example.invalid, not an obvious placeholder. This proves local config shape only; it does not verify Cloudflare-side bucket ACL/ privacy - see "What the R2 preflight does and does not prove" below.
  4. GHCR pull authentication - stdin token login (token unset from the shell immediately after) or a pre-existing persistent docker credential; fails closed if neither is available.
  5. Pull the exact candidate images by tag - no latest anywhere in this path.
  6. Containerized schema migration (--migrate-only) followed by policy registry bootstrap (--bootstrap-policy-registry, no --policy-package - publishes only the built-in draft placeholder package; see Policy Registry & Acceptance Ledger for exactly what that does and does not do in Production). Any failure here triggers automatic database recovery from the Step 1 snapshot.
  7. Start the candidate containers (staging_api, staging_frontend).
  8. Verify container + Redis health via Docker's own healthchecks - any failure triggers the same automated recovery/rollback path as Step 6.
  9. External smoke test + public readiness probe (scripts/smoke-test.sh, then https://stg.vayzomart.com/api/health/ready must return 200 over real, verified TLS - no -k/--insecure). Only after this passes does the script promote the candidate to current_release.env and generate mandatory release evidence.

What the R2 preflight does and does not prove

The Step 3 R2 configuration preflight (R2_CONFIG_PREFLIGHT_RESULT in the deploy result and evidence pack) proves exactly, and only:

  • The public and private R2 bucket names are both present and different from each other.
  • The R2 endpoint URL and access/secret keys are present (not empty).
  • STAGING_R2_PUBLIC_BASE_URL is an absolute https:// URL that isn't localhost/loopback and doesn't match a known placeholder pattern (.example.invalid, CHANGE_ME, YOUR_, DOMAIN_PLACEHOLDER, etc.).

It does not, and cannot, verify that the private R2 bucket is actually configured non-public on Cloudflare's side - that would require an authenticated call to the R2/S3 API, which this script does not make. Do not read a PASS here as "bucket privacy verified by Cloudflare" - read it as "the config we can see locally is internally consistent and not an obvious placeholder." The application itself independently fails closed at startup (Program.ValidateRuntimeConfiguration, backend) if FileStorage:R2:PublicBaseUrl is missing or placeholder-shaped, and PrivateFileStorageConfiguration.Validate separately refuses to start if the private bucket name is missing or identical to the public one.

Verification

curl -sSf https://stg.vayzomart.com/api/health/ready   # 200 = DB + Redis + schema all healthy
curl -sSf https://stg.vayzomart.com/health/live         # 200 = process is up (no dependency checks)

/health/ready checks database connectivity, Redis connectivity (reusing the same IConnectionMultiplexer the app already uses - PINGs it, does not open a second connection), and whether any EF Core migration is pending (cached up to 30s so frequent probing doesn't hammer __EFMigrationsHistory). /health/live is a pure liveness check with no dependency calls, by design - a transient Redis blip should never make an orchestrator recycle an otherwise-healthy process. Razorpay and notification-provider availability are deliberately not readiness dependencies; those already fail closed independently, at startup and at call time.

Rollback

./scripts/deploy-staging-vps.sh rollback

Restores the previous image tag and .env in an isolated env -i subshell (no candidate-env leakage), restores the database from the pre-migration snapshot when one was captured, and re-proves public readiness before declaring the rollback itself successful. Rollback is also triggered automatically by the deploy script on any Step 6-9 failure - see handle_post_migration_failure() in scripts/deploy-staging-vps.sh.

SuperAdmin bootstrap

Not part of this deploy script and never wired into it. Manual, operator-run, --bootstrap-superadmin - see SuperAdmin Bootstrap Runbook → Staging / Non-Production.

Evidence

scripts/generate-staging-vps-evidence.sh runs automatically as the deploy script's final step and is itself mandatory (a missing or failing evidence generator fails the whole deploy). It writes artifacts/release-evidence/staging-<tag>.md with every gate's state as a live measurement (PASS/FAIL/NOT_RUN/UNVERIFIED), never a static assertion - including a live re-probe of public readiness at evidence-generation time, so a stale historical PASS can never be reported as current.

Refund execution

Razorpay:RefundExecutionEnabled is off by default in the application (see RazorpayClient.CreateRefundAsync's own doc comment) - refund records are always created, but the actual outbound Razorpay refund API call is a no-op unless this flag is explicitly on. Staging wires it through docker-compose.staging.yml's Razorpay__RefundExecutionEnabled: ${STAGING_RAZORPAY_REFUND_EXECUTION_ENABLED:-false} mapping - missing or unset resolves to false (disabled), matching the application's own default. This staging-only variable is not referenced by any other current Compose file. Ports 3300/3301 are reserved for a future production marketplace stack on the shared VPS; this mapping does not configure or modify that future production stack.

This mapping is configuration readiness only. DO NOT set STAGING_RAZORPAY_REFUND_EXECUTION_ENABLED=true in the current staging environment yet.

Both Razorpay__WebhookEnabled and Razorpay__RefundExecutionEnabled remain disabled by default in staging as of this PR - see "Webhook readiness" immediately below for why refund execution must not be turned on until webhook ingress has been verified separately.

Once webhook readiness (below) has landed and been certified end-to-end on staging, exercising real Razorpay TEST refund dispatch means adding to the VPS .env:

STAGING_RAZORPAY_REFUND_EXECUTION_ENABLED=true

Only while STAGING_RAZORPAY_KEY_ID is confirmed to start with rzp_test_ - this deploy script's existing, unconditional RAZORPAY_TEST_MODE_RESULT gate (see "Deploy" above) already refuses to deploy staging at all if the key isn't test-mode, so there is no path through this procedure that can ever pair refund execution with a live key. This flag only changes whether staging's own already-test-mode Razorpay account executes refunds; it does not configure or affect any production deployment on the shared VPS.

After confirming the flag is live (and only once webhook readiness below has landed), check the API container logs for RefundExecutionEnabled=True at startup to verify it took effect before relying on it.

Webhook readiness

Razorpay__WebhookEnabled and Razorpay__WebhookSecret are wired through docker-compose.staging.yml's ${STAGING_RAZORPAY_WEBHOOK_ENABLED:-false} / ${STAGING_RAZORPAY_WEBHOOK_SECRET:-} mappings - missing or unset resolves to false/empty, matching the application's own default. Neither is turned on in the live VPS .env by this PR - this only makes it possible to enable them later, once webhook ingress (routing/TLS from the public internet to RazorpayWebhookController) has been verified independently and a real webhook secret has been generated in the Razorpay dashboard.

RazorpayWebhookController.Receive checks Razorpay:WebhookEnabled first, before reading the request body, before signature verification, and before any dispatch - if it is not exactly true, the controller fails closed with 503 regardless of whether a real, valid Razorpay:WebhookSecret is also configured. A real secret alone was previously sufficient to accept and dispatch signed webhooks even with the flag off; that gap is closed, so STAGING_RAZORPAY_WEBHOOK_ENABLED left at its default false is sufficient on its own to keep webhook ingress fully disabled on staging, independent of whatever secret value is present.

Program.ValidateRuntimeConfiguration fails startup outside Development in two layers: - Razorpay:WebhookEnabled=true with a missing or placeholder Razorpay:WebhookSecret (pre-existing check, unchanged). - New: Razorpay:RefundExecutionEnabled=true outside Development now additionally requires Razorpay:WebhookEnabled=true with a real, non-placeholder Razorpay:WebhookSecret - not merely that Razorpay credentials exist. This makes it impossible to accidentally activate automated refund execution without a working terminal-reconciliation path able to confirm each refund's outcome (see docs/devops/architecture/payment-production-hardening.md "Refund webhook / reconciliation" for the correlation mechanism this protects).

Refund-webhook-correlation P1 - resolved

A previously known P1 in the refund-webhook reconciliation path - refund.processed/ refund.failed could arrive before the local RazorpayRefundAttempt.ProviderRefundId was persisted, producing NoMatchingRefundAttempt and potentially leaving that attempt row Submitted/Processing indefinitely - is fixed as of this PR (see the architecture doc's "Refund webhook / reconciliation" section for the deterministic merchant-reference correlation mechanism, and MySqlRazorpayHardeningConcurrencyTests.cs for the real-InnoDB concurrency proof). This PR does not yet enable staging webhook ingress itself - that remains a separate, subsequent step: standing up real routing/TLS to RazorpayWebhookController on the shared VPS, generating a real webhook secret in the Razorpay dashboard, and certifying end-to-end delivery on staging before ever setting STAGING_RAZORPAY_WEBHOOK_ENABLED=true there.

Known follow-up tranches (explicitly out of scope here)

The following product-facing gaps were identified during staging readiness review and are not addressed by this runbook or its deploy script - they are separate, future work once PR #66's financial semantics land:

  • Web Purchase History / Buy Again UI (backend endpoints exist; no frontend page consumes them yet).
  • Mobile order cancellation UI (backend supports it; the mobile app has no cancel action).
  • Mobile invoice/purchase-history UI (does not exist yet).
  • Customer-facing refund-status visibility after an order cancellation specifically (as opposed to after a return) may be incomplete - confirm during UAT.