Skip to content

Fully Automated Vayzo Delivery

Objective

Routine Vayzo delivery must not require SSH sessions, hand-written kubectl commands, manual EF Core migration execution, browser-by-browser health checking, provider credential checking, load smoke commands, or manual rollback. The supported path is:

GitHub -> Jenkins -> Docker/GHCR -> Terraform -> Kubernetes/Helm -> automated verification

Production intentionally keeps one human approval after the exact immutable image has passed staging. Everything after that approval is automated.

Delivery architecture

Developer / agent
      |
      v
GitHub branch / PR
      |
      v
GitHub Actions gates
      +--> .NET build + xUnit
      +--> ephemeral MySQL 8.2 concurrency smoke
      +--> frontend auth/session tests + lint + build
      +--> NuGet/npm vulnerability gates
      +--> Terraform fmt/validate
      +--> Helm lint/render
      +--> shell/script syntax + MkDocs strict
      |
      v
Jenkins multibranch pipeline
      +--> same application quality gates
      +--> Docker immutable build + Trivy
      |
      v
GHCR :<12-char-git-sha>
      |
      v
Terraform plan/apply (staging)
      |
      v
Provider preflight before Secret sync
      +--> R2 unique object upload/read/compare/delete
      +--> Razorpay credential/connectivity GET (no transaction)
      |
      v
Kubernetes Secret synchronization
      |
      v
Helm --atomic migration + rollout
      |
      v
Automated staging verification
      +--> Kubernetes/Helm/pods/services/images
      +--> in-cluster live + ready (DB + Redis)
      +--> external frontend + API proxy
      +--> security headers/CORS/anonymous authorization
      +--> Playwright deployed storefront/route guards
      +--> ephemeral MySQL concurrency smoke
      +--> k6 lightweight performance smoke
      |
      +--> FAIL -> automatic Helm rollback
      |
      v
Release evidence
      |
      v
v* release tag only
      |
      v
Production Terraform plan
      |
      v
ONE APPROVAL
      |
      v
Terraform apply -> provider preflight -> Secret sync -> Helm -> migration -> rollout
      |
      v
Production verification
      +--> Kubernetes/API/security/Playwright
      +--> no recurring load test
      +--> no real-money transaction
      |
      +--> FAIL -> automatic Helm rollback
      |
      v
Production release evidence

Branch and release behavior

Source Build/test/scan Push SHA images Staging Production
Feature/PR Yes No No No
master Yes Yes No No
release/* Yes Yes Automatic No
SemVer tag v*.*.* Yes Yes Automatic One approval, then automatic

A production release never rebuilds after staging. The backend/frontend images deployed to production use the same immutable commit-SHA tag that staging already validated.

Jenkins actions

The root Jenkinsfile exposes three modes:

  • Deploy - normal CI/CD behavior described above.
  • Rollback - explicit rollback to a previous Helm revision followed by automated verification.
  • Infrastructure - one-off Terraform plan/approval/apply for platform, staging, production, or shared infrastructure.

Routine release/* staging infrastructure reconciliation is automatic. Production Terraform apply happens only after the release approval gate.

One-time Jenkins credentials

These are configuration prerequisites, not per-release manual steps. Values live only in Jenkins Credentials and must never be committed.

Jenkins credential ID Type Purpose
container-registry-creds Username/password or token GHCR push
k3s-kubeconfig-dev Secret file Dev Kubernetes access
k3s-kubeconfig-staging Secret file Staging Kubernetes access
k3s-kubeconfig-production Secret file Production Kubernetes access
terraform-state-r2-creds Username/password R2 S3-compatible Terraform-state access/secret keys
terraform-platform-tfvars Secret file Platform Terraform variables
terraform-staging-tfvars Secret file Staging Terraform variables
terraform-production-tfvars Secret file Production Terraform variables
terraform-shared-tfvars Secret file Shared/Jenkins Terraform variables
vayzo-staging-app-env Secret file Staging Kubernetes application Secret data
vayzo-production-app-env Secret file Production Kubernetes application Secret data

Application Secret file shape

The two vayzo-*-app-env credentials are plain KEY=value secret files in Jenkins. Staging and production values must be completely independent.

ConnectionStrings__DefaultConnection=<environment DB connection string>
ConnectionStrings__Redis=<environment Redis connection string>
Jwt__SecretKey=<32+ byte environment-specific random value>
FileStorage__R2__AccessKey=<bucket-scoped R2 access key>
FileStorage__R2__SecretKey=<bucket-scoped R2 secret key>
Razorpay__KeySecret=<test-mode in staging; live-mode in production>

Before those values are synchronized into Kubernetes, scripts/sync-k8s-secrets.sh validates the required fields and exercises the R2/Razorpay credentials. Secret values are never printed.

One-time non-secret Jenkins environment configuration

Shared Terraform state

TERRAFORM_STATE_R2_ENDPOINT=https://<cloudflare-account-id>.r2.cloudflarestorage.com

This removes the old requirement to edit ACCOUNT_ID manually in committed Terraform files. For real plan/apply, scripts/terraform-ci.sh creates an ignored temporary Terraform worktree under artifacts/, substitutes the endpoint there, and leaves repository source untouched.

Staging

STAGING_FRONTEND_HOST
STAGING_R2_ENDPOINT_URL
STAGING_R2_PUBLIC_BASE_URL
STAGING_RAZORPAY_KEY_ID   # must start rzp_test_

Production

PRODUCTION_FRONTEND_HOST
PRODUCTION_WWW_HOST
PRODUCTION_R2_ENDPOINT_URL
PRODUCTION_R2_PUBLIC_BASE_URL
PRODUCTION_RAZORPAY_KEY_ID   # must start rzp_live_

The pipeline fails before infrastructure/deployment if any required value is missing or if the Razorpay mode is wrong.

[!WARNING] This Jenkins production stage belongs to the on-hold Kubernetes/Helm path and predates business decision B-28 (Production V1 = PayU only, Razorpay disabled). Its PRODUCTION_RAZORPAY_KEY_ID=rzp_live_* requirement is therefore obsolete and must be made provider-aware before this path is ever used for production (tracked in docs/legal/HUMAN_GATE_CLOSURE_MATRIX.md). The authoritative Docker Compose/VPS path (docker-compose.production.yml) already wires PayU LIVE and does not require Razorpay.

Terraform automation

scripts/terraform-ci.sh runs Terraform in the official hashicorp/terraform:1.9 Docker image, so no Terraform binary has to be maintained on the Jenkins agent.

PR/static validation:

scripts/terraform-ci.sh validate

Real Jenkins reconciliation:

scripts/terraform-ci.sh plan staging
scripts/terraform-ci.sh apply staging

The apply consumes the exact saved plan generated in the same Jenkins workspace; it does not calculate a different plan at apply time.

Provider preflight and Kubernetes Secret automation

Before Kubernetes Secret synchronization, staging and production run two provider checks:

  1. Cloudflare R2 - scripts/r2-e2e.sh writes a unique ci/probes/... object, reads it back, byte-compares it, deletes it and confirms deletion.
  2. Razorpay - scripts/razorpay-preflight.sh validates the environment-specific key mode and performs an authenticated read-only collection request. It creates no order and no payment.

Only after both succeed does scripts/sync-k8s-secrets.sh reconcile the namespace and Secret.

Helm and database migration

Helm receives both the immutable repository and tag explicitly. It never relies on the chart's placeholder ghcr.io/OWNER/... repository or a floating latest tag.

The chart's pre-install/pre-upgrade migration Job executes the backend image with --migrate-only. If migration fails, Helm fails before the new serving pods are promoted.

All staging/production upgrades use:

--wait
--atomic
--history-max 10

--atomic covers migration/rollout failure. Behavioral failures after Kubernetes reports Ready are covered by the verification/rollback stage.

Automated application and concurrency testing

Frontend

frontend/package.json exposes:

npm test -> node --test src/auth-hardening.test.js

The existing test suite validates session bootstrap, refresh-token behavior, concurrent 401 refresh, in-memory access-token handling, logout, and customer/seller/admin route-state rules.

Backend

The normal xUnit suite already contains customer, seller and admin API E2E/authorization coverage, payment/inventory rules, coupons, returns, settlements, notifications and other marketplace invariants.

The destructive real-MySQL concurrency tests remain separated with Category=mysql-smoke. scripts/run-mysql-smoke.sh starts its own ephemeral mysql:8.2 container and runs the inventory reservation and refresh-token concurrency classes against that disposable database. It must never be pointed at staging or production data.

Automated deployed-environment verification

scripts/verify-deployment.sh verifies the deployment itself rather than repeating application business tests already covered by xUnit.

Core checks for staging and production:

  1. Kubernetes cluster connectivity.
  2. Namespace existence.
  3. Helm release visibility.
  4. Deployment rollout completion.
  5. Pod phase/readiness.
  6. Immutable image tag identity.
  7. Backend/frontend Service endpoints.
  8. Backend /health/live from the real frontend pod.
  9. Backend /health/ready from the real frontend pod (DB + Redis dependency gate).
  10. External frontend HTTP 200.
  11. External frontend -> /api/products proxy HTTP 200.
  12. Security headers.
  13. CORS preflight for the configured frontend origin.
  14. Anonymous rejection of customer/admin/seller protected API paths.
  15. Playwright storefront rendering and client-side routing.
  16. Playwright anonymous customer/admin/seller route-guard redirects.
  17. Accessible login validation smoke.

Staging additionally runs:

  1. Real MySQL concurrency smoke against an ephemeral CI database.
  2. k6 lightweight smoke for /, /products, and catalog API, with <1% request failure and p95 <1.5s thresholds.

Production intentionally skips recurring load/concurrency activity. It promotes the exact artifact already validated in staging and performs only lightweight/read-only verification.

Automatic rollback

There are two layers:

  1. Helm --atomic - if migration/rollout does not become healthy, Helm restores the prior release automatically.
  2. Behavior verification rollback - if verify-deployment.sh fails after a technically successful rollout, Jenkins runs helm rollback, verifies the restored release, archives the evidence and fails the pipeline.

The explicit Jenkins ACTION=Rollback mode remains available for operator-initiated recovery.

Release evidence

After successful staging or production verification, scripts/generate-release-evidence.sh records the Git SHA/tag, immutable image tag, namespace, Helm revision/status, deployed images, target URL and mandatory automated gates. Jenkins archives evidence and verification output. Secrets are never written to evidence.

Remaining controlled validation

The automated release path intentionally does not create recurring real customer/seller/admin records or real payments just to prove a deployment. Those business invariants are already covered by the existing .NET E2E/authorization/payment suites.

Before first public launch, one controlled staging UAT tranche should still prove the full authenticated browser journey with disposable test accounts and Razorpay test mode checkout:

  • customer registration/login/cart/checkout/order;
  • seller onboarding/listing/inventory/fulfilment;
  • admin management views/actions;
  • Razorpay test checkout callback/signature path.

If these become stable enough to run repeatedly without polluting staging data, they can later be promoted to credentialed Playwright fixtures. Production must never run recurring real-money test transactions.