Skip to content

DevOps Automation

This section documents the feature/devops-automation branch's work: a repository audit, environment standardization, PowerShell deployment/verification scripts, and Docker/Helm/ Terraform review. Written to be portable into the MkDocs + Cloudflare Pages DevOps portal - each page below stands alone with Purpose/Prerequisites/Files/Command/Expected result/Verify/ Failure scenarios/Rollback for anything that changes cluster or infrastructure state.

Pages

  • Environments - which config backs local/dev/staging/prod, and why (lives outside this documentation portal's own docs/devops/ tree, in the repository's environments/ directory - linked via GitHub, not a portal page).
  • Scripts reference - every script in scripts/, what it does, how to run it, how to roll it back.
  • Verification - the verify-staging.ps1 check pyramid and how to read its output.
  • Terraform - module/environment structure, the new dev scaffold, module-naming mapping.
  • Rollback procedures - Helm and Terraform rollback, consolidated in one place.

Repository audit findings

Full repository review performed 2026-08-18 against review/inventory-reservation (branched into feature/devops-automation). Findings below are grouped by the audit's own checklist.

What's already solid (no change needed)

  • Dockerfiles (backend/Dockerfile, frontend/Dockerfile): multi-stage builds, pinned non-root users (uid 1000 / nginx's uid 101), HEALTHCHECK directives, OCI metadata labels populated from build args, .dockerignore present and correct for both. Already reflects the non-root/UID-pinning fixes from the local-staging debugging pass earlier in this branch's history.
  • docker-compose.yml/docker-compose.dev.yml: per-service healthchecks, depends_on with condition: service_healthy, .env-driven config with safe local-only fallback defaults (never used outside docker compose up on a developer's own machine), a dev-profile-gated Adminer service that never runs by default.
  • CI/CD split: .github/workflows/ci.yml (fast build+test signal, gates PRs) and Jenkinsfile (full build → Docker image → Helm deploy → smoke test pipeline, described in docs/CI_CD.md) are a deliberate two-tier setup, not duplication.
  • Jenkinsfile secrets handling: every credential is referenced by Jenkins Credentials-store ID only (withCredentials); no literal secret anywhere in the file - verified by direct read.
  • Terraform remote state / secrets discipline: terraform/.gitignore already excludes **/.terraform/, *.tfstate*, and real *.tfvars (keeping only *.tfvars.example); every *.tfvars.example file checked contained only CHANGE_ME placeholders.

Missing configuration (fixed on this branch)

  • helm/akshayabazaar/values-dev.yaml existed but was stale: its existingSecretName still pointed at the pre-isolation shared Secret name, it had no backend.resources/ frontend.resources despite its own header claiming to be the lowest-footprint tier, and its ingress.tls.enabled: true pointed at infrastructure (terraform/environments/dev, a dev DNS record/cert) that never existed. All three fixed - see environments/dev/README.md "Known gaps" for the full before/after.
  • terraform/environments/dev/ did not exist at all, despite values-dev.yaml implying a dev tier was intended. Added as a structural scaffold (validates, never applied) - see legacy-terraform-notes.md.
  • .github/workflows/ci.yml tested the frontend on Node 20 while frontend/Dockerfile's build stage (and the lockfile fix from earlier in this branch) require Node 22 for the same jsdom transitive-dependency reason. CI now matches the Dockerfile.
  • No PowerShell deployment/verification scripts existed - only bash (scripts/deploy-staging.sh, scripts/deploy-production.sh, scripts/smoke-test.sh), which don't run natively on this team's Windows workstations without WSL/Git Bash. Added the requested seven .ps1 scripts - see scripts.md.

Duplicated configuration (found, not duplicated further)

  • The audit's suggested values-prod.yaml would have duplicated the existing, already-working values-production.yaml. environments/prod/README.md documents the existing name instead of introducing a second file for the same environment - see environments/README.md "Why this structure".
  • The audit's suggested Terraform module names (network/compute/database/kubernetes) already exist as more specific modules (hetzner-network, hetzner-server, hetzner-firewall, cloudflare-dns, cloudflare-security, cloudflare-r2). Renaming them would risk breaking terraform/environments/{platform,staging,production}/main.tf's already-validated, never-yet-broken module references for a cosmetic-only gain - see legacy-terraform-notes.md "Module naming" for the mapping instead.

Obsolete/dead files found and removed

  • Two empty, stray directories with corrupted names - helm/akshayabazaar;C and scripts/mysql-backup.sh;C (0 bytes, no children, not tracked by git since empty directories aren't tracked). Almost certainly an artifact of an earlier shell command gone wrong; removed as harmless cleanup.
  • Seven root-level debug/render files left over from live-diagnosing the local k3d NetworkPolicy issue earlier in this branch's history (*-debug.txt, *-rendered.yaml, local-*.yaml, mysql-local.yaml) - deleted, and matching patterns added to .gitignore so this class of file doesn't get re-proposed for commit. One of them (local-infra.yaml) held plaintext local-only dev passwords - see "Secrets" below.

Security / hard-coded values

  • No real secrets found committed to source control. Scanned every tracked .yml/.yaml/ .json/.env*/.tf/.tfvars file for common secret patterns; every match was a CHANGE_ME placeholder. .env exists at the repo root but is gitignored and was never tracked (confirmed via git ls-files / git check-ignore).
  • One near-miss, already remediated: local-infra.yaml (an untracked, root-level debug file from earlier local-staging troubleshooting) held plaintext dev-only passwords for the local k3d cluster's hand-rolled MySQL/Redis pods. Never committed, but was sitting on disk in the repo root; deleted as part of this pass's cleanup (see above).
  • Weak fallback defaults in docker-compose.yml (e.g. ${MYSQL_ROOT_PASSWORD:-rootpassword}) are a low-severity, local-only concern: they only ever apply when .env is absent, on a developer's own machine, never in any Kubernetes environment (which requires an explicit, pre-created Secret with no fallback). Left unchanged - flagged here for visibility rather than changed, since altering compose's local-dev ergonomics wasn't asked for and isn't remotely reachable to be an actual risk.

Safety guardrails observed

No Terraform apply/destroy was run against any environment. No helm install/upgrade/ rollback was run against staging or production's real infrastructure - only against the already-running local k3d cluster's read-only verify-staging.ps1 check. No DNS (Cloudflare or otherwise) was modified. No secret was rotated, deleted, or printed - every Secret check in every script/verification run is existence-only (kubectl get secret <name>, never -o yaml/-o json on Secret contents). Nothing was merged to master; all work is on feature/devops-automation.

Tests/validations performed

Check Result
dotnet build / dotnet test (backend) Build succeeded; 410/410 tests passed
npm test (frontend) 108/108 tests passed
helm lint - local-staging / staging / production / dev overlays 0 failures on all four
helm template - all four overlays (monitoring+backup enabled where applicable) rendered successfully
terraform fmt -check -recursive (all environments) no diffs - already correctly formatted
terraform validate - platform / staging / production / dev "Success! The configuration is valid." on all four
PowerShell AST parse-validation, all 7 new .ps1 scripts 0 syntax errors; fixed one PS 7-only ?? usage and two unbraced $var: string-interpolation bugs found this way
verify-staging.ps1 executed live against the local k3d cluster 14 PASS, 0 WARNING, 0 FAIL

Items requiring your approval

Nothing here was done - each is a deliberate stop, documented for you to decide:

  1. Applying terraform/environments/dev/ - would provision a real Hetzner VM, Cloudflare DNS record, and R2 bucket, and incur real cost. Structural scaffold only; see environments/dev/README.md.
  2. Flipping values-dev.yaml's ingress.tls.enabled back to true - needs the above applied first (a real cert-issuing path for dev.akshayathreadworld.in).
  3. Any deploy to the real staging cluster - deploy-staging.ps1/rollback-staging.ps1/ stop-staging.ps1 were written and parse-validated but never executed against real staging (no reachable real-staging kubeconfig from this environment - only the local k3d cluster was reachable). Recommend a dry run (stop-staging.ps1/rollback-staging.ps1 without -Yes, which only print state) before the first real use.
  4. Anything involving production - not touched, not planned to be touched, per your explicit instruction.

Items intentionally not changed

  • run-local-api.ps1 (repo root) - a pre-existing, unrelated local dev helper (runs the API directly against the local MySQL on port 3308) - out of scope for this pass, left as-is.
  • docker-compose.yml's weak local-only fallback passwords - see "Security / hard-coded values" above.
  • Terraform module directory names (hetzner-*/cloudflare-* instead of network/compute/ database/kubernetes) - see "Duplicated configuration" above.
  • No PodDisruptionBudget added to the Helm chart - not in the audit's explicit Kubernetes/Helm checklist, and every current environment runs replicaCount: 1, where a PDB with minAvailable: 1 would block voluntary node drains rather than protect availability. Worth revisiting once any environment runs 2+ replicas (production's HPA can already scale to 2-6, see values.yaml).