Skip to content

Local Development Troubleshooting

Issues below were diagnosed live against this project's own local k3d cluster during the DevOps automation work - not hypothetical.

Python still not found

If python --version still returns the Microsoft Store stub message after installing:

  1. Close and reopen your terminal (or open a fresh VS Code terminal window) - PATH is only re-read from the registry when a new shell process starts. Do not edit PATH by hand as a first step.
  2. Still failing after a fresh terminal? Check the Windows App Execution Alias for Python: Settings → Apps → Advanced app settings → App execution aliases - if "App Installer" is toggled on for python.exe/python3.exe, Windows intercepts the command before it ever reaches a real install and opens the Microsoft Store instead. Turn those toggles off.
  3. Confirm where Python actually installed: 
    Get-ChildItem "$env:LOCALAPPDATA\Programs\Python" -ErrarAction SilentlyContinue
where.exe python
  4. If winget install itself hangs at "Starting package install..." for several minutes with no progress, it is very likely a Microsoft Store-mediated package (e.g. the Python Install Manager, 9NQ7512CXL7T) waiting on an interactive Store sign-in/consent that a scripted/ automated session can never satisfy. Cancel it (Ctrl+C, or close the terminal) and install the official python.org build instead, which downloads directly and does not go through the Store: 
    winget install Python.Python.3.12 -e --accept-package-agreements --disable-interactivity
    Do not run both installers "just in case" - pick one, per "do not install multiple competing Python distributions."

k3d cluster create fails: "port already allocated"

Something else on your machine already owns host port 80 (or 443). Check what's already listening before changing this project's own port mapping:

Get-NetTCPConnection -LocalPort 80 -State Listen -ErrorAction SilentlyContinue

If the local cluster's own load balancer already exists but only maps to a non-80 port (e.g. it was created with -p 8080:80@loadbalancer originally), you do not need to recreate the whole cluster - k3d cluster edit can add the missing mapping without touching the server/agent nodes (and therefore without touching any pod/data on them):

k3d cluster edit akshaya-staging --port-add "80:80@loadbalancer"
docker ps --filter "name=k3d-akshaya-staging-serverlb"   # confirm the new mapping

Migration Job fails: BackoffLimitExceeded, MySQL connection error in the logs

kubectl get pods -n staging -l job-name=akshaya-local-backend-migrate
kubectl logs -n staging <pod-name> --all-containers --tail=200

If the log shows a MySQL connection error to mysql-local, check the default-deny-ingress NetworkPolicy's selector - it must scope to this release's own pods, not an empty {} (which Kubernetes resolves to "every pod in the namespace," silently blocking a namespace-local mysql-local/redis-local pod the chart doesn't know about):

kubectl get networkpolicy -n staging akshaya-local-default-deny-ingress -o jsonpath='{.spec.podSelector}'
Expect {"matchLabels":{"app.kubernetes.io/instance":"akshaya-local","app.kubernetes.io/name":"akshayabazaar"}} - if it's {} instead, the fix in helm/akshayabazaar/templates/networkpolicy.yaml hasn't reached the live cluster. Because this policy is a normal (non-hook) chart resource, Helm only syncs it after the pre-upgrade migration hook succeeds - if the hook is what's failing because of this exact policy, retrying helm upgrade alone cannot fix itself. Break the deadlock once by applying just the corrected NetworkPolicy directly, then retry:

helm template akshaya-local helm/akshayabazaar -f helm/akshayabazaar/values-local-staging.yaml `
  --show-only templates/networkpolicy.yaml | kubectl apply -f -
Then retry .\scripts\deploy-local.ps1.

Backend Service unreachable from an ad-hoc debug pod, but the app itself is healthy

If kubectl exec from a throwaway/unlabeled pod into the backend Service fails, but .\scripts\verify-staging.ps1 shows the backend itself Healthy - the debug pod almost certainly doesn't carry the exact app.kubernetes.io/name/app.kubernetes.io/instance/ app.kubernetes.io/component: frontend labels the backend's *-allow-ingress NetworkPolicy requires. Test from the real frontend pod instead:

kubectl exec -n staging deploy/akshaya-local-frontend -- wget -qO- http://akshaya-local-backend:8080/health/live

CreateContainerConfigError: image has non-numeric user

The image wasn't rebuilt after a Dockerfile change (both Dockerfiles pin non-root numeric UIDs - appuser at uid 1000 for backend, nginx at uid 101 for frontend - required for Kubernetes' runAsNonRoot: true to verify without a container-runtime start). Rebuild and re-import:

.\scripts\build.ps1 -Local

Frontend/Ingress returns nothing on http://staging.akshaya.local

Check the load balancer's actual port mapping (k3d frequently maps container port 80 to a different host port unless created with an explicit -p 80:80@loadbalancer):

docker port k3d-akshaya-staging-serverlb

If it shows 80/tcp -> 0.0.0.0:8080 instead of :80, either recreate/edit the mapping (see above), or pass the actual port to the verification script:

.\scripts\verify-staging.ps1 -Context k3d-akshaya-staging -Namespace staging -ReleaseName akshaya-local -IngressHost staging.akshaya.local -IngressPort 8080