
Goal & Abstract of Blog Post:
- Goal:
- Argo CD deploys your signed image from ECR, and Envoy Gateway routes traffic to it, all driven by Git. Changing a chore means editing YAML in a PR, and the cluster follows.

- Abstract of Blog Post:
- The signed image from Fall #1 now runs in the cluster
- Deployed only from Git by Argo CD 3.6 and reached only through a platform-owned Envoy Gateway.
- A chore change is a one-line pull request: merged.
- It was live in about 90 seconds with no image build, no redeploy and no pod restart.
| Part | Repo | What |
|---|---|---|
| A. Tidy the shed | infra, gitops | A1: Argo CD 2.11 → 3.x. A2: Popeye can read Gateway API objects |
| B. App manifests | gitops | New lab-app AppProject, an Application, then namespace, ConfigMap (chores), Deployment, Service |
| C. The route | gitops | A platform-owned Gateway for the app, plus the app’s HTTPRoute |
| D. Bring it up and prove it | — | Cluster up, watch Argo sync, curl through Envoy, change a chore via a PR, tear down |

Part A1: Upgrade Argo CD to 3.x (infra repo)
- Render your real values against the new chart, then question the output. The rendered YAML is exactly what will be installed, so it’s the source of truth.
- Sources:
- Noticed in previous blog post the schema error called terminatingReplicas under Deployments, it was build before newer k8s versions added that field to the deployment status – meaning Argo did not recognize that.
- Here i want to ArgoCD to deploy the app, so had to update the gitops engine.
============helm repo add argo https://argoproj.github.io/argo-helm 2>/dev/null============helm repo update argo============helm search repo argo/argo-cd --versions | head -5============NAME CHART VERSION APP VERSION DESCRIPTION argo/argo-cd 10.18.2 v3.6.2 A Helm chart for Argo CD, a declarative, GitOps...argo/argo-cd 10.18.1 v3.6.1 A Helm chart for Argo CD, a declarative, GitOps...============
infra/helm-charts/argocd/values.yaml
- Update 3 areas of the yaml
Delete applicationSet & enableNewGenerators
- Renders as
map[enableNewGenerators:true], literally a stringified Go map inargocd-cm. Argo ignores it.
============configs: params: server.insecure: true cm: applicationSet: enableNewGenerators: "true" # Give child Applications a real health status so sync waves in ============
Delete insecure
- The chart has no
server.insecurevalue. The only real one isconfigs.params."server.insecure", which you already set. - The UI still runs without TLS inside the cluster (you reach it through port-forward).
configs.params."server.insecure": trueis the setting that actually controls that.
============server: enabled: true insecure: true service:============
CRDs
- Delete installCRDs
============installCRDs: true============# Chart 8+ moved CRD settings under "crds". install: chart manages the# Application/AppProject CRDs. keep: they survive a helm uninstall, so a# bad release can't delete every Application with them.crds: install: true keep: true============
infra/argocd.tf
- Change version
============resource "helm_release" "argocd_east" { version = "7.1.0" # Argo CD v3.6.2============resource "helm_release" "argocd_east" { version = "10.18.2" # Argo CD v3.6.2============
Part B: the app’s home in Argo CD.
applications/lab/projects/lab-app.yaml
- Sources:
- AppProject is a boundary & sets
- Which Git repos an Application may pull from
- Which namespaces it may deploy into
- Which kinds of objects it may create.
- Currently my platform tools sit in
lab-platform& the demos inlab-workloads.- The real app gets its own project, so a mistake in the app can never reach platform namespaces.
sourcerepos
- Only your gitops repo. It can’t deploy from a random Helm chart or a fork.
destinations
- Only
say-when. Even a typo’d namespace in a manifest gets refused.
No secret, role, or rolebinding
- The app doesn’t need them yet, so it isn’t allowed them. When Fall #3 adds External Secrets, we’ll add
ExternalSecrethere deliberately, in a reviewed PR.
HTTPRoute is allowed, Gateway isn’t.
- The app can ask for traffic, but only the platform owns the front door. That’s the Part C split.
orphanedResources.warn
- If someone
kubectl applys something intosay-whenby hand, Argo shows a warning. It’s drift you can see.
Wave -10
- Matches your other projects. Projects must exist before Applications, which was the Fall #0 deadlock lesson.
applications/lab/app/say-when-app.yaml
- Sources
- An Application is an instruction to Argo: “keep the cluster matching this folder.”
monitoring-rootalready recurses through everything underapplications/lab/, so dropping this file in is all it takes for Argo to discover it.
project: lab-app
- Puts this Application inside theproject lab-app fence.
targetRevision: main
- Only merged, reviewed code deploys. The same rule as CI.
prune: true
- Remove a manifest from Git and Argo deletes the object. Git is the whole truth, not just the additions.
selfheal: true
- If someone runs
kubectl editon the Deployment, Argo puts it back. That’s a nice demo for Part D.
No Createnamespace=true
- Unlike your demos, the Namespace will be a real manifest in, so its security labels are reviewed in Git. Argo applies Namespaces before the objects inside them.
Wave 1:
- After Envoy Gateway (
-1) and the platform Gateway config (0).
Part B3: the app’s own manifests
- Sources:
- We’ll create 5 small files under applications/lab/app/say-when-app.yaml
apps/say-when-app/namespace.yaml
- Purpose:
- The app’s namespace, with the Kubernetes “restricted” Pod Security. Standard ENFORCED: pods that run as root, keep capabilities or allow privilege escalation are rejected at admission.
- Why
enforce?- Demo namespaces only warn and audit.
- he real app gets the strictest built-in level, enforced, so Kubernetes itself refuses an insecure pod.
- The Dockerfile already qualifies: non-root UID 10001, and no root-only port.
- Sources:
========metadata: name: say-when labels: pod-security.kubernetes.io/enforce: restricted========
apps/say-when-app/serviceaccount.yaml
- Purpose:
- Pods use the
defaultservice account unless told otherwise. ID w/no keys. Dedicated identity for the app. No Kubernetes API token is mounted:- the app never talks to the API. Fall #5 attaches AWS access (S3/SQS) to this account through EKS Pod Identity.
- Pods use the
- Sources:
apps/say-when-app/configmap-chores.yaml
- Rather than retyping all 10 chores, generate the ConfigMap from the app repo’s file, so the two start identical
- Image ships own copy of chores.yaml, the cluster utilzied ConfigMap to override through the CHORES_FILE variable main.py already reads
apps/say-when-app/app/deployment.yaml
- Purpose:
- Run the signed say-when-app image from ECR (built and signed by say_when_app CI). Chores come from the ConfigMap, so a chore change in a Git PR, not a new image.
- Sources:
- https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
- https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
- https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
- https://kubernetes.io/docs/concepts/configuration/configmap/#configmaps-and-pods
| Choice | Why |
|---|---|
image: …:64025c4d… | The exact commit you verified with cosign verify. Never :latest, which Kyverno would deny anyway. |
replicas: 2 + topology spread | One Spot node being reclaimed doesn’t take the app down. |
nodeSelector: arm64 | The image is single-architecture. This makes that explicit, so an x86 node added later can’t cause an exec format error. |
runAsUser: 10001 | Matches the Dockerfile, and satisfies the restricted namespace and Kyverno’s require-run-as-nonroot. |
readOnlyRootFilesystem: true | Safe because of PYTHONDONTWRITEBYTECODE=1: the app writes nothing to disk. |
drop: ALL, no privilege escalation | Required by restricted, and by your Kyverno policies. |
Probes on /healthz | Readiness gates traffic until the app is up; liveness restarts it if it hangs. |
requests and limits | Kyverno’s require-resource-limits, and good for Karpenter’s bin-packing. |
| ConfigMap mounted as a folder | Not subPath, so kubelet refreshes the file when the ConfigMap changes. main.py re-reads chores on every request, so a merged chore edit appears without restarting pods (within about a minute). That’s the Part D demo. |
apps/say-when-app/app/servicee.yaml
- Purpose:
- Stable in-cluster address for the app pods. Envoy Gateway’s HTTPRoute
- Part C sends traffic here. ClusterIP only: nothing public yet.
- Sources:
Part C: Gateway & HTTPRoute
- The B1 fence enforces this:
lab-appcan create an HTTPRoute but not a Gateway on the platform side. - This goes in the existing
envoy-gateway-configfolder, so my existing platform Application picks it up. lab-platformalready allows theenvoy-gateway-systemnamespace and both of these kinds.- Sources:
| Object | Who owns it | Lives in | Why |
|---|---|---|---|
| Gateway (+ EnvoyProxy) | Platform (lab-platform) | envoy-gateway-system | The listener, hostname and (in Fall #4) TLS and the load balancer are infrastructure decisions |
| HTTPRoute | App (lab-app) | say-when | The app decides which of its paths are reachable, and nothing else |
apps/envoy-gateway-config/say-when-gateway.yaml
- Purpose:
- Platform-owned front door for the Say When app.
- The platform owns the listener and hostname
- The app only attaches an HTTPRoute from its own namespace.
- ClusterIP for now (no AWS load balancer, no cost);
- Fall #4 switches this EnvoyProxy to an NLB and adds HTTPS.
- Sources
allowedRoutes with a namespace selector:
- Demo Gateway uses
from: Same, which only allows routes from its own namespace. This Gateway sits inenvoy-gateway-systembut serves an app insay-when, so it explicitly allows exactly one other namespace. The labelkubernetes.io/metadata.nameis set by Kubernetes itself, so nobody can just add a label to their namespace and attach a route to your front door.
hostname: cleanup.local
- Pplaceholder name for testing. In Fall #4 it becomes a real subdomain with a certificate.
ClusterIP
- No AWS load balancer yet, so it costs nothing. You reach it with
port-forwardin Part D.
Separate EnvoyProxy
- Envoy Gateway runs one Envoy deployment per Gateway. Giving the app its own means it can later be scaled, or given an NLB, without touching the demo.
apps/say-when-app/httproute.yaml
- Purpose:
- Attach the app to the platform’s say-when Gateway and choose which paths are reachable from outside the cluster.
- /metrics is deliberately NOT routed:
- Prometheus scrapes the Service directly (Fall #7), so the public front door never exposes internal counters.
- Sources:
An allow-list of paths, not / for everything.
- Only the five public endpoints are routed; anything else gets a 404 from Envoy.
/metrics stay private
- Prometheus will scrape it inside the cluster. Request counters aren’t for the public internet, and this matters once Fall #4 makes the Gateway public.
/docs & /openapi.json
- Docs page loads its spec from
/openapi.json, so routing one without the other breaks the page.
Separate EnvoyProxyparentRefs
- Names the gateways namespace & the http listener to the route attaches to exactly one listener on exactly one Gateway.
Git status
- git status –short
- A applications/lab/app/say-when-app.yaml
- A applications/lab/projects/lab-app.yaml
- A apps/envoy-gateway-config/say-when-gateway.yaml
- A apps/say-when-app/configmap-chores.yaml
- A apps/say-when-app/deployment.yaml
- A apps/say-when-app/httproute.yaml
- A apps/say-when-app/namespace.yaml
- A apps/say-when-app/service.yaml
- A apps/say-when-app/serviceaccount.yaml
Part D: the cluster session
- About 75 minutes total
- Sources:
Bring up: HCP Terraform → say_when_infra → New run → Plan and apply
- About 20 minutes
=========aws eks update-kubeconfig --name say-when-east --region us-east-1=========kubectl get nodes -L kubernetes.io/arch # all arm64=========
Bring up: Argo CD 3.6’s first boot
- About 10 minutes
- Can use CLI commands below or K9s or PF ArgoCD
=========kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d; echo=========kubectl -n argocd port-forward svc/argocd-server 8080:80=========http://localhost:8080=========
=========kubectl -n argocd get deploy argocd-server -o jsonpath='{..image}{"\n"}' # → quay.io/argoproj/argocd:v3.6.2=========kubectl -n argocd get applications -w=========kubectl -n argocd get applicationsNAME READY STATUS RESTARTS AGEargocd-application-controller-0 1/1 Running 1 (2m55s ago) 6m27sargocd-applicationset-controller-7544645995-rcw5h 1/1 Running 0 6m28sargocd-notifications-controller-75fc8799c-gbpp7 1/1 Running 0 8m55sargocd-redis-544cdb5f85-jnz88 1/1 Running 0 8m55sargocd-repo-server-5d6674f45-qvxpr 1/1 Running 0 8m55sargocd-server-7f5f8d887d-z5j6j 1/1 Running 0 8m55squay.io/argoproj/argocd:v3.6.2NAME SYNC STATUS HEALTH STATUSargo-rollouts Synced Healthybootstrap-root Synced Healthyenvoy-gateway Synced Healthyenvoy-gateway-config Synced Healthykube-prometheus-stack Synced Healthymonitoring-root Synced Healthynetwork-policies Synced Healthy=========
Check the App

=========kubectl get ns say-when --show-labels | tr ',' '\n' | grep pod-security=========kubectl -n envoy-gateway-system get gateway say-when=========kubectl -n say-when get httproute say-when-app \ -o jsonpath='{range .status.parents[0].conditions[*]}{.type}={.status}{"\n"}{end}'=========pod-security.kubernetes.io/audit=restrictedpod-security.kubernetes.io/enforce=restrictedpod-security.kubernetes.io/warn=restricted=========NAME CLASS ADDRESS PROGRAMMED AGEsay-when envoy 172.20.99.215 True 6m38s=========Accepted=TrueResolvedRefs=True=========

Curl thru Envoy – the real slim shady aka front door
- About 10 minutes
- /version returning prove end to end –
- SHA → Git → Argo → signed ECR image → pod → Service → HTTPRoute → Gateway → Envoy
- The 2 404s prove allow-list & hostname both work
=========SVC=$(kubectl -n envoy-gateway-system get svc -l gateway.envoyproxy.io/owning-gateway-name=say-when -o name)echo "$SVC"=========kubectl -n envoy-gateway-system port-forward "$SVC" 8888:80 >/dev/null 2>&1 &=========curl -s -H 'Host: cleanup.local' localhost:8888/version; echo # → the commit SHA, not "dev"=========curl -s -H 'Host: cleanup.local' localhost:8888/chores/rake-leaves; echo=========curl -s -o /dev/null -w '/metrics → %{http_code}\n' -H 'Host: cleanup.local' localhost:8888/metrics # → 404=========curl -s -o /dev/null -w 'wrong host → %{http_code}\n' localhost:8888/version # → 404=========service/envoy-envoy-gateway-system-say-when-0539cc1a[1] 348167{"version":"64025c4d4bb0d0ced62593cae2067f8fb202c2de"}{"id":"rake-leaves","name":"Rake leaves","expect":["Rake","Leaf"],"reject":["Basketball","Sports Ball"],"min_confidence":80}=========/metrics → 404=========wrong host → 404=========
PF 8081/docs
=========kubectl -n say-when port-forward svc/say-when-app 8081:8080 >/dev/null 2>&1 &=========http://cleanup.local:8888/docs=========

Self-Heal Demo – Scaling/Replicas
- About 15 minutes

=========kubectl -n say-when scale deploy say-when-app --replicas=5=========kubectl -n say-when get deploy say-when-app -w=========

- It jumps to 5, and Argo puts it back to 2 within seconds, because Git says 2


TEAR DOWN ROME!! or just your Cluster..
- HCP Terraform → Settings → Destruction & Deletion → →
- Confirm cluster is down
=========aws ec2 describe-volumes --region us-east-1 \ --filters Name=status,Values=available \ Name=tag:kubernetes.io/created-for/pvc/namespace,Values=monitoring \ --query 'Volumes[].VolumeId' --output text