Fall #2, “Leaf It to GitOps (Argo CD & Envoy Gateway)

Goal & Abstract of Blog Post:

  • Goal:
    • Argo CD deploys your signed image from ECR, and Envoy Gateway routes traffic to it, all driven by Git. Changing a chore means editing YAML in a PR, and the cluster follows.
  • Abstract of Blog Post:
    • The signed image from Fall #1 now runs in the cluster
    • Deployed only from Git by Argo CD 3.6 and reached only through a platform-owned Envoy Gateway.
    • A chore change is a one-line pull request: merged.
    • It was live in about 90 seconds with no image build, no redeploy and no pod restart.
PartRepoWhat
A. Tidy the shedinfra, gitopsA1: Argo CD 2.11 → 3.x. A2: Popeye can read Gateway API objects
B. App manifestsgitopsNew lab-app AppProject, an Application, then namespace, ConfigMap (chores), Deployment, Service
C. The routegitopsA platform-owned Gateway for the app, plus the app’s HTTPRoute
D. Bring it up and prove it—Cluster up, watch Argo sync, curl through Envoy, change a chore via a PR, tear down

Part A1: Upgrade Argo CD to 3.x (infra repo)

  • Render your real values against the new chart, then question the output. The rendered YAML is exactly what will be installed, so it’s the source of truth.
  • Sources:
  • Noticed in previous blog post the schema error called terminatingReplicas under Deployments, it was build before newer k8s versions added that field to the deployment status – meaning Argo did not recognize that.
  • Here i want to ArgoCD to deploy the app, so had to update the gitops engine.
============
helm repo add argo https://argoproj.github.io/argo-helm 2>/dev/null
============
helm repo update argo
============
helm search repo argo/argo-cd --versions | head -5
============
NAME CHART VERSION APP VERSION DESCRIPTION
argo/argo-cd 10.18.2 v3.6.2 A Helm chart for Argo CD, a declarative, GitOps...
argo/argo-cd 10.18.1 v3.6.1 A Helm chart for Argo CD, a declarative, GitOps...
============

infra/helm-charts/argocd/values.yaml

  • Update 3 areas of the yaml

Delete applicationSet & enableNewGenerators

  • Renders as map[enableNewGenerators:true], literally a stringified Go map in argocd-cm. Argo ignores it.
============
configs:
params:
server.insecure: true
cm:
applicationSet:
enableNewGenerators: "true"
# Give child Applications a real health status so sync waves in
============

Delete insecure

  • The chart has no server.insecure value. The only real one is configs.params."server.insecure", which you already set.
  • The UI still runs without TLS inside the cluster (you reach it through port-forward).
    • configs.params."server.insecure": true is the setting that actually controls that.
============
server:
enabled: true
insecure: true
service:
============

CRDs

  • Delete installCRDs
============
installCRDs: true
============
# Chart 8+ moved CRD settings under "crds". install: chart manages the
# Application/AppProject CRDs. keep: they survive a helm uninstall, so a
# bad release can't delete every Application with them.
crds:
install: true
keep: true
============

infra/argocd.tf

  • Change version
============
resource "helm_release" "argocd_east" {
version = "7.1.0" # Argo CD v3.6.2
============
resource "helm_release" "argocd_east" {
version = "10.18.2" # Argo CD v3.6.2
============

Part B: the app’s home in Argo CD.

applications/lab/projects/lab-app.yaml

sourcerepos

  • Only your gitops repo. It can’t deploy from a random Helm chart or a fork.

destinations

  • Only say-when. Even a typo’d namespace in a manifest gets refused.

No secret, role, or rolebinding

  • The app doesn’t need them yet, so it isn’t allowed them. When Fall #3 adds External Secrets, we’ll add ExternalSecret here deliberately, in a reviewed PR.

HTTPRoute is allowed, Gateway isn’t.

  • The app can ask for traffic, but only the platform owns the front door. That’s the Part C split.

orphanedResources.warn

  • If someone kubectl applys something into say-when by hand, Argo shows a warning. It’s drift you can see.

Wave -10

  • Matches your other projects. Projects must exist before Applications, which was the Fall #0 deadlock lesson.

applications/lab/app/say-when-app.yaml

project: lab-app

  • Puts this Application inside theproject lab-app fence.

targetRevision: main

  • Only merged, reviewed code deploys. The same rule as CI.

prune: true

  • Remove a manifest from Git and Argo deletes the object. Git is the whole truth, not just the additions.

selfheal: true

  • If someone runs kubectl edit on the Deployment, Argo puts it back. That’s a nice demo for Part D.

No Createnamespace=true

  • Unlike your demos, the Namespace will be a real manifest in, so its security labels are reviewed in Git. Argo applies Namespaces before the objects inside them.

Wave 1:

  • After Envoy Gateway (-1) and the platform Gateway config (0).

Part B3: the app’s own manifests

  • We’ll create 5 small files under applications/lab/app/say-when-app.yaml

apps/say-when-app/namespace.yaml

========
metadata:
name: say-when
labels:
pod-security.kubernetes.io/enforce: restricted
========

apps/say-when-app/serviceaccount.yaml

  • Purpose:
    • Pods use the default service account unless told otherwise. ID w/no keys. Dedicated identity for the app. No Kubernetes API token is mounted:
      • the app never talks to the API. Fall #5 attaches AWS access (S3/SQS) to this account through EKS Pod Identity.
  • Sources:

apps/say-when-app/configmap-chores.yaml

apps/say-when-app/app/deployment.yaml

ChoiceWhy
image: …:64025c4d…The exact commit you verified with cosign verify. Never :latest, which Kyverno would deny anyway.
replicas: 2 + topology spreadOne Spot node being reclaimed doesn’t take the app down.
nodeSelector: arm64The image is single-architecture. This makes that explicit, so an x86 node added later can’t cause an exec format error.
runAsUser: 10001Matches the Dockerfile, and satisfies the restricted namespace and Kyverno’s require-run-as-nonroot.
readOnlyRootFilesystem: trueSafe because of PYTHONDONTWRITEBYTECODE=1: the app writes nothing to disk.
drop: ALL, no privilege escalationRequired by restricted, and by your Kyverno policies.
Probes on /healthzReadiness gates traffic until the app is up; liveness restarts it if it hangs.
requests and limitsKyverno’s require-resource-limits, and good for Karpenter’s bin-packing.
ConfigMap mounted as a folderNot subPath, so kubelet refreshes the file when the ConfigMap changes. main.py re-reads chores on every request, so a merged chore edit appears without restarting pods (within about a minute). That’s the Part D demo.

apps/say-when-app/app/servicee.yaml

Part C: Gateway & HTTPRoute

  • The B1 fence enforces this: lab-app can create an HTTPRoute but not a Gateway on the platform side.
  • This goes in the existing envoy-gateway-config folder, so my existing platform Application picks it up.
  • lab-platform already allows the envoy-gateway-system namespace and both of these kinds.
  • Sources:
ObjectWho owns itLives inWhy
Gateway (+ EnvoyProxy)Platform (lab-platform)envoy-gateway-systemThe listener, hostname and (in Fall #4) TLS and the load balancer are infrastructure decisions
HTTPRouteApp (lab-app)say-whenThe app decides which of its paths are reachable, and nothing else

apps/envoy-gateway-config/say-when-gateway.yaml

  • Purpose:
    • Platform-owned front door for the Say When app.
    • The platform owns the listener and hostname
    • The app only attaches an HTTPRoute from its own namespace.
      • ClusterIP for now (no AWS load balancer, no cost);
    • Fall #4 switches this EnvoyProxy to an NLB and adds HTTPS.

allowedRoutes with a namespace selector:

  • Demo Gateway uses from: Same, which only allows routes from its own namespace. This Gateway sits in envoy-gateway-system but serves an app in say-when, so it explicitly allows exactly one other namespace. The label kubernetes.io/metadata.name is set by Kubernetes itself, so nobody can just add a label to their namespace and attach a route to your front door.

hostname: cleanup.local

  • Pplaceholder name for testing. In Fall #4 it becomes a real subdomain with a certificate.

ClusterIP

  • No AWS load balancer yet, so it costs nothing. You reach it with port-forward in Part D.

Separate EnvoyProxy

  • Envoy Gateway runs one Envoy deployment per Gateway. Giving the app its own means it can later be scaled, or given an NLB, without touching the demo.

apps/say-when-app/httproute.yaml

  • Purpose:
    • Attach the app to the platform’s say-when Gateway and choose which paths are reachable from outside the cluster.
    • /metrics is deliberately NOT routed:
      • Prometheus scrapes the Service directly (Fall #7), so the public front door never exposes internal counters.
  • Sources:

An allow-list of paths, not / for everything.

  • Only the five public endpoints are routed; anything else gets a 404 from Envoy.

/metrics stay private

  • Prometheus will scrape it inside the cluster. Request counters aren’t for the public internet, and this matters once Fall #4 makes the Gateway public.

/docs & /openapi.json

  • Docs page loads its spec from /openapi.json, so routing one without the other breaks the page.

Separate EnvoyProxyparentRefs

  • Names the gateways namespace & the http listener to the route attaches to exactly one listener on exactly one Gateway.

Git status

  • git status –short
  • A applications/lab/app/say-when-app.yaml
  • A applications/lab/projects/lab-app.yaml
  • A apps/envoy-gateway-config/say-when-gateway.yaml
  • A apps/say-when-app/configmap-chores.yaml
  • A apps/say-when-app/deployment.yaml
  • A apps/say-when-app/httproute.yaml
  • A apps/say-when-app/namespace.yaml
  • A apps/say-when-app/service.yaml
  • A apps/say-when-app/serviceaccount.yaml

Part D: the cluster session

Bring up: HCP Terraform → say_when_infra → New run → Plan and apply

  • About 20 minutes
=========
aws eks update-kubeconfig --name say-when-east --region us-east-1
=========
kubectl get nodes -L kubernetes.io/arch # all arm64
=========

Bring up: Argo CD 3.6’s first boot

  • About 10 minutes
  • Can use CLI commands below or K9s or PF ArgoCD
=========
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d; echo
=========
kubectl -n argocd port-forward svc/argocd-server 8080:80
=========
http://localhost:8080
=========
=========
kubectl -n argocd get deploy argocd-server -o jsonpath='{..image}{"\n"}' # → quay.io/argoproj/argocd:v3.6.2
=========
kubectl -n argocd get applications -w
=========
kubectl -n argocd get applications
NAME READY STATUS RESTARTS AGE
argocd-application-controller-0 1/1 Running 1 (2m55s ago) 6m27s
argocd-applicationset-controller-7544645995-rcw5h 1/1 Running 0 6m28s
argocd-notifications-controller-75fc8799c-gbpp7 1/1 Running 0 8m55s
argocd-redis-544cdb5f85-jnz88 1/1 Running 0 8m55s
argocd-repo-server-5d6674f45-qvxpr 1/1 Running 0 8m55s
argocd-server-7f5f8d887d-z5j6j 1/1 Running 0 8m55s
quay.io/argoproj/argocd:v3.6.2
NAME SYNC STATUS HEALTH STATUS
argo-rollouts Synced Healthy
bootstrap-root Synced Healthy
envoy-gateway Synced Healthy
envoy-gateway-config Synced Healthy
kube-prometheus-stack Synced Healthy
monitoring-root Synced Healthy
network-policies Synced Healthy
=========

Check the App

=========
kubectl get ns say-when --show-labels | tr ',' '\n' | grep pod-security
=========
kubectl -n envoy-gateway-system get gateway say-when
=========
kubectl -n say-when get httproute say-when-app \
-o jsonpath='{range .status.parents[0].conditions[*]}{.type}={.status}{"\n"}{end}'
=========
pod-security.kubernetes.io/audit=restricted
pod-security.kubernetes.io/enforce=restricted
pod-security.kubernetes.io/warn=restricted
=========
NAME CLASS ADDRESS PROGRAMMED AGE
say-when envoy 172.20.99.215 True 6m38s
=========
Accepted=True
ResolvedRefs=True
=========

Curl thru Envoy – the real slim shady aka front door

  • About 10 minutes
  • /version returning prove end to end –
  • SHA → Git → Argo → signed ECR image → pod → Service → HTTPRoute → Gateway → Envoy
  • The 2 404s prove allow-list & hostname both work
=========
SVC=$(kubectl -n envoy-gateway-system get svc -l gateway.envoyproxy.io/owning-gateway-name=say-when -o name)
echo "$SVC"
=========
kubectl -n envoy-gateway-system port-forward "$SVC" 8888:80 >/dev/null 2>&1 &
=========
curl -s -H 'Host: cleanup.local' localhost:8888/version; echo # → the commit SHA, not "dev"
=========
curl -s -H 'Host: cleanup.local' localhost:8888/chores/rake-leaves; echo
=========
curl -s -o /dev/null -w '/metrics → %{http_code}\n' -H 'Host: cleanup.local' localhost:8888/metrics # → 404
=========
curl -s -o /dev/null -w 'wrong host → %{http_code}\n' localhost:8888/version # → 404
=========
service/envoy-envoy-gateway-system-say-when-0539cc1a
[1] 348167
{"version":"64025c4d4bb0d0ced62593cae2067f8fb202c2de"}
{"id":"rake-leaves","name":"Rake leaves","expect":["Rake","Leaf"],"reject":["Basketball","Sports Ball"],"min_confidence":80}
=========
/metrics → 404
=========
wrong host → 404
=========

PF 8081/docs

=========
kubectl -n say-when port-forward svc/say-when-app 8081:8080 >/dev/null 2>&1 &
=========
http://cleanup.local:8888/docs
=========

Self-Heal Demo – Scaling/Replicas

  • About 15 minutes
=========
kubectl -n say-when scale deploy say-when-app --replicas=5
=========
kubectl -n say-when get deploy say-when-app -w
=========
  • It jumps to 5, and Argo puts it back to 2 within seconds, because Git says 2

TEAR DOWN ROME!! or just your Cluster..

  • HCP Terraform → Settings → Destruction & Deletion → →
  • Confirm cluster is down
=========
aws ec2 describe-volumes --region us-east-1 \
--filters Name=status,Values=available \
Name=tag:kubernetes.io/created-for/pvc/namespace,Values=monitoring \
--query 'Volumes[].VolumeId' --output text