Fall Cleaning #0: The Game Plan (Baseline, Budget & Roadmap)

Theme & Goal

  • Series of blog posts showing progress of creating an app on top of my EKS Cluster
  • Goal:
    • Assign 10 chores by email
    • Take photos in
    • Merge the infra and GitOps cleanup patches
    • Move the cluster from Kubernetes 1.31 to 1.35
    • Get a clean baseline.
    • Share what was learned & anything that that broke along the way.

Surprises & Troubleshooting

  • CI failure
    • The patch upgraded the optional west cluster’s EKS module from v19 to v20. v19 needed the hashicorp/kubernetes provider and v20 doesn’t, so the providers Terraform needs changed.
    • CI uses a read-only lock file, so it refused the change
======
terraform init -backend=false # rewrites .terraform.lock.hcl locally, no backend or AWS calls
======
git diff .terraform.lock.hcl # only the hashicorp/kubernetes block removed (-21 lines)
======
git add .terraform.lock.hcl && git commit && git push
======
  • TFLink Failure
    • 403 API rate limit exceeded & TFLint downloads its plugins from GitHub’s API without logging in, and shared CI runners hit the anonymous rate limit.
- name: Initialize TFLint plugins
run: tflint --init
env:
GITHUB_TOKEN: ${{ github.token }}
  • Orphaned EBS volumes
    • Found listed 33 unattached volumes, but not a big deal from a data POV cuz the data was safe to lose:
      • Prometheus held only 24 hours of local data
      • Thanos store gateway is a cache over S3.
        • There were 13 × 2 GB Prometheus volumes and 20 × 10 GB Thanos store gateway volumes, going back to Aug 14.
    • Prometheus and the Thanos store gateway each ask Kubernetes for a disk, and the EBS CSI driver creates an EBS volume for each request.
      • terraform destroy deleted the cluster without first deleting those disk requests, so EBS never cleaned up their volumes.
      • Every rebuild left another set behind.
    • Delete the apps that own disks before terraform destroy, and make “no available volumes” part of every teardown check.
======
aws eks list-clusters --region us-east-1
======
aws ec2 describe-volumes --region us-east-1 --filters Name=status,Values=available --query "Volumes[].VolumeId"
======

Part A: cluster off (no AWS costs)

======
mkdir $HOME\Documents\cluster-app\fall-00
======
mkdir $HOME\Documents\cluster-app\photos\fall
======
cd $HOME\Documents\cluster-app
======

Take Fall Photos

  • Take two photos per chore, one pass and one fail.
  • Name them like 01-rake-leaves-pass.jpg and save them in photos\fall\.
  • Never put these in a repo, because they can contain your home’s location.
#ChorePass photoFail photo
01Rake leavesRake in a leaf pileBasketball
02Bag leavesFull leaf bagsEmpty driveway
03Trim trees and bushesShears and cut branchesCoffee mug
04Put away hosesCoiled hose in the shedBike
05Take down the gazeboFolded gazeboTV
06Store deck furnitureStacked, covered chairsDog
07Cover the grillCovered grillCar
08Cover the AC unitCovered AC unitFront door
09Clear flower bedsCleared bed and trowelShoes
10Last mowMower on short grassFootball

Set the Budget Alert

  • AWS Console
    • Budgets → Create budget → Use a template → Monthly cost budget.
      • Name it say-when-lab, set the amount to 40, enter your email, and create it.
    • Open the Budget
      • Edit → Alerts. Set Actual 50%, Actual 80% and Forecasted 100%.
    • Take a screenshot & save it in fall-00\.

Part B: Turn Cluster on & Check your HCP Terraform Workspace

  • Edit variables for version, AZs, & subnets.

Apply Infra Patch

  • You should see 3 new commits.
======
git checkout main
======
git pull
======
git checkout -b fix/readme-and-cleanup
======
git am $HOME\Downloads\say_when_infra.patch
======
git log --oneline -4
======
git push -u origin fix/readme-and-cleanup
======

Merge & Apply

  • Can either do in CLI or UI for Github & HCP connector
======
git checkout main
======
git pull
======
terraform login
======
terraform init
======
terraform apply
======

Apply Gitops Patch

  • Open the PR, merge it, & ArgoCD applies it.
======
git checkout main
======
git pull
======
git checkout -b fix/readme-and-cleanup
======
git am $HOME\Downloads\say_when_git_ops.patch
======
git push -u origin fix/readme-and-cleanup
======

Check Cluster & Edit Waves

  • Use CLI or K9s
======
aws eks update-kubeconfig --region us-east-1 --name say-when-east
======
aws eks describe-cluster --name say-when-east --query cluster.version
======
kubectl get nodes
======
kubectl get applications -n argocd -w
======
  • Files changed: 7 files, 2 lines added to each, nothing removed
    • The 5 AppProjects in applications/lab/projects/:
      • lab-ai-ops.yaml
      • lab-observability.yaml
      • lab-platform.yaml
      • lab-security.yaml
      • lab-workloads.yaml
  • This makes the projects the very first thing Argo CD creates, before any app that belongs to them. It prevents the deadlock you just hit.
    • The 2 cluster-foundation files in applications/lab/platform/:
      • gp3-storageclass.yaml
      • monitoring-namespace.yaml
metadata:
name: lab-platform
namespace: argocd
annotations: # ← added
argocd.argoproj.io/sync-wave: "-10" # ← added
labels:
...
  • This makes the monitoring namespace and the gp3 storage class exist right after the projects and before any app that needs them, like Prometheus and its disks.
metadata:
name: monitoring
annotations: # ← added
argocd.argoproj.io/sync-wave: "-5" # ← added

Quick fix: status.terminatingReplicas: field not declared in schema

  • Problem
    • The repo server keeps crashing (the “EOF” errors). Kubernetes 1.35 added a new field to Deployment status, and Argo CD (chart 7.1.0) is too old to know about it, so its diff engine fails on any app with a Deployment.
      • Example: argo-rollouts, envoy-gateway, k8sgpt-operator, karpenter.
  • Fix:
    • Raise argocd repo server memory to 1gi cuz was OOMKilled for
      • Memory limit goes from 384Mi to 1Gi
        • Rendering ~30 charts on a fresh cluster runs it out of memory, so it restarts, and every chart it was working on fails.
      • CPU goes from 300m to 1 core
        • Helps render finishes faster instead of being throttled
    • That’s blocking the monitoring stack (Prometheus, Loki, Tempo, Thanos) from being installed at all, which you need today.

The repo server keeps crashing (the “EOF” errors). It renders every Helm chart, and its memory limit is only 384Mi. Rendering ~30 charts on a fresh cluster runs it out of memory, so it restarts, and every chart it was working on fails. That’s blocking the monitoring stack (Prometheus, Loki, Tempo, Thanos) from being installed at all, which you need today.

======
resources:
requests:
cpu: 50m
memory: 160Mi
memory: 256Mi
limits:
cpu: 300m
memory: 384Mi
cpu: "1"
memory: 1Gi
======
kubectl -n argocd rollout status deployment argocd-repo-server
kubectl -n argocd get deploy argocd-repo-server -o jsonpath='{.spec.template.spec.containers[0].resources}{"\n"}'
kubectl -n argocd get applications -o name | xargs -I{} kubectl -n argocd annotate {} argocd.argoproj.io/refresh=hard --overwrite
======
kubectl -n argocd get pods # repo-server restarts stay at 0
kubectl get applications -n argocd
kubectl get pods -n monitoring

Create Grafana Secret

======
kubectl -n monitoring get secrets | grep -i grafana
kube-prometheus-stack-grafana Opaque 3 83m
======
kubectl -n monitoring get secret kube-prometheus-stack-grafana -o jsonpath='{.data.admin-password}' | base64 -d; echo
======

Rebuild the Popeye image

  • If Docker buildx, Syft or Cosign is missing, skip this step and use the fallback at the end of h.
======
cd /mnt/c/Users/<you>/Documents/say_when_infra
======
./scripts/build-secure-popeye.sh
======
head -30 "$F/popeye-baseline.txt"
popeye:
report_time: "2026-11-16T19:37:30Z"
score: 80
grade: B
sections:
- linter: cluster
gvr: cluster
tally:
ok: 1
info: 0
warning: 0
error: 0
score: 100
issues:
Version:
- group: __root__
gvr: cluster
level: 0
message: '[POP-406] K8s version OK'
- linter: clusterroles
gvr: rbac.authorization.k8s.io/v1/clusterroles
tally:
ok: 138
info: 2
warning: 0
error: 0
score: 100
issues:
external-secrets-servicebindings:
- group: __root__
======
grep -m3 -iE '^\s*(score|grade)' "$F/popeye-baseline.txt"
score: 80
grade: B
score: 100
======
grep -c 'POP-' "$F/popeye-baseline.txt"
586
======
grep -oE 'POP-[0-9]+' "$F/popeye-baseline.txt" | sort | uniq -c | sort -rn | head -8
118 POP-1204
59 POP-400
48 POP-106
46 POP-206
42 POP-306
36 POP-102
33 POP-301
32 POP-1109
======

Run & Save the Baselines

  • Write down the PASS / FAIL / WARN totals and the Popeye score, which is near the end of popeye-baseline.txt.
======
cd $HOME\Documents\cluster-app\fall-00
======
kubectl create job kube-bench-baseline --from=cronjob/kube-bench -n kube-bench
======
kubectl create job popeye-baseline --from=cronjob/popeye -n popeye
======
kubectl wait --for=condition=complete job/kube-bench-baseline -n kube-bench --timeout=10m
======
kubectl wait --for=condition=complete job/popeye-baseline -n popeye --timeout=10m
======
kubectl logs job/kube-bench-baseline -n kube-bench | Out-File -Encoding utf8 kube-bench-baseline.txt
======
kubectl logs job/popeye-baseline -n popeye | Out-File -Encoding utf8 popeye-baseline.txt
======
grep -m3 -iE '^\s*(score|grade)' "$F/popeye-baseline.txt"
grep -c 'POP-' "$F/popeye-baseline.txt"
grep -oE 'POP-[0-9]+' "$F/popeye-baseline.txt" | sort | uniq -c | sort -rn | head -8
score: 80
grade: B
score: 100
586
118 POP-1204
59 POP-400
48 POP-106
46 POP-206
42 POP-306
36 POP-102
33 POP-301
32 POP-1109
======

SCREEEEEEEEEEEEENSHOTZZZZZ for Proof

======
ubectl get applications -n argocd
NAME SYNC STATUS HEALTH STATUS
argo-rollouts Synced Healthy
bootstrap-root Synced Healthy
envoy-gateway Synced Healthy
envoy-gateway-config Synced Healthy
external-secrets Synced Healthy
external-secrets-config Synced Healthy
falco Synced Healthy
gateway-demo Synced Healthy
k8sgpt Synced Healthy
k8sgpt-operator Synced Healthy
karpenter Synced Healthy
karpenter-config Synced Healthy
karpenter-crd Synced Healthy
kube-bench Synced Healthy
kube-prometheus-stack Synced Healthy
kyverno Synced Healthy
kyverno-policies Synced Healthy
loki Synced Healthy
metrics-server Synced Healthy
monitoring-root Synced Healthy
network-policies Synced Healthy
popeye Synced Healthy
rollouts-bluegreen Synced Healthy
rollouts-demo Synced Healthy
secret-demo Synced Healthy
tempo Synced Healthy
thanos Synced Healthy
trivy-operator Synced Healthy
======
kubectl get secret argocd-initial-admin-secret -n argocd -o jsonpath="{.data.password}"
| base64 -d
======
kubectl -n argocd port-forward svc/argocd-server 8080:80
======
======
kubectl -n monitoring port-forward svc/kube-prometheus-stack-grafana 3000:80
======
kubectl get nodes
======
  • Save everything in fall-00\.

Part C: Turn down for what? The Cluster….

  • Either in CLI or HCP Terraform (Settings → Destruction and Deletion → Queue destroy plan),
======
cd $HOME\Documents\say_when_infra
======
terraform apply -destroy
======

DESTROYYYYYYYYYYYYYYYYYYYYY

  • Use whatever commands you prefer, these below can be used as a starter
  • If the destroy gets stuck on the Thanos S3 bucket, empty the bucket in the S3 console and run the destroy again.
======
# Check 1: no EKS cluster. Should print "clusters": []
aws eks list-clusters --region us-east-1
======
# Check 2: no Karpenter nodes left running. Should print []
aws ec2 describe-instances --region us-east-1 \
--filters Name=tag-key,Values=karpenter.sh/nodepool Name=instance-state-name,Values=running,pending \
--query "Reservations[].Instances[].InstanceId"
======
# Check 3: delete leftover monitoring disks...
for v in $(aws ec2 describe-volumes --region us-east-1 --filters Name=status,Values=available Name=tag:kubernetes.io/created-for/pvc/namespace,Values=monitoring --query "Volumes[].VolumeId" --output text); do
echo "Deleting $v"; aws ec2 delete-volume --region us-east-1 --volume-id "$v"
done
======
# ...then confirm no loose disks remain. Should print []
aws ec2 describe-volumes --region us-east-1 --filters Name=status,Values=available --query "Volumes[].VolumeId"
======

Leave a Reply