EKS Cluster: Part 4 – TF Bootstrapin, .sh Scriptin, & Telemetrin’

by EarpMoonWater, posted in aws, terraform

Series of blog posts show progress of updating/adding to EKS Cluster, this post covers adding Terraform bootstrap for enhanced automation, start-up sh script, & connecting Grafana w/Prometheus, Loki, Tempo, & Thanos to S3 for further telemetry. See below for past posts:

Bootstrappin TF w/thuh Ferrrrr’:

Terraform Apply
│
├── VPC
├── EKS
├── EBS CSI
├── IAM / IRSA
├── S3
├── Argo CD
│
└── bootstrap-root
│
▼
GitHub
│
▼
monitoring-root
│
├── kube-prometheus-stack
├── Loki
├── Tempo
├── Thanos
└── metrics-server

Providers.tf:

terraform-aws-modules/eks = module.eks_east.cluster_name
↓
creates EKS cluster & aws_eks_cluster_auth data source
↓
exports cluster_certificate_authority_data & temporary authentication token
↓
kubectl provider uses it

Bootstrap.tf

EKS

↓

Argo Helm installation

↓

bootstrap-root

Lil Scripty Script to make this Reproducible:

  • Oh & did a simple post-build verification script too, basically runs some commands to ensure i can get my pods & stuff.
  • Before Makeover
    • Terraform Apply
    • aws eks update-kubeconfig….
    • kubectl apply -f bootstrap-root.yaml
    • …wait…
    • Verify
  • After Makeover
    • Terraform Apply
    • awks eks update-kubeconfig..
    • ./scripts/verify-cluster.sh

Verify-cluster.sh:

Prometheus, Sidecar Named Thanos & AWS IRSA S3:

Prometheus
│
├── local 24h metrics
│
└── Thanos sidecar
↓
S3
↓
Thanos Store Gateway
↓
Thanos Query

Objstore.yaml:

Thanos-IAM.tf:

Values.tf:

Application-bootstrap.yaml:

  annotations:
    argocd.argoproj.io/sync-wave: "-1"

Visualzz of what had happen was:

TF Commandzz:

  • terraform init -upgrade
  • terraform fmt
  • terraform validate
  • terraform plan
  • terraform apply

Thanos Storage Gateway Issue:

  • Oddly ran on an AZ that was not available 
kubectl delete pvc data-thanos-storegateway-0 -n monitoring

Prometheus Application OutOfSync/Missing:

  • Argo was stuck on kube-prometheus-stack-admission & are no admissions jobs hard refresh, so Argo syncs kept getting stuck on the admission hook
# ============================
# PROMETHEUS OPERATOR (CHEAP MODE)
# ============================
prometheusOperator:
  resources:
    requests:
      cpu: 20m
      memory: 50Mi
    limits:
      cpu: 100m
      memory: 100Mi
  prometheusOperator:
  admissionWebhooks:
    patch:
      enabled: false
  • Pod-count capacity on the nodes, not CPU/memory. Daemon-set wants 1 node-exporter pod per node, have 3 ready nodes, but only one exporter is running & other two are pending. Aka —- too many pods.
  • AWS Docx Reference
  • Cuz i have spot, i have different sizes & smaller instance support fewer pods network interfaces/IPs & EKS pod allocatable limit.
node_instance_types = [
  "t4g.small",
  "t4g.medium",
  "t4g.large"
]
t4g.medium → 10 pods
t4g.small  → 9 pods
t4g.small  → 8 pods
  1. Check CNI version/config
  2. Enable prefix delegation
  3. Replace/update nodes
  4. Verify MAXPODS increases
  5. Let node-exporter schedule
  6. Verify Argo → Synced / Healthy

ebs-csi.tf:

resource "aws_eks_addon" "vpc_cni_east" {
  cluster_name = module.eks_east.cluster_name
  addon_name   = "vpc-cni"
  most_recent  = true
  configuration_values = jsonencode({
    env = {
      ENABLE_PREFIX_DELEGATION = "true"
      WARM_PREFIX_TARGET       = "1"
    }
  })
}

vpc-cni.tf

EKS is giving you the CNI as part of the cluster/node setup, but Terraform has no dedicated block you can configure yet.

aws eks list-nodegroups \
  --cluster-name say-when-east \
  --region us-east-1
  aws eks update-nodegroup-version \
  --cluster-name say-when-east \
  --nodegroup-name <NODEGROUP_NAME> \
  --region us-east-1
  aws eks update-nodegroup-version \
  --cluster-name say-when-east \
  --nodegroup-name default-20260817193826579400000012 \
  --region us-east-1

Temporary Nodegroup added, cuz did not want to terraform destroy.. cuz it takes too long:

 kubectl get nodes \
  -L eks.amazonaws.com/nodegroup \
  -o custom-columns='NAME:.metadata.name,NODEGROUP:.metadata.labels.eks\.amazonaws\.com/nodegroup,TYPE:.metadata.labels.node\.kubernetes\.io/instance-type,MAXPODS:.status.allocatable.pods'
NAME                         NODEGROUP                            TYPE        MAXPODS
ip-10-0-1-110.ec2.internal   default-20260817193826579400000012   t4g.small   11
ip-10-0-1-164.ec2.internal   default-20260817193826579400000012   t4g.small   11
ip-10-0-1-209.ec2.internal   prefix-20260817212214558700000007    t4g.small   110
ip-10-0-1-217.ec2.internal   default-20260817193826579400000012   t4g.small   11
ip-10-0-1-231.ec2.internal   prefix-20260817212214558700000007    t4g.small   110
ip-10-0-1-242.ec2.internal   prefix-20260817212214558700000007    t4g.small   110
ip-10-0-2-133.ec2.internal   prefix-20260817212214558700000007    t4g.small   110
ip-10-0-2-81.ec2.internal    default-20260817193826579400000012   t4g.small   11

Prometheus Still OutOfSync:

prometheusOperator:
  admissionWebhooks:
    enabled: false
    patch:
      enabled: false
kubectl annotate application kube-prometheus-stack -n argocd \
  argocd.argoproj.io/refresh=hard --overwrite
application.argoproj.io/kube-prometheus-stack annotated
Unknown's avatar

Published by EarpMoonWater

I like to write stuff about stuff I'm doing.

Leave a comment