Say When..

Series of blog posts show progress of updating/adding to EKS Cluster, this post covers adding Karpenter. See other blog-posts below:

Karpenter: Dynamically launches compute based on pending workload needs rather than relying only on fixed managed node groups.

Karpenter uses NodePool and AWS-specific EC2NodeClass resources & addresses:
- Spot interruption handling
- Dynamic node provisioning
- Better instance-type selection
- Scaling for unschedulable pods
- Availability Zone-aware scheduling
- Removing empty or underutilized nodes

GitOps repository

Below are steps to follow
- GitOps files under application & apps.
- Update the lab-platform AppProject permissions.
- Install the controller first.
- Confirm controller health.
- Add the EC2NodeClass and ARM64 Spot NodePool.
- Deploy a disposable workload that cannot fit on current nodes and watch Karpenter create a NodeClaim and EC2 instance.
We’ll deploy the Karpenter CRDs and controller first. We will add the EC2NodeClass and NodePool only after the controller is healthy, which keeps troubleshooting much easier.

applications/lab/platform/karpenter.yaml:

Purpose
- Installs the Karpenter controller through its official OCI Helm chart.
- Uses apps/karpenter/values.yaml.
- Sets skipCrds: true because CRDs are managed separately.
- Creates the karpenter namespace.
Sources:

kubectl get applications -n argocd -w

apps/karpenter/values.yaml

Purpose
- Supplies the cluster name and interruption queue.
- Configures the karpenter ServiceAccount used by Pod Identity.
- Sets controller resources and replica configuration.
- Does not store AWS credentials.
Sources:

terraform destroy

applications/lab/projects/karpenter-crd.yaml

Purpose
- Creates the Argo CD Application that installs Karpenter CRDs.
- Manages EC2NodeClass, NodePool, and NodeClaim definitions separately from the controller.
- Runs before the controller using a negative sync wave.
Sources:
- Karpenter CRD chart in ECR Public
- Karpenter CRD chart source
  - When gitops manage karpenter, use crd & set skipcrds on the controller chart
- Karpenter CRD upgrade guidance
  - Karpenter specifically recommends the independent CRD chart for managing CRD lifecycle
- Argo CD declarative setup
- Argo CD sync waves

applications/lab/platform/karpenter-config.yaml

Purpose
- Creates a separate Argo CD Application for the EC2NodeClass and NodePool.
- Keeps controller installation separate from capacity policy.
- Applies the configuration after the CRDs and controller.
Sources:

			
======================
# Argo CD manages the Karpenter EC2 configuration.
    - group: karpenter.k8s.aws
      kind: EC2NodeClass
# Argo CD manages the Karpenter capacity and scheduling policy.
    - group: karpenter.sh
      kind: NodePool
======================

		

apps/karpenter-config/ec2nodeclass.yaml

Purpose: this uses the stable instance profile created by Terraform
- AL2023 AMI family
- Pinned EKS AMI release
- ARM64 compatibility
- Stable IAM instance profile
- Subnet discovery
- Security-group discovery
- Public IPv4 assignment
- IMDSv2 enforcement
- Encrypted gp3 boot volume
- EC2 tags
Sources:

KarpenterNodeInstanceProfile

ec2nodeclass

nodepool

apps/karpenter-config/nodepool.yaml

Purpose: Defines the capacity and scheduling policy:
- ARM64 Linux nodes
- Spot capacity
- Allowed EC2 instance categories
- Allowed CPU sizes
- Instance-generation restriction
- Maximum CPU and memory
- Consolidation behavior
- Disruption budget
Sources:

Quick Tip: GiB Sizing

This is a safety ceiling — not a reservation. It does not mean AWS immediately provisions 8 CPUs or 32 GiB.
The constraints also require:
- ARM64
- Spot
- Linux
- 2- or 4-vCPU instances
- Newer instance generations
Karpenter can still compare several eligible instance families instead of relying on only one EC2 type.

			
======================
limits:
  cpu: "8"
  memory: 32Gi
======================

		

applications/lab/projects/lab-platform.yaml

Update destinations to the spec

			
======================
spec:
  description: Core Kubernetes platform services for the lab cluster
  sourceRepos:
    - https://github.com/earpjennings37/say_when_git_ops.git
    - https://kubernetes-sigs.github.io/metrics-server/
    - public.ecr.aws/karpenter
  destinations:
    - namespace: kube-system
      server: https://kubernetes.default.svc
    - namespace: karpenter
      server: https://kubernetes.default.svc
======================

		

Update clusterrole & role for whitelisting

			
======================
spec:
  description: Core Kubernetes platform services for the lab cluster
  sourceRepos:
    - https://github.com/earpjennings37/say_when_git_ops.git
    - https://kubernetes-sigs.github.io/metrics-server/
    - public.ecr.aws/karpenter
  destinations:
    - namespace: kube-system
      server: https://kubernetes.default.svc
    - namespace: karpenter
      server: https://kubernetes.default.svc
======================
spec:
clusterResourceWhitelist:
    # Metrics Server API registration.
    - group: apiregistration.k8s.io
      kind: APIService
    # Karpenter CRDs.
    - group: apiextensions.k8s.io
      kind: CustomResourceDefinition
    # Karpenter and Metrics Server cluster RBAC.
    - group: rbac.authorization.k8s.io
      kind: ClusterRole
    - group: rbac.authorization.k8s.io
      kind: ClusterRoleBinding
    # Required by CreateNamespace=true.
    - group: ""
      kind: Namespace
======================
namespaceResourceWhitelist:
    - group: rbac.authorization.k8s.io
      kind: Role
    - group: rbac.authorization.k8s.io
      kind: RoleBinding
    - group: policy
      kind: PodDisruptionBudget
======================

		

Only part 1 remember, check the other 2 parts to the karpenter series..