EKS Cluster: Part 12.33 – Karpenter aka AWS & Terraform (infra)

Karpenter: Dynamically launches compute based on pending workload needs rather than relying only on fixed managed node groups.

Series of 3 blog posts show progress of updating/adding to EKS Cluster, this post covers adding Karpenter. This in particular covers the infrastructure AWS & Terraform portion of Karpenter.

infra/Terraform repository

karpenter.tf
├── controller IAM role
├── controller IAM policy
├── Pod Identity association
├── Karpenter EC2 node role
├── EKS node access entry
├── SQS interruption queue
├── EventBridge rules
├── queue policy
└── subnet/security-group discovery tags

vpc.tf

• Karpenter can discover subnets through this tag and selects the subnet with the most available IP addresses in the required Availability Zone
  • https://karpenter.sh/docs/concepts/nodeclasses/

==================
   # Allows the Karpenter EC2NodeClass to discover both east subnets.
  public_subnet_tags = {
    "karpenter.sh/discovery" = local.eks_east_name
  }
==================

eks.tf

• Only the node security group should receive the discovery tag. Karpenter attaches every security group matching its selector, so tagging multiple EKS security groups could attach more groups than intended.

==================
    node_security_group_tags = {
    "karpenter.sh/discovery" = local.eks_east_name
  }
==================

karpenter.tf

• The module internally creates interruption handling & sub-module contains resources similar to below. It creates the SQS queue, queue policy, EventBridge rules, and EventBridge targets when native interruption handling is enabled.
  • AWS interruption event
  • EventBridge rule
  • SQS queue: Karpenter-say-when-east
  • Karpenter controller
  • Drain/replace affected node

==================
resource "aws_sqs_queue" "this" {
  # Interruption queue
}
resource "aws_sqs_queue_policy" "this" {
  # Permits EventBridge to send messages
}
resource "aws_cloudwatch_event_rule" "this" {
  # Spot interruption, rebalance and EC2 lifecycle rules
}
==================

• My TF expresses the intent w/one flag, while the module contains AWS resources
  • https://terraform-aws-modules-terraform-aws-eks.mintlify.app/modules/karpenter

==================
enable_spot_termination = true
==================

• Instance profile is the bridge between karpenter ec2 & my existing eks worker-node iam role. IAM role contains permissions, but ec2 cant use role directly, so ec2 receives the role from the iam instance profile.
  • https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/20.37.2/submodules/karpenter

karpenter.tf

• Karpenter introduced this permission for its instance-profile garbage-collection controller. Resource = "*" is expected for this IAM list operation

======================
resource "aws_iam_role_policy" "karpenter_instance_profile_gc" {
  provider = aws.east
  name = "KarpenterInstanceProfileGarbageCollection"
  role = module.karpenter_east.iam_role_name
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Sid      = "AllowListingInstanceProfiles"
        Effect   = "Allow"
        Action   = "iam:ListInstanceProfiles"
        Resource = "*"
      }
    ]
  })
}
======================

Only part 1 remember, check the other 2 parts to the karpenter series..

• EKS Cluster: Part 12.99 – Karpenter aka Check your Work (Grand Finale)
• EKS Cluster: Part 12.66 – Karpenter aka Helm-Charts (gitops)