EKS Cluster: Part 5 – S3 & Thanos

Series of blog posts show progress of updating/adding to EKS Cluster, this post covers adding Thanos to S3 for further telemetry. See below for past posts:

• EKS Cluster: Part 1 – AWS Infrastructure
• EKS Cluster: Part 2 – Orchestration & Monitoring

Summary of Thanos & S3 – allow-Thanos-egress:

Thanos Store Gateway / Prometheus Sidecar
↓
Kubernetes ServiceAccount
↓
IRSA IAM Role
↓
IAM S3 Policy
↓
S3 Gateway VPC Endpoint
↓
Endpoint Policy
↓
S3 Bucket Policy
↓
thanos-infinity-37

• S3 is outside the cluster & K8 Network Policies dont understand that s3.amazonaws.com as a domain name – so I wont pin an S3 IP range.
• Even thought pod IPs are ephemeral & unpredictable, ipBlock for egress that connect to service IPs that get re-written to cluster-external IPs may or may not be subject to ipBlock.
  • Already have IAM/IRSA, so lets tighten down endpoint & bucket policy AWS network traffic.

• Refer to K8s docx here:
  • https://kubernetes.io/docs/concepts/services-networking/network-policies/?#behavior-of-to-and-from-selectors
• Refer to AWS docx here:
  • https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html

s3-endpoint.tf:

• Instead on whitelisting changing public S3, we’ll constrain S3 access by:
  • IAM Policy — > S3 Bucket Policy — > VPC Endpoint Policy
  • Thanos Pod — > S3 Gateway VPC Endpoint — > S3 Bucket
• See github repo code:
  • https://github.com/earpjennings37/say_when_infra/blob/main/s3-endpoint.tf
• Refer to Terraform docx here:
  • https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/vpc_endpoint

Thanos Pod
↓
EKS node/VPC routing
↓
S3 Gateway Endpoint
↓
S3

Thanos-iam.tf & Thanos-policy.tf:

• IRSA role control who calls S3 & IAM Policy handles ID & authorization. IRSA places temporary AWS credz to k8s service account…. so now Thanos does not need node-role permissions.
• Refer to AWS docx here:
  • https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html?

Kubernetes ServiceAccount
↓
IRSA IAM Role
↓
IAM Policy
↓
Allowed S3 actions

data.tf:

• Refer to AWS docx here:
  • https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-access.html?
  • https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies-vpc-endpoint.html?utm_source=chatgpt.com
• Refer to Terraform docx here:
  • https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document
• A VPC endpoint policy controls what can pass through the S3 endpoints & can attach to an endpoint to restrict what principals can use & access.

1. git add applications/network-policies.yaml apps/network-policies/
2. kubectl annotate application monitoring-root -n argocd \
  argocd.argoproj.io/refresh=hard --overwrite
application.argoproj.io/monitoring-root annotated
3. kubectl get applications -n argocd -w
NAME                    SYNC STATUS   HEALTH STATUS
bootstrap-root          Synced        Healthy
kube-prometheus-stack   Synced        Healthy
loki                    Synced        Healthy
metrics-server          Synced        Healthy
monitoring-root         Synced        Healthy
network-policies        Synced        Healthy
tempo                   Synced        Healthy
thanos                  Synced        Healthy
4. kubectl get networkpolicy -n monitoring
NAME                          POD-SELECTOR                                                                                               AGE
allow-grafana-to-loki         app.kubernetes.io/component=gateway,app.kubernetes.io/name=loki                                            11m
allow-grafana-to-prometheus   app.kubernetes.io/name=prometheus                                                                          11m
allow-grafana-to-tempo        app.kubernetes.io/name=tempo                                                                               11m
thanos-query                  app.kubernetes.io/component=query,app.kubernetes.io/instance=thanos,app.kubernetes.io/name=thanos          179m
thanos-storegateway           app.kubernetes.io/component=storegateway,app.kubernetes.io/instance=thanos,app.kubernetes.io/name=thanos   179m

S3 Gateway Endpoint:

• Only permits “thanos-infinity-37” & other io are blocked
  • Deleted data.policy for s3_endpoint only to S3 bucket
• Then manage the endpoint explicitly

kubectl delete pod kube-prometheus-stack-kube-state-metrics-84bd847f67-clbfw \
  -n monitoring
kubectl delete pod kube-prometheus-stack-prometheus-node-exporter-n5p28 \
  -n monitoring

• Still retain the flow of:

Thanos Pod
↓
IRSA
↓
thanos-irsa-role
↓
thanos S3 IAM policy
↓
S3 Gateway Endpoint
↓
thanos-infinity-37

EKS node
↓
registry.k8s.io / quay.io
↓
registry S3 backing buckets

• AWS Resources
  • https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html?