Say When..

Series of blog posts show progress of updating/adding to EKS Cluster, this post covers adding connecting Grafana w/Prometheus, Loki, Tempo, & Thanos for further telemetry some {Deny} & Allow Egress Network Policies. Below is the flow of what was accomplished:

			
default deny ingress
→ explicit ingress allows
→ enable VPC CNI NetworkPolicy enforcement
→ verify PolicyEndpoint CRs
→ add namespace-wide egress isolation
→ explicitly allow DNS/API/Prometheus/Loki/Thanos traffic
→ validate Prometheus + Thanos still work

		

See below for past posts:

Enable Default-Deny Ingress Only:

default-deny.yaml:

https://github.com/earpjennings37/say_when_git_ops/blob/main/apps/network-policies/default-deny.yaml
Documentation Referenced:
- K8s is allow-list based, if a pod matches policy then only explicit traffic can pass
  - https://kubernetes.io/docs/concepts/services-networking/network-policies/?#default-deny-all-ingress-and-all-egress-traffic

Either check K9s or run this command to validate:

			
1. kubectl get networkpolicy -n monitoring
kubectl get networkpolicy -n monitoring
NAME                                     POD-SELECTOR                                                                                               AGE
allow-grafana-to-loki                    app.kubernetes.io/component=gateway,app.kubernetes.io/name=loki                                            82m
allow-grafana-to-prometheus              app.kubernetes.io/name=prometheus                                                                          82m
allow-grafana-to-tempo                   app.kubernetes.io/name=tempo                                                                               82m
allow-prometheus-to-kube-state-metrics   app.kubernetes.io/name=kube-state-metrics                                                                  82m
allow-prometheus-to-node-exporter        app.kubernetes.io/name=prometheus-node-exporter                                                            82m
allow-thanos-query-to-sidecar            app.kubernetes.io/name=prometheus                                                                          82m
allow-thanos-query-to-storegateway       app.kubernetes.io/component=storegateway,app.kubernetes.io/name=thanos                                     82m
default-deny-ingress                     <none>                                                                                                     2m36s
thanos-query                             app.kubernetes.io/component=query,app.kubernetes.io/instance=thanos,app.kubernetes.io/name=thanos          82m
thanos-storegateway                      app.kubernetes.io/component=storegateway,app.kubernetes.io/instance=thanos,app.kubernetes.io/name=thanos   82m

		

Test w/Port-Forward Prometheus

			
1. kubectl port-forward -n monitoring \
  svc/kube-prometheus-stack-prometheus 9090:9090
2. http://localhost:9090/targets

Test w/Port-Forward Thanos

			
kubectl port-forward -n monitoring svc/thanos-query 9091:9090
http://localhost:9091

6 Allow-Egress:

For all the yaml below, you can either run CLI commands to get svc & endpoints or go to K9s & search

apps/network-policies/
├── allow-egress-dns.yaml
├── allow-egress-prometheus.yaml
├── allow-egress-thanos.yaml
├── allow-egress-grafana.yaml
└── allow-egress-loki.yaml

Allow DNS
Allow Prometheus required egress
Allow Thanos → S3 / HTTPS
Allow Loki internal egress where needed
Allow Tempo egress where needed
Test
Enable default-deny-egress last

allow-egress-dns:

Documentation Source:
- https://kubernetes.io/docs/concepts/services-networking/network-policies/?#allow-all-egress-traffic
Selects all monitoring pods & permits DNS to CoreDNS, empty podselector matches pod in namespace

allow-egress-grafana:

Documentation Source:
- https://kubernetes.io/docs/concepts/services-networking/network-policies/?#behavior-of-to-and-from-selectors
podSelector a single to/from entry selects particular pods in particular namespace.
- So added 3 to the egress podSelector
  - In code below you will notice Loki has to labels,

 egress:
    # Grafana -> Prometheus
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: prometheus
      ports:
        - protocol: TCP
          port: 9090

    # Grafana -> Loki gateway
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: loki
              app.kubernetes.io/component: gateway
      ports:
        - protocol: TCP
          port: 8080

    # Grafana -> Tempo
    - to:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: tempo
      ports:
        - protocol: TCP
          port: 3100

allow-egress-thanos:

Documentation Leveraged:
- https://thanos.io/v0.15/components/query.md/?
- https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html
So the flow is:
- Thanos Query -> Store Gateway
- Thanos Query -> Prometheus Thanos sidecar
- Store Gateway -> S3 over HTTPS
2 policies in one file for query & store gateway separate in selecting pods & egress, K8s egress rules allow traffic matching both destination & port criteria.
Under storage gateway, needed object-storage serves history blocks & S3 Gateway Endpoints handles AWS routing path for S3 traffic
- 443 is for HTTPs, not just S3 cuz K8s NetworkPolicy is not just “s3.amazonaws.com” – hence your IAM/IRSA come into play for security controls.

  egress:
    # Store Gateway -> S3 over HTTPS
    - ports:
        - protocol: TCP
          port: 443

allow-egress-prometheus.yaml:

Thanos sidecar is in prometheus pod, pod needs outbound to s3 for block uploads, network policy is set at pod level – not individual containers.
- https://kubernetes.io/docs/concepts/services-networking/network-policies/?#pod-lifecycle
Add under eks.tf brief code of enablenetworkpolicy=true to ensure agent is running in VPC CNI
- https://repost.aws/articles/ARq5x53–OSPihwF3Y8C50wg/troubleshooting-kubernetes-network-policies-enforcing-policies-in-amazon-eks?
Feel free to port-forward (steps above) to double-check, but you can run CLI or K9s to see your policy endpoints took.

kubectl get policyendpoints.networking.k8s.aws -A

Seems long cuz prometheus connects to all, so egress is lengthy –
- Prometheus -> kube-state-metrics
- Prometheus -> node-exporter
- Prometheus -> Prometheus Operator
- Prometheus -> Grafana metrics
- Prometheus -> itself / sidecar metrics
- Prometheus -> CoreDNS metrics
- Prometheus -> node/control-plane targets inside VPC
  - Broad cuz to nodes & control plane in VPC
- Prometheus Thanos sidecar -> S3/HTTPS
Again get all these easiest in K9s : endpoints & scroll to view
- https://kubernetes.io/docs/concepts/services-networking/network-policies/#networkpolicy-resource

allow-prometheus-internal-scrapes.yaml

Cant forget about scraping ya’ll..so create 3 policies here
- allow-prometheus-to-grafana
  - prometheus
- allow-prometheus-to-operator
  - kube-prometheus-stack-prometheus-operator
- allow-prometheus-self-scrape
  - prometheus

allow-egress-loki:

Created 2 policies called, basically 7946 is a “memberlist gossip” & gateway forwards request to Loki on 3100.
- allow-egress-loki-gateway
  - Loki gateway -> Loki single-binary backend
- allow-egress-loki-memberlist
  - Loki memberlist/gossip
    - TCP/UDP
- https://grafana.com/docs/loki/latest/configure/examples/configuration-examples/?
- https://grafana.com/docs/loki/latest/operations/troubleshooting/troubleshoot-operations/?

allow-loki-internal:

Similar as the other loki file,
- allow-loki-gateway-to-backend
- ingress
  - gateway –> 3100
  - loki — > 7946

allow-egress-kubernetes-api:

allow-kube-state-metrics-egress-api
- kube-state-metrics
allow-prometheus-operator-egress-api
- kube-prometheus-stack-prometheus-operator
allow-grafana-egress-api
- grafana
Add grafana so sidecars are watched further by k8s config maps/secrets
2 IPs that are egresss rules – allow both k8s service IP & VPC API Path
- https://kubernetes.io/docs/concepts/services-networking/network-policies/?behavior-of-to-and-from-selectors

			
egress:
    - to:
        - ipBlock:
            cidr: 172.20.0.1/32
        - ipBlock:
            cidr: 10.0.0.0/16

		

Final Tests:

Again use the CLI commands below or use ArgoCD Ui and K9s
- ArgoCD Ui
  - Refresh or Sync
- K9s
  - Check logs, deployment, svc, endpoints, network policies, etc

			
kubectl annotate application network-policies -n argocd \
  argocd.argoproj.io/refresh=hard --overwrite
kubectl get application network-policies -n argocd -w
kubectl get networkpolicy -n monitoring
kubectl get policyendpoints.networking.k8s.aws -n monitoring

		

Prometheus:

Thanos:

Check logs, deployment, svc, endpoints, network policies in either CLI or K9s

			
1. kubectl get deployment thanos-query -n monitoring \
  -o jsonpath='{.spec.template.spec.containers[0].args}' | jq
kubectl get deployment thanos-query -n monitoring \
  -o jsonpath='{.spec.template.spec.containers[0].args}' | jq
[
  "query",
  "--log.level=info",
  "--log.format=logfmt",
  "--grpc-address=0.0.0.0:10901",
  "--http-address=0.0.0.0:10902",
  "--query.replica-label=replica",
  "--endpoint=dnssrv+_grpc._tcp.thanos-storegateway.monitoring.svc.cluster.local",
  "--endpoint=dnssrv+_grpc._tcp.thanos-storegateway.monitoring.svc.cluster.local",
  "--endpoint=prometheus-operated.monitoring.svc.cluster.local:10901",
  "--alert.query-url=http://thanos-query.monitoring.svc.cluster.local:9090"
]

		

Annnnnnnnnnnnnnnnnnnnnnnnnnnnnnd that’s a wrap: