helmcharts – Say When..

Series of blog posts show progress of updating/adding to EKS Cluster, this post covers adding a few K8 Security Tooling; Trivy & Kubescape.

Blog is broken up into:

Trivy on EKS
- Phase 1 Trivy – Local Scans
- Phase 2 Trivy – Live EKS Scans
- Phase 3 Trivy – Continuous Scanning
Kubescape
- kubectl get vulnerabilityreports -A
- kubectl get rbacassessmentreports -A
- kubectl get vulnerabilityreport

See below for past posts:

Trivy on EKS, to scan stuff, cuz we like stuff:

Can be integrated before Terraform plan after terraform init during terraform fmt/validate stage
Scan container images
Scan Kubernetes manifests
Scan Terraform/IaC
Show CVEs, misconfigurations, secrets
Best first security-tool post because it is straightforward and immediately useful

Part 1 Trivy – local scans:

Started w/a local scanner against repos I already have to get immediate value cuz Trivy can look for misconfigurations, specific severity findings, & support HCL, & Terraform plan JSON files snapshots. Examples found maybe include;
- k8s manifests
- Helm-related configration
- Dockerfiles if present
- other IaC/config files
- securityContext
- privileged containers
- runAsNonRoot
- resource requests/limits
- capabilities
- read-only filesystem
- RBAC
- NetworkPolicy

Pre-Commands:

			
1. sudo apt-get update
2. sudo apt-get install -y wget gnupg
--------------------------------------------
Trivy Signing key:
3. wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | \
gpg --dearmor | \
4. sudo te /usr/share/keyrings/trivy.gpg > /dev/null
--------------------------------------------
Add Repo:
5.echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb generic main" | \
6. sudo tee /etc/apt/sources.list.d/trivy.list
--------------------------------------------
Install:
7. sudo apt-get update
8. sudo apt-get install -y trivy

		

trivy –version

Version: 0.74.0

trivy config .

for infra —
- I ignored the modules findings, i cant do much there, but the S3 bucket stuff was good to find some basic AWS security steps to take:
  - Bucket has logging disabled
  - Bucket does not have versioning enabled
  - Bucket does not encrypt data with a customer managed key.
Sources
- https://www.trivy.dev/docs/latest/guide/coverage/iac/terraform/
- https://trivy.dev/docs/latest/tutorials/misconfiguration/terraform/

gitops stuff clean!

trivy config –severity HIGH,CRITICAL .

Even though findings arent much (ignoring modules folders), you can tailor just a specific severity – which is clutch to eliminate noise before you even try to terraform plan.

trivy fs –scanners misconfig,secret .

Same as above, nothing new to see here dawg. Using fs tho does combine misconfigurations & secret scanners.
Sources
- https://trivy.dev/docs/dev/docs/references/configuration/cli/trivy_filesystem/

terraform plan -out=tfplan
trivy config tfplan

Same findings as above, but atleast shows example of what Terraform plans to deploy

terraform show -json tfplan > tfplan.json
trivy config tfplan.json

trivy config –tf-exclude-downloaded-modules .

This is what i wanted the whole time, forget the module/3rd-party stuff.

Part 2 Trivy – Live EKS Scans:

Trivy has a k8s scan mode of current kubeconfig context, its expirmental for now – not ideal for prod stuff. But shows delineation of static scan of both repos vs live scan of the cluster in AWS.
Sources
- https://www.trivy.dev/docs/v0.69/guide/references/configuration/cli/trivy_kubernetes/

Pre-Commands:

These are just the Terraform & Trivy CLI commands scanning a live cluster.

			
Rebuild the Cluster:
1. terraform fmt -recursive
2. terraform validate
3. terraform plan
4. terraform apply
--------------------------------------------
Update kubeconfig:
5. aws eks update-kubeconfig \
  --region us-east-1 \
  --name say-when-east
--------------------------------------------
Verify Cluster Access:
6. kubectl get nodes
7. kubectl get pods -A
--------------------------------------------
Wait for Argo Apps:
8. kubectl get applications -n argocd
--------------------------------------------
9. mkdir -p ~/.docker-trivy
10. echo '{}' > ~/.docker-trivy/config.json

		

Basic Trivy K8s Scan:

			
DOCKER_CONFIG=$HOME/.docker-trivy \
trivy k8s --report summary

FYI takes a few minutes so grab a cup of joe right quick
- Workload Assesment
- Infra Assesment
- RBAC Assesment

Scan ns monitoring exclusively:

			
DOCKER_CONFIG=$HOME/.docker-trivy \
trivy k8s \
  --include-namespaces monitoring \
  --report all

A lot quicker, shows all 3 assessments again – but monitor has 0 infra or RBAC assessment findings

High/Criticals Only:

			
DOCKER_CONFIG=$HOME/.docker-trivy \
trivy k8s \
  --include-namespaces monitoring \
  --severity HIGH,CRITICAL \
  --report all

		

Be prepped, its a lot of stuff.

Scan for misconfigurations:

			
DOCKER_CONFIG=$HOME/.docker-trivy \
trivy k8s \
  --include-namespaces monitoring \
  --scanners misconfig \
  --report all

		

Look for vulnerabilities:

			
DOCKER_CONFIG=$HOME/.docker-trivy \
trivy k8s \
  --include-namespaces monitoring \
  --scanners vuln \
  --report all

		

Save a Baseline

			
1. DOCKER_CONFIG=$HOME/.docker-trivy \
trivy k8s \
  --include-namespaces monitoring \
  --severity HIGH,CRITICAL \
  --report all \
  > trivy-k8s-high-critical.txt
2. less trivy-k8s-high-critical.txt

		

Part 3 Trivy – Continuous Scanning:

Gitops-managed Trivy Operator w/ArgoCD installing the Helm-Chart & generates kubernetes security reports continuously. The values depending on Helm-Chart version & the The Trivy Operator adds examples such as;
- VulnerabilityReport
  - ConfigAuditReport
  - ClusterVulnerabilityReport
  - ClusterConfigAuditReport
  - ExposedSecretReport
  - RbacAssessmentReport
Sources
- applications/trivy-operator.yaml
  - https://argo-cd.readthedocs.io/en/latest/user-guide/application-specification/
  - https://argo-cd.readthedocs.io/en/latest/user-guide/multiple_sources/
- apps/trivy-operator/values.yaml
  - https://github.com/aquasecurity/trivy-operator/blob/main/deploy/helm/values.yaml

Pre-Commands:

			
/gitops - Add gitops files:
- applications/trivy-operator.yaml
- apps/trivy-operator/values.yaml
--------------------------------------------
Helm CLI:
- helm repo add aqua https://aquasecurity.github.io/helm-charts/
helm repo update
- helm template trivy-operator aqua/trivy-operator \
  --version 0.35.0 \
  --namespace trivy-system \
  -f apps/trivy-operator/values.yaml \
  > /tmp/trivy-rendered.yaml
--------------------------------------------
Git commands:
- git status
- git diff
- git add applications/trivy-operator.yaml apps/trivy-operator/values.yaml
- git commit -m "add Trivy Operator through Argo CD"
- git push
--------------------------------------------
/infra:
- terraform fmt -recursive
- terraform validate
- terraform plan
- terraform apply
--------------------------------------------
AWS CLI:
- aws eks update-kubeconfig \
  --region us-east-1 \
  --name say-when-east
--------------------------------------------

		

Kubectl aka Trivy Commands w/new YAML:

			
Kubectl CLI - Optional:
- kubectl annotate application monitoring-root -n argocd \
  argocd.argoproj.io/refresh=hard --overwrite
- kubectl get applications -n argocd OR check k9s
- kubectl get pods -n trivy-system OR check k9s

		

Use a consistent broad → narrow → specific → remediate → recheck workflow
- Broad Cluster View
- Namespace Summary
- Sort Worst-First
- Inspect one workload
- Tighten to Critical only
- Then critical & high
- Check actual image/version
- Check config-audit
- Make gitops fix

Dont be alarmed for about 10 minutes you will notice 1 – 3 pods & 1 – 6 containers moving from init, pending, error, etc. Around the 10 minute mark everything will stabilize, or should…. bump bump buhhhhh!

kubectl get vulnerabilityreports -A:

			
NAMESPACE      NAME                                                   REPOSITORY                                       TAG                    SCANNER   AGE
argocd         statefulset-78579cd8ff                                 argoproj/argocd                                  v2.11.2                Trivy     25m
kube-system    replicaset-metrics-server-6c68cdb89-metrics-server     metrics-server/metrics-server                    v0.7.0                 Trivy     20m
monitoring     statefulset-tempo-tempo                                grafana/tempo                                    2.3.0                  Trivy     24m
monitoring     statefulset-thanos-storegateway-storegateway           bitnamilegacy/thanos                             0.39.2-debian-12-r2    Trivy     25m
trivy-system   replicaset-trivy-operator-646b6cb4cf-trivy-operator    aquasec/trivy-operator                           0.33.0                 Trivy     21m

		

kubectl get vulnerabilityreports -n monitoring:

			
NAME                                           REPOSITORY                                       TAG                   SCANNER   AGE
daemonset-6c9bb4f54f                           prometheus/node-exporter                         v1.12.1-distroless    Trivy     39m
replicaset-55447bbb5f                          prometheus-operator/prometheus-operator          v0.92.1               Trivy     38m
replicaset-5b9c78c99f                          kiwigrid/k8s-sidecar                             2.10.0                Trivy     33m
replicaset-5f5454ccf5                          kube-state-metrics/kube-state-metrics            v2.19.1               Trivy     38m
replicaset-loki-gateway-5bb6df84df-nginx       nginxinc/nginx-unprivileged                      1.24-alpine           Trivy     38m
replicaset-thanos-query-f4f8b7ff7-query        bitnamilegacy/thanos                             0.39.2-debian-12-r2   Trivy     39m
statefulset-6547f6bbc9                         prometheus/prometheus                            v3.13.1-distroless    Trivy     37m
statefulset-6ddfb59cf5                         prometheus-operator/prometheus-config-reloader   v0.92.1               Trivy     37m
statefulset-loki-loki                          grafana/loki                                     3.0.0                 Trivy     38m
statefulset-tempo-tempo                        grafana/tempo                                    2.3.0                 Trivy     40m
statefulset-thanos-storegateway-storegateway   bitnamilegacy/thanos                             0.39.2-debian-12-r2   Trivy     40m

		

kubectl get configauditreports -A:

			
NAMESPACE      NAME                                                             SCANNER   AGE
argocd         service-argocd-server                                            Trivy     27m
argocd         statefulset-argocd-application-controller                        Trivy     27m
default        service-kubernetes                                               Trivy     27m
kube-system    service-kube-dns                                                 Trivy     27m
kube-system    service-kube-prometheus-stack-coredns                            Trivy     26m
kube-system    service-metrics-server                                           Trivy     27m
monitoring     service-thanos-storegateway                                      Trivy     27m
monitoring     statefulset-loki                                                 Trivy     27m
monitoring     statefulset-prometheus-kube-prometheus-stack-prometheus          Trivy     25m
monitoring     statefulset-tempo                                                Trivy     27m
monitoring     statefulset-thanos-storegateway                                  Trivy     27m
trivy-system   replicaset-trivy-operator-646b6cb4cf                             Trivy     27m
trivy-system   service-trivy-operator                                           Trivy     27m

		

kubectl get rbacassessmentreports -A:

			
NAMESPACE      NAME                                             SCANNER   AGE
argocd         role-argocd-application-controller               Trivy     29m
argocd         role-argocd-server                               Trivy     29m
kube-system    role-extension-apiserver-authentication-reader   Trivy     29m
trivy-system   role-trivy-operator                              Trivy     29m
trivy-system   role-trivy-operator-leader-election              Trivy     29m

		

kubectl get vulnerabilityreports -A -o wide:

kubectl get clustervulnerabilityreports:

			
NAME                                       REPOSITORY   TAG                   SCANNER   AGE
clustersbomreport-8566b66bdc-k8s-cluster   kubernetes   1.31.14-eks-bca9cf6   Trivy     36m

kubectl get clusterconfigauditreports:

			
NAME                                                        SCANNER   AGE
persistentvolume-pvc-08ed4945-09ac-4361-a920-0e1ccaeb4ff3   Trivy     37m

Sort the worst stuff:

			
kubectl get vulnerabilityreports -n monitoring -o json | jq -r '
.items[] |
[
  (.report.summary.criticalCount // 0),
  (.report.summary.highCount // 0),
  .metadata.name
] | @tsv' | sort -nr
--------------------------------------------
4       22      statefulset-thanos-storegateway-storegateway
4       22      replicaset-thanos-query-f4f8b7ff7-query
3       26      statefulset-tempo-tempo
3       16      statefulset-loki-loki
3       14      replicaset-loki-gateway-5bb6df84df-nginx
0       2       statefulset-6547f6bbc9
0       1       replicaset-5f5454ccf5
0       1       replicaset-5b9c78c99f
0       1       replicaset-55447bbb5f
0       0       statefulset-6ddfb59cf5
0       0       daemonset-6c9bb4f54f

		

kubectl describe:

			
- kubectl describe vulnerabilityreport \
  -n monitoring \
  statefulset-thanos-storegateway-storegateway

kubectl get vulnerabilityreport:

			
kubectl get vulnerabilityreport \
  -n monitoring \
  statefulset-thanos-storegateway-storegateway \
  -o json | jq -r '
.report.vulnerabilities[] |
select(.severity == "CRITICAL") |
"\(.vulnerabilityID)  \(.resource)  \(.installedVersion) -> \(.fixedVersion)"' | sort -u
CVE-2025-68121  stdlib  v1.25.0 -> 1.24.13, 1.25.7, 1.26.0-rc.3
CVE-2026-33186  google.golang.org/grpc  v1.63.2 -> 1.79.3

		

Kubescape, cuz we wanna see more stuff:

Sources:
Cluster posture assessment
NSA/CISA-style checks
RBAC, pod security, workload misconfigurations
Compare findings against Trivy so the posts build on each other

Pre-Commands:

			
- sudo add-apt-repository ppa:kubescape/kubescape
- sudo apt update
- sudo apt install kubescape
- kubescape version

kubectl scan framework nsa:

			
- kubescape scan framework nsa \
  --include-namespaces monitoring
  --output kubescape-nsa-monitoring.md

kubescape list frameworks:

			
│ Supported frameworks │
├──────────────────────┤
│      AllControls     │
├──────────────────────┤
│       ArmoBest       │
├──────────────────────┤
│      DevOpsBest      │
├──────────────────────┤
│         MITRE        │
├──────────────────────┤
│          NSA         │
├──────────────────────┤
│         SOC2         │
├──────────────────────┤
│    cis-aks-t1.2.0    │
├──────────────────────┤
│    cis-aks-t1.8.0    │
├──────────────────────┤
│    cis-eks-t1.7.0    │
├──────────────────────┤
│    cis-eks-t1.8.0    │
├──────────────────────┤
│    cis-gke-v1.9.0    │
├──────────────────────┤
│      cis-v1.10.0     │
├──────────────────────┤
│      cis-v1.12.0     │

		

kubescape scan view control:

			
- kubescape scan framework nsa \
  --include-namespaces monitoring \
  --view control

kubescape scan framework mitre:

			
kubescape scan framework mitre \
  --include-namespaces monitoring

Tag: helmcharts

EKS Cluster: Part 8 – K8 Security Tooling; Trivy & Kubescape