kyverno – Say When..

Series of blog posts show progress of updating/adding to EKS Cluster, this post covers adding a few K8 Security Tooling; Kyverno.

See below for past posts:

Kyverno, stuff like Policy as Code:

Sources:
Start in Audit mode, then move selected policies to Enforce
Good policies for lab:
- require standard labels
- require CPU/memory requests
- require CPU/memory limits
- disallow privileged containers
- restrict hostNetwork
- block privilege escalation
- require runAsNonRoot
- restrict dangerous volume types

Flow/Series of Steps Installing:

Let the official Helm chart own Kyverno’s infrastructure and let your Git repo own only the Helm overrides + policy definitions. That’s cleaner and closer to how I’d structure this in an actual platform/security GitOps repository

Where Kyverno works w/other tools installed:

Syft
SBOM
Cosign
Sign Popeye Image
Kyverno ImageValidatePolicy
Refuse unsigned/untrusted images

Steps of adding Kyverno:

Deploy Kyverno Helm chart
Confirm all Kyverno controllers are Running
Add kyverno-policies Argo Application
Add the three policy YAML files
Start them in Audit mode
Review violations
Enforce later

Folders & files organized as such:

Applications
- kyverno.yaml
- kyverno-policies.yaml
Apps/kyverno
- values.yaml
Apps/policies
- require-labels.yaml
- disallow-privileged.yaml
- require-resource-limits.yaml
- require-run-as-nonroot.yaml
- require-drop-all-capabilities.yaml
- disallow-host-namespaces.yaml

kyverno.yaml:

Snag the first chart version you see aka latest

			
===============================================================
- helm repo add kyverno https://kyverno.github.io/kyverno/
- helm repo update
- helm search repo kyverno/kyverno --versions | head -10
===============================================================
NAME                            CHART VERSION   APP VERSION     DESCRIPTION                                       
kyverno/kyverno                 3.9.1           v1.19.1         Kubernetes Native Policy Management               
kyverno/kyverno                 3.9.0           v1.19.0         Kubernetes Native Policy Management               
kyverno/kyverno                 3.8.2           v1.18.2         Kubernetes Native Policy Management               
kyverno/kyverno                 3.8.1           v1.18.1         Kubernetes Native Policy Management               
kyverno/kyverno                 3.8.0           v1.18.0         Kubernetes Native Policy Management               
kyverno/kyverno                 3.7.2           v1.17.2         Kubernetes Native Policy Management               
kyverno/kyverno                 3.7.1           v1.17.1         Kubernetes Native Policy Management               
kyverno/kyverno                 3.7.0           v1.17.0         Kubernetes Native Policy Management               
kyverno/kyverno                 3.6.4           v1.16.4         Kubernetes Native Policy Management    
===============================================================

		

apps/kyverno/values.yaml:

A default Kyverno install has four controller Deployments, normally one replica each for a non-production environment.
- Production HA uses multiple replicas, but that would be unnecessary overhead for this lab.
Do NOT disable the reports controller, because policy reports are one of the most useful parts for learning Kyverno in audit mode.

Git push & Terraform apply:

Get Kyverno installed & then add policies

			
============================
- kubectl annotate application kyverno -n argocd \
  argocd.argoproj.io/refresh=hard \
  --overwrite
============================
- kubectl rollout restart deployment kyverno-admission-controller -n kyverno
============================
- kubectl patch application monitoring-root -n argocd \
  --type merge \
  -p '{"operation":{"sync":{"revision":"main","prune":true}}}'
application.argoproj.io/monitoring-root patched
============================

		

applications/kyverno-policies.yaml:

Policies are not placed in Helm values – Kyverno engine to applications/kyverno.yaml & security rules to applications/kyverno-policies.yaml
Create a second ArgoCD application tile.
- This separation is cleaner because you can upgrade Kyverno without mixing that change with your organization’s policy code.

apps/kyverno/policies/require-labels.yaml:

This checks Pods for – app.kubernetes.io/name and only audits violations for now. Kyverno’s official examples use that label as a best-practice identifier.
- https://kyverno.io/policies/best-practices/require-labels/require-labels/
- https://kyverno.io/docs/policy-types/validating-policy

apps/kyverno/policies/disallow-privileged.yaml:

This detects privileged normal containers, init containers, and ephemeral containers. This is based directly on Kyverno’s current ValidatingPolicy pattern.
- https://kyverno.io/policies/pod-security-vpol/baseline/disallow-privileged-containers/disallow-privileged-containers/

apps/kyverno/policies/require-resource-limits.yaml:

Kyverno’s best-practice policy checks CPU/memory requests and memory limits. Make ours slightly stricter and require both CPU and memory requests and limits.

Commands to view Kyverno:

Below are commands to try & confirm your work in K9s or the CLI

			
============================
-  kubectl get validatingpolicies.policies.kyverno.io
NAME                      AGE     READY
disallow-privileged       2m22s   true
require-labels            2m22s   true
require-resource-limits   2m22s   true
============================

		

			
============================
- kubectl get policyreports -A \
  -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name,PASS:.summary.pass,FAIL:.summary.fail,WARN:.summary.warn,ERROR:.summary.error'
NAMESPACE      NAME                                   PASS   FAIL   WARN   ERROR
argocd         6a1bdd52-e37b-4bbb-8db2-bc7d73ec6d57   3      0      0      0
argocd         72824d76-163f-4be3-9dc7-2325ba5e45db   2      1      0      0
argocd         7d5e4826-4119-4329-a50b-31f43bb538b3   3      0      0      0
argocd         840ab0c3-f855-4883-8536-45c5c2cc005e   3      0      0      0
argocd         891dbf2a-1c1c-47f8-8ac8-4f7b61eb28c9   2      1      0      0
argocd         9b126396-97a5-4ada-85df-f96d79cf1013   3      0      0      0
argocd         b34d8b2f-5c24-49d6-a8f4-ebc540c9535e   2      1      0      0
argocd         d95e9e92-99e7-4cd5-9d97-a96b81e09430   2      1      0      0
============================
-  kubectl get policyreports -n falco \
  -o json \
| jq -r '
  .items[]
  | select(.scope.name=="falco-qkvlx")
  | .results[]
  | "\(.policy) | \(.result) | \(.message)"
'
disallow-privileged | pass | success
require-labels | pass | success
require-resource-limits | fail | Containers should define CPU and memory requests and limits.
============================

		

			
============================
- kubectl get policyreports -A -o json \
| jq -r '
  .items[].results[]
  | "\(.policy) \(.result)"
' \
| sort \
| uniq -c \
| sort -nr
     91 require-labels pass
     91 disallow-privileged pass
     63 require-resource-limits fail
     40 require-resource-limits pass
     12 require-labels fail
     12 disallow-privileged fail
============================

		

			
============================
- kubectl top nodes; kubectl top pods -A --sort-by=memory | head -25
kubectl top nodes
NAME                         CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)   
ip-10-0-1-180.ec2.internal   287m         14%      1896Mi          57%         
ip-10-0-1-188.ec2.internal   244m         12%      1413Mi          45%         
ip-10-0-1-190.ec2.internal   999m         51%      1467Mi          44%         
ip-10-0-2-7.ec2.internal     42m          2%       753Mi           24%       
============================
NAMESPACE      NAME                                                   CPU(cores)   MEMORY(bytes)   
argocd         argocd-application-controller-0                        260m         462Mi           
monitoring     prometheus-kube-prometheus-stack-prometheus-0          39m          407Mi           
monitoring     kube-prometheus-stack-grafana-6867978bff-87tmb         8m           314Mi           
kube-system    aws-node-swnsp                                         140m         181Mi           
trivy-system   trivy-operator-646b6cb4cf-bl2pb                        125m         167Mi           
kube-system    aws-node-7ggks                                         105m         151Mi         
============================    

		

disallow-latest-tag.yaml:

			
============================
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kyverno-policies
  namespace: argocd
  annotations:
    argocd.argoproj.io/sync-wave: "1"
============================
- kubectl get validatingpolicies.policies.kyverno.io
NAME                      AGE    READY
disallow-latest-tag       5s     true
disallow-privileged       111m   true
require-labels            111m   true
require-resource-limits   111m   true
============================

		

disallow-privilege-escalation.yaml:

			
============================
- kubectl get policyreports -A -o json |
jq -r '
  .items[].results[]? |
  select(.result == "fail") |
  [.policy, .rule, .message] |
  @tsv
' |
sort |
uniq -c |
sort -nr
     63 require-resource-limits
     46 disallow-privilege-escalation
     21 require-labels
     12 disallow-privileged
============================

		

disallow-privilege.yaml:

			
============================
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: disallow-privileged
  annotations:
    policies.kyverno.io/title: Disallow Privileged Containers
spec:
  validationActions:
    - Audit
============================

		

Test pod – Audit:

			
============================
- kubectl run privileged-test \
  --image=busybox:1.36 \
  --restart=Never \
  --overrides='{
    "spec": {
      "containers": [{
        "name": "privileged-test",
        "image": "busybox:1.36",
        "command": ["sleep", "3600"],
        "securityContext": {
          "privileged": true
        }
      }]
    }
  }'
============================

		

			
============================
-  kubectl get policyreports -n default -o json |
jq -r '
  .items[] |
  select(.scope.name == "privileged-test") |
  .results[] |
  select(.policy == "disallow-privileged") |
  [.policy, .result, .message] |
  @tsv
'
disallow-privileged     fail    Privileged containers are not allowed.
============================

		

disallow-privilege-escalation.yaml:

Test Pod – Deny:

disallow-latest-tag.yaml:

This permits
- busybox:1.36
- ghcr.io/example/application:v2.4.1
- busybox@sha256:abc123…
- registry.example.com:5000/application:v1
It flags
- busybox
- busybox:latest
- ghcr.io/example/application
- ghcr.io/example/application:latest

Test Pod – Audit:

			
============================
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: disallow-latest-tag
  annotations:
    policies.kyverno.io/title: Require Explicit Image Version
    policies.kyverno.io/category: Best Practices
    policies.kyverno.io/severity: medium
    policies.kyverno.io/subject: Pod
  matchConstraints:
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: NotIn
          values:
            - kube-system
  validations:
    - expression: >-
        variables.allContainers.all(
          container,
          container.image.contains("@") ||
          (
            container.image.matches("^.+:[^/:]+$") &&
            !container.image.endsWith(":latest")
          )
        )
      message: >-
        Container images must use an explicit version tag or immutable digest.
        The latest tag and images without a tag are not permitted.
============================

		

 ============================
 - kubectl get validatingpolicy disallow-latest-tag \
  -o jsonpath='{.metadata.name}{" | "}{.spec.validationActions}{"\n"}'
disallow-latest-tag | ["Audit"]
============================

Test Pod – Deny:

Missing tag—should be denied:

			
============================
- kubectl run missing-tag-test \
  --image=busybox \
  --restart=Never \
  --dry-run=server
============================
Error from server: admission webhook "vpol.validate.kyverno.svc-fail-e52f2f0f" denied the request: 
Policy disallow-latest-tag failed: 
Container images must use an explicit version tag or immutable digest. 
The latest tag and images without a tag are not permitted.

		

Versioned image—should succeed as a server dry run:

			
============================
- kubectl run versioned-test \
  --image=busybox:1.36 \
  --restart=Never \
  --dry-run=server
============================
Error from server: admission webhook "vpol.validate.kyverno.svc-fail-e94d8fa6" denied the request: 
Policy disallow-privilege-escalation failed: 
Containers should set securityContext.allowPrivilegeEscalation to false.
============================

		

Run the versioned test again with a compliant security context:

			
============================
- kubectl run versioned-test \
  --image=busybox:1.36 \
  --restart=Never \
  --dry-run=server \
  --overrides='{
    "spec": {
      "containers": [{
        "name": "versioned-test",
        "image": "busybox:1.36",
        "command": ["sleep", "3600"],
        "securityContext": {
          "allowPrivilegeEscalation": false,
          "privileged": false
        }
      }]
    }
  }'
  ============================
  pod/versioned-test created (server dry run)
  ============================

		

Test the 4 new policies:

Healthy pods & application

  ============================
- kubectl get validatingpolicies.policies.kyverno.io \
  -o custom-columns='NAME:.metadata.name,ACTIONS:.spec.validationActions[*]'
NAME                            ACTIONS
disallow-host-namespaces        Audit
disallow-latest-tag             Deny
disallow-privilege-escalation   Deny
disallow-privileged             Audit
require-drop-all-capabilities   Audit
require-labels                  Audit
require-resource-limits         Audit
require-run-as-nonroot          Audit
  ============================

Test all five Audit policies at once

Test pod to see if the kyverno policies work properly

			
============================
- kubectl run kyverno-audit-test \
  --image=busybox:1.36 \
  --restart=Never \
  --overrides='{
    "spec": {
      "hostPID": true,
      "containers": [{
        "name": "kyverno-audit-test",
        "image": "busybox:1.36",
        "command": ["sleep", "300"],
        "securityContext": {
          "privileged": false,
          "allowPrivilegeEscalation": false
        }
      }]
    }
  }'
pod/kyverno-audit-test created
============================

		

Privileged Container – must be denied

			
============================
- kubectl run privileged-test \
  --image=busybox:1.36 \
  --restart=Never \
  --dry-run=server \
  --overrides='{
    "spec": {
      "containers": [{
        "name": "privileged-test",
        "image": "busybox:1.36",
        "securityContext": {
          "privileged": true,
          "allowPrivilegeEscalation": false
        }
      }]
    }
  }'
The Pod "privileged-test" is invalid:
============================

		

Privilege escalation—must be denied

			
============================
- kubectl run escalation-test \
  --image=busybox:1.36 \
  --restart=Never \
  --dry-run=server \
  --overrides='{
    "spec": {
      "containers": [{
        "name": "escalation-test",
        "image": "busybox:1.36",
        "securityContext": {
          "privileged": false,
          "allowPrivilegeEscalation": true
        }
      }]
    }
  }'
Error from server: admission webhook "vpol.validate.kyverno.svc-fail-e94d8fa6" denied the request
============================

		

Latest image—must be denied

			
============================
- kubectl run latest-test \
  --image=busybox:latest \
  --restart=Never \
  --dry-run=server \
  --overrides='{
    "spec": {
      "containers": [{
        "name": "latest-test",
        "image": "busybox:latest",
        "securityContext": {
          "privileged": false,
          "allowPrivilegeEscalation": false
        }
      }]
    }
  }'
Error from server: admission webhook "vpol.validate.kyverno.svc-fail-e52f2f0f" denied the request:
============================

		

PolicyReports:

 ============================
 - kubectl get policyreports -n default -o json | jq -r '
  .items[] |
  select(.scope.name == "kyverno-audit-test") |
  .results[] |
  [.policy, .result, .message] |
  @tsv
' | sort | column -t
disallow-host-namespaces       fail 
disallow-latest-tag            pass                                                                                                                               
disallow-privilege-escalation  pass                                                                                                                               
disallow-privileged            pass                                                                                                                                
require-drop-all-capabilities  fail  
require-labels                 fail                                                             
require-resource-limits        fail      
require-run-as-nonroot         fail      
============================

Tag: kyverno

EKS Cluster: Part 8.75 – K8 Security Tooling; Kyverno