Tag: trivy

Series of blog posts show progress of updating/adding to EKS Cluster, this post covers adding Application Sets & App Projects.

As my EKS home lab grew, so did the number of Argo CD Application manifests in my GitOps repository. Each application had its own YAML file, but several of them followed exactly the same pattern:

• Read Kubernetes manifests from Git
• Deploy them into the same EKS cluster
• Enable automatic synchronization
• Enable pruning and self-healing
• Create the destination namespace when necessary

Maintaining separate, repetitive Application files worked, but it created unnecessary duplication. To improve the design, I introduced an Argo CD ApplicationSet. Meaning instead of maintaining four nearly identical Application manifests, I can define the common configuration once and provide a list of application-specific values.

Although it still has its original name, monitoring-root now manages all lab application categories—not only monitoring. The root Application discovers the ApplicationSet, and the ApplicationSet generates its child Applications.

References

• Argo CD ApplicationSet documentation
• ApplicationSet specification
• Argo CD Projects
• Argo CD automated synchronization

Repository organization

I reorganized the GitOps repository, the applications directory defines what Argo CD should deploy. The apps directory contains the Helm values and Kubernetes manifests those Applications use.

 README.md
├── applications
│   └── lab
│       ├── ai-ops
│       │   ├── k8sgpt-operator.yaml
│       │   └── k8sgpt.yaml
│       ├── observability
│       │   ├── kube-prometheus-stack.yaml
│       │   ├── loki.yaml
│       │   ├── tempo.yaml
│       │   └── thanos.yaml
│       ├── platform
│       │   ├── gp3-storageclass.yaml
│       │   ├── metrics-server.yaml
│       │   └── monitoring-namespace.yaml
│       ├── projects
│       │   ├── lab-ai-ops.yaml
│       │   ├── lab-observability.yaml
│       │   ├── lab-platform.yaml
│       │   ├── lab-security.yaml
│       │   └── lab-workloads.yaml
│       ├── security
│       │   ├── external-secrets.yaml
│       │   ├── falco.yaml
│       │   ├── kyverno.yaml
│       │   ├── security-manifests-appset.yaml
│       │   └── trivy-operator.yaml
│       └── workloads
│           ├── external-secrets-config.yaml
│           └── secret-demo.yaml
├── apps
│   ├── 
│   └── trivy-operator
│       └── values.yaml
├── bootstrap
│   ├── argocd-namespace.yaml
│   ├── bootstrap-root.yaml
│   └── monitoring.yaml
└── docs
    └── kubectl-ai
        └── config.example.yaml

ApplicationSet configuration

The ApplicationSet uses a list generator. Each list entry provides three values:

• Application name
• Destination namespace
• Git path

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: lab-security-manifests
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions:
    - missingkey=error
  generators:
    - list:
        elements:
          - name: kube-bench
            namespace: kube-bench
            path: apps/kube-bench
  template:
    metadata:
      name: "{{ .name }}"
      namespace: argocd
      labels:
        app.kubernetes.io/managed-by: applicationset
        environment: lab
        category: security
    spec:
      project: lab-security
      source:
        repoURL: https://github.com/earpjennings37/say_when_git_ops.git
        targetRevision: main
        path: "{{ .path }}"
      destination:
        server: https://kubernetes.default.svc
        namespace: "{{ .namespace }}"
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        retry:
          limit: 5
          backoff:
            duration: 10s
            factor: 2
            maxDuration: 3m
        syncOptions:
          - CreateNamespace=true
          - ApplyOutOfSyncOnly=true
  syncPolicy:
    applicationsSync: create-update
    preserveResourcesOnDeletion: true

Important synchronization settings

• Automated-Pruning w/self-healing causes Argo CD to correct changes made directly in the cluster. Pruning removes resources that are removed from Git.
• The ApplicationSet controller can create and update generated Applications.
• This helps prevent an ApplicationSet deletion from automatically cascading into deletion of all workloads managed by its generated Applications.

=====
automated:
  prune: true
  selfHeal: true
=====
applicationsSync: create-update
=====
preserveResourcesOnDeletion: true
=====

Safely migrating existing Applications

• The four Applications already existed before I created the ApplicationSet. That meant I needed to transfer ownership without unnecessarily deleting their workloads.

Step 1: Temporarily disable root pruning

I temporarily changed the root Application – this prevented the root Application from deleting resources during the ownership transition:

automated:
  prune: false
  selfHeal: true

Step 2: Add the ApplicationSet

• I committed and pushed security-manifests-appset.yaml while the original standalone Application files still existed.

Step 3: Verify ApplicationSet ownership

I confirmed that the generated Applications had an ownerReference pointing to the ApplicationSet

=====
for APP in kube-bench kyverno-policies network-policies popeye
do
  kubectl get application "$APP" -n argocd \
    -o custom-columns='APP:.metadata.name,OWNER:.metadata.ownerReferences[0].name,SYNC:.status.sync.status,HEALTH:.status.health.status' \
    --no-headers
done
=====
lab-security-manifests

Step 4: Remove the redundant Application files

After ownership was verified, I removed the old standalone Application manifests, only the redundant Argo CD Application definitions were removed

applications/lab/security/kube-bench.yaml
applications/lab/security/kyverno-policies.yaml
applications/lab/security/network-policies.yaml
applications/lab/security/popeye.yaml

Step 5: Remove old root tracking metadata

• The Applications still contained resource-tracking metadata from monitoring-root.
• I removed only the old root tracking annotation and label after confirming that the ApplicationSet owned each Application.
• This completed the ownership handoff.

Step 6: Re-enable pruning

• After the migration was healthy, I restored
• Pruning was disabled only during the controlled ownership transition.

automated:
  prune: true
  selfHeal: true

Why I did not convert every Application

I intentionally kept the observability Applications as separate YAML files, these applications have different Helm repositories, chart structures, values, synchronization behavior, and specialized configuration.

kube-prometheus-stack
loki
tempo
thanos

For example, kube-prometheus-stack uses:

• An upstream Helm chart
• A Git repository for values
• An additional Git source for manifests
• Custom ignoreDifferences rules
• Specialized server-side apply settings

Technically, I could force all of those differences into one highly parameterized ApplicationSet. However, that would make the template more complicated and harder to understand. ApplicationSets provide the most value when the generated Applications follow a genuinely similar pattern.

Adding AppProjects

ApplicationSets reduce duplication, but they do not provide security boundaries by themselves. I also introduced Argo CD AppProject resources. Each project restricts:

• Approved source repositories
• Approved destination namespaces
• Permitted namespaced resource kinds
• Permitted cluster-scoped resource kinds

Only the root Applications remain in the built-in default project:

bootstrap-root
monitoring-root

Security project boundary

The ApplicationSet-generated Applications use:

project: lab-security

The lab-security project allows only the repositories, namespaces, and resource types required by the security tooling. For example, the project permits the generated Applications to create:

• CronJob
• ServiceAccount
• NetworkPolicy
• ClusterRole
• ClusterRoleBinding
• Kyverno ValidatingPolicy

Viewing ApplicationSets

The ApplicationSet can always be inspected with Kubernetes:

============
kubectl get applicationsets -n argocd
============
kubectl describe applicationsets
============

Final result

The completed design provides:

• Less repetitive Application YAML
• Clear repository organization
• Automatic Application generation
• Verified ownership relationships
• Self-healing and pruning
• Retry behavior
• Namespace creation
• Least-privilege AppProject boundaries
• Automatic reconstruction after cluster rebuilds
• A cleaner separation between AWS infrastructure and Kubernetes GitOps

The most important lesson was that ApplicationSets are not simply a way to eliminate files.They work best when a collection of Applications follows the same deployment pattern. When applications have significantly different behavior, keeping separate Application manifests can be clearer and safer.

The goal is not to create the fewest YAML files possible.The goal is to create a GitOps structure that is understandable, repeatable, secure, and easy to operate.