EKS Cluster: Part 9 – Ai Ops/Dev = K8sGPT

Series of blog posts show progress of updating/adding to EKS Cluster, this post covers adding K8sGPT under Ai Operations. See below for past posts:

• EKS Cluster: Part 1 – AWS Infrastructure
• EKS Cluster: Part 2 – Orchestration & Monitoring

K8sGPT: feed it unhealthy workloads to analyze K8s events/errors & compare to kubectl describe/logs

• Good opportunity to intentionally break something and see whether it identifies the root cause

Folders & files organized as such:

• Flow order:
  • K8sGPT Bedrock IAM policy
  • K8sGPT IRSA role
  • EKS OIDC trust restricted to
    • ServiceAccount: k8sgpt:k8sgpt-bedrock
  • Permissions include –
    • bedrock:InvokeModel
    • bedrock:InvokeModelWithResponseStream

• gitops
  • Applications
    • k8sgpt-operator.yaml
    • k8sgpt.yaml
  • Apps
    • k8sgpt-operator
      • values.yaml
    • k8sgpt
      • k8sgpt.yaml
      • serviceaccount.yaml
• infra
  • k8sgpt-iam.tf
  • k8sgpt-policy.tf

Pre-Commands:

==========================================
- helm repo add k8sgpt https://charts.k8sgpt.ai/
- helm repo update
==========================================
- helm search repo k8sgpt/k8sgpt-operator \
  --versions |
head -10
==========================================
- helm show values k8sgpt/k8sgpt-operator --version 0.2.29 |
grep -n -A20 -B5 -E \
'serviceAccount:|podSecurityContext|securityContext|resources:'
==========================================
- helm search repo k8sgpt
==========================================

Applications/k8sgpt-operator.yaml:

• Sources:
  • https://kubernetes.io/docs/tasks/configure-pod-container/security-context/
  • https://github.com/k8sgpt-ai/k8sgpt-operator?
  • https://github.com/k8sgpt-ai/k8sgpt-operator/tree/v0.2.29/chart/operator
  • https://argo-cd.readthedocs.io/en/stable/user-guide/multiple_sources/
• Tells ArgoCD to install the official K8sGPT Operator Helm Chart.

Variety of kubectl commands to check status:

==========================================
- kubectl get application k8sgpt-operator -n argocd
==========================================
==========================================
- kubectl explain k8sgpt.spec.ai --recursive
==========================================
==========================================
- kubectl get pods -n k8sgpt-operator-system
==========================================
==========================================
- kubectl get serviceaccount -A | grep -i k8sgpt
==========================================

Apps/k8sgpt-operator/values.yaml:

• Sources:
  • https://github.com/k8sgpt-ai/k8sgpt-operator?
  • https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html
• Resources, security context and IRSA service account & configures operator Deployment installed by Helm.

Apps/k8sgpt/k8sgpt.yaml:

• Sources:
  • https://argo-cd.readthedocs.io/en/stable/user-guide/application-specification/
  • https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/
  • https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/
  • https://github.com/k8sgpt-ai/k8sgpt-operator/blob/v0.2.29/config/samples/exhaustive_sample.yaml
  • https://github.com/k8sgpt-ai/k8sgpt-operator/blob/v0.2.29/config/samples/secure_k8sgpt_example.yaml
  • https://github.com/k8sgpt-ai/k8sgpt-operator/blob/v0.2.29/config/samples/filters_example.yaml
  • https://github.com/k8sgpt-ai/k8sgpt-operator/blob/v0.2.29/config/crd/bases/core.k8sgpt.ai_k8sgpts.yaml
• Makes Argo CD deploy the K8sGPT custom resource from your repository after the operator exists & configures Bedrock, filters and analysis schedule

CRDs: to help create infra for AWS IAM ISRA & Policy as well as Bedrock

• Show all top-level K8sGPT
• Show complete Ai schmea w/out terminal formatting
• Search CRD for ServiceAccount & Pod Settings
• Confirm Bedrock backend name

==========================================
- kubectl get crd k8sgpts.core.k8sgpt.ai -o json |
jq -r '
  paths
  | select(.[-1] == "serviceAccountIRSA")
  | map(tostring)
  | join(".")
'
==========================================
-  kubectl get crd k8sgpts.core.k8sgpt.ai -o json |
jq -r '
  .spec.versions[]
  | select(.name == "v1alpha1")
  | .schema.openAPIV3Schema.properties.spec.properties
  | keys[]
'
ai
sink
targetNamespace
version
==========================================

infra/k8sgpt.yaml:

• Sources:
  • https://docs.aws.amazon.com/nova/latest/userguide/what-is-nova.html?
• Leverage current IAM.tf & policy-tf as well as list the bedrock foundational models

Applications/k8sgpt.yaml:

• Sources:
  • https://argo-cd.readthedocs.io/en/stable/user-guide/sync-waves/
  • https://argo-cd.readthedocs.io/en/stable/user-guide/application-specification/
  • https://argo-cd.readthedocs.io/en/stable/user-guide/auto_sync/
• Makes Argo CD deploy the K8sGPT custom resource from your repository after the operator exists.

infra/k8sgpt-iam.tf:

• Sources:
  • https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html
  • https://docs.aws.amazon.com/eks/latest/userguide/associate-service-account-role.html
  • https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/iam_policy_document
• Creates the IAM role and allows the dynamically created K8sGPT ServiceAccount to assume it.

infra/k8sgpt-policy.tf:

• Sources:
  • https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/iam_policy
• Permits the K8sGPT analyzer to invoke only the selected Amazon Nova model.