Say When..

Series of blog posts show progress of updating/adding to EKS Cluster, this post covers adding K8sGPT under Ai Operations. See below for past posts:

K8sGPT: feed it unhealthy workloads to analyze K8s events/errors & compare to kubectl describe/logs

Below are ways to check if K8sGPT is working
- RBAC
- K8sGPT Day-to-Day CLI Commands
- Place K8sGPT commands in an shortcut format
- See where code lives & errors example

Verify RBAC:

			
==========================================
- kubectl auth can-i get events \
  --all-namespaces \
  --as=system:serviceaccount:k8sgpt-operator-system:k8sgpt-k8sgpt-operator-system
==========================================
- kubectl auth can-i create results.core.k8sgpt.ai \
  --all-namespaces \
  --as=system:serviceaccount:k8sgpt-operator-system:k8sgpt-operator-system
==========================================
kubectl get k8sgpt k8sgpt-bedrock \
  -n k8sgpt-operator-system \
  -o json |
jq '.status'
==========================================
- kubectl api-resources --api-group=core.k8sgpt.ai
==========================================
- kubectl auth can-i create results.core.k8sgpt.ai \
  --all-namespaces \
  --as=system:serviceaccount:k8sgpt-operator-system:k8sgpt-operator-controller-manager
==========================================
NAMESPACE                NAME                        KIND   BACKEND         AGE
k8sgpt-operator-system   defaultk8sgpttest           Pod    amazonbedrock   23s
k8sgpt-operator-system   popeyepopeye29833440r22wn   Pod    amazonbedrock   23s
==========================================

		

K8sGPT Day-to-Day CLI Commands:

			
==========================================
- kubectl get results.core.k8sgpt.ai -A
NAMESPACE                NAME                        KIND   BACKEND         AGE
k8sgpt-operator-system   defaultk8sgptimagetest      Pod    amazonbedrock   11m
k8sgpt-operator-system   popeyepopeye29833440r22wn   Pod    amazonbedrock   21m
==========================================
- kubectl describe pod k8sgpt-image-test -n default |
sed -n '/Events:/,$p'
Events:
  Type     Reason     Age                  From               Message
  ----     ------     ----                 ----               -------
  Normal   Scheduled  13m                  default-scheduler  Successfully assigned default/k8sgpt-image-test to ip-10-0-2-8.ec2.internal
  Normal   Pulling    11m (x4 over 13m)    kubelet            spec.containers{test}: Pulling image "busybox:this-version-does-not-exist"
  Warning  Failed     11m (x4 over 13m)    kubelet            spec.containers{test}: Failed to pull image "busybox:this-version-does-not-exist"
  Warning  Failed     11m (x4 over 13m)    kubelet            spec.containers{test}: Error: ErrImagePull
  Warning  Failed     11m (x6 over 13m)    kubelet            spec.containers{test}: Error: ImagePullBackOff
  Normal   BackOff    3m2s (x43 over 13m)  kubelet            spec.containers{test}: Back-off pulling image "busybox:this-version-does-not-exist"
==========================================
- kubectl get results.core.k8sgpt.ai -A
NAMESPACE                NAME                        KIND   BACKEND         AGE
k8sgpt-operator-system   defaultk8sgptimagetest      Pod    amazonbedrock   11m
k8sgpt-operator-system   popeyepopeye29833440r22wn   Pod    amazonbedrock   21m

		

Place K8sGPT commands in an shortcut format

			
==========================================
- kgpt-scan
NAMESPACE                NAME                        KIND   BACKEND         AGE
k8sgpt-operator-system   defaultk8sgptimagetest      Pod    amazonbedrock   24m
k8sgpt-operator-system   popeyepopeye29833440r22wn   Pod    amazonbedrock   34m
==========================================
- kgpt-results
K8sGPT analysis triggered.
==========================================
- kgpt-show k8sgpt-image-test
==========================================
- kgpt-logs
==========================================

		

			
==========================================
- cat >> ~/.bashrc <<'EOF'
# ----------------------------
# K8sGPT helper commands
# ----------------------------
kgpt-scan() {
  kubectl annotate k8sgpt k8sgpt-bedrock \
    -n k8sgpt-operator-system \
    troubleshooting.k8sgpt.ai/retry="$(date +%s)" \
    --overwrite >/dev/null
  echo "K8sGPT analysis triggered."
}
kgpt-results() {
  kubectl get results.core.k8sgpt.ai -A
}
kgpt-show() {
  if [ -z "$1" ]; then
    echo "Usage: kgpt-show <kubernetes-resource-name>"
    return 1
  fi
  local result_name
  result_name="$(
    kubectl get results.core.k8sgpt.ai \
      -n k8sgpt-operator-system \
      -o json |
    jq -r --arg resource_name "$1" '
      .items[]
      | select(.spec.name == $resource_name)
      | .metadata.name
    ' |
    head -1
  )"
  if [ -z "$result_name" ]; then
    echo "No K8sGPT result found for: $1"
    return 1
  fi
  kubectl get result "$result_name" \
    -n k8sgpt-operator-system \
    -o json |
  jq '{
    result: .metadata.name,
    resource: {
      kind: .spec.kind,
      name: .spec.name,
      namespace: .spec.parentObject
    },
    errors: .spec.error,
    details: .spec.details
  }'
}
kgpt-logs() {
  kubectl logs \
    -n k8sgpt-operator-system \
    deployment/k8sgpt-operator-controller-manager \
    -c manager \
    --since=30m |
  grep -E \
    'AnalysisStep response|Created result|Updated result|Deleted result|panic|Reconciler error'
}
EOF
==========================================

		

			
==========================================
- source ~/.bashrc
==========================================

See where K8sGPT Errors are found & where code lives:

See code here:
- https://github.com/earpjennings37/say_when_git_ops/blob/main/apps/k8sgpt/k8sgpt.yaml

			
==========================================
- kgpt-results
NAMESPACE                NAME                                                 KIND          BACKEND         AGE
k8sgpt-operator-system   argocdargocdapplicationcontroller                    StatefulSet   amazonbedrock   12s
k8sgpt-operator-system   kubesystemkubeprometheusstackkubecontrollermanager   Service       amazonbedrock   12s
k8sgpt-operator-system   kubesystemkubeprometheusstackkubeetcd                Service       amazonbedrock   12s
k8sgpt-operator-system   kubesystemkubeprometheusstackkubescheduler           Service       amazonbedrock   12s
k8sgpt-operator-system   monitoringtempo                                      StatefulSet   amazonbedrock   12s
k8sgpt-operator-system   monitoringthanosstoregateway                         StatefulSet   amazonbedrock   12s
k8sgpt-operator-system   popeyepopeye29833440r22wn                            Pod           amazonbedrock   45m
==========================================

		