Fall Cleaning #1: Rakes & Repos (CI, ECR & Signed Images)

Theme & Goal of Blog Post:

  • Turn an empty repo into a signed, scanned container image in Amazon ECR, built by a pipeline that never stores an AWS key.
  • The flow includes the following
    • main: tests → ARM64 build → Trivy scan → push to ECR → SBOM → keyless signature

say_when_infra-github-actions-bootstrap.patch:

  • A one-time Terraform module, like your HCP Terraform bootstrap. It creates before any app code:
    • 5 AWS resources give CI ability to push w/a way to login w/out any keys
    • an say-when-app ECR repository that survives cluster rebuilds
    • Finally an IAM role that only main of say_when_app can use, with push access to that one repository and nothing else

say_when_app-fall-01.patch:

  • say_when_app repo and an app skeleton + 5 tests
  • FastAPI “cleanup” app
  • then a CI pipeline that builds on native ARM64
    • Multi-stage, non-root Dockerfile (138 MB → 51 MB compressed)
      • .gitignore
      • .dockerignore
      • multi-stage non-root Dockerfile
      • GitHub Actions workflow
  • scans, signs image SBOM and pushes it to ECR.
    • Trivy gate
    • Trust policy via Cloudtrail w/immutable IDs
    • Independently verified w/cosign
  • uses OIDC w/no AWS keys stored anywhere.

Part A: AWS Infra Side – 5 Questions i based my approach off of

  1. How long should this live? Does it come and go with the cluster, or stay forever?
  2. Who should be able to use it?
  3. What should they be allowed to do? As little as possible.
  4. What does it cost if I forget about it?
  5. What happens if someone runs destroy by accident?
============
aws sso login \
--profile "$AWS_PROFILE" \
--use-device-code
============
aws sts get-caller-identity
============

A separate bootstrap/github-actions module instead of adding to the main Terraform

  • Question 1
    • Lifetime – destroy the cluster after every session, but the app’s images and their signatures need to survive those rebuilds. Anything in the main workspace gets destroyed with it. Today’s Popeye ECR repo proves the point: it has force_delete = true, so every teardown wipes its image and you rebuild it each time.

OIDC & NOT AWS Access Keys in Github Secrets

  • Questions 2 and 4.
    • Stored keys never expire, and anyone who reads them can use them. With OIDC, GitHub issues a temporary token for each workflow run, AWS swaps it for credentials that expire after an hour, and there’s nothing to leak or rotate.
      • When connecting 2 services, OIDC or workload ID help – its the same as IRSA & Pod ID already in my cluster.

Trust policys 2 condition – aud & sub

  • Question 2
    • Without conditions, any GitHub repo in the world could present a token from GitHub’s OIDC provider and try to use your role.
      • aud checks the token was meant for AWS.
      • sub pins it to your repo, your main branch.
    • A fork, a PR branch or someone else’s repo gets denied. IAM itself recommends this: it recommends that users evaluate the IAM condition key, token.actions.githubusercontent.com:sub, in the trust policy of any role that trusts GitHub’s OIDC identity provider (IdP). github
    • StringEquals rather than StringLike: StringLike allows wildcards like repo:earpjennings37/*.
      • That’s handy, but it opens the role to every repo you’ll ever create. An exact match is the safe default.

Permission list (ecr: action)

  • Question 3
    • Least privilege – CI needs to log in, push layers and read back what it pushed. Nothing else, and only for this one repository.
    • GetAuthorizationToken uses "*": some AWS actions can’t be limited to one resource. Logging in to ECR is account-wide by design, so "*" is required there. Every other action is pinned to the repo’s ARN.

ECR repository settings.

  • Question 5
  • scan_on_push = true:
    • a free second opinion. Trivy scans in CI before the push, and AWS scans again after it.
  • force_delete = false:
    • If this module is ever destroyed by accident, AWS refuses to delete a repository that still has images. That’s a deliberate speed bump. (Popeye’s repo has the opposite setting, because it’s meant to be disposable.)
  • MUTABLE tags:
    • this one’s a judgment call. IMMUTABLE sounds safer, but Cosign stores signatures as extra tags, and immutable tags can break re-running a workflow.
      • The real protection is that Kyverno (Fall #3) will check the signature on the image’s digest, the fingerprint of its exact contents. A moved tag can’t fake that.

Lifecycle policy – untagged images expire after 14 days

  • Question 4
    • Cost – ECR keeps everything forever unless told otherwise. Every build leaves untagged layers behind, and they’d quietly add up, just like the 33 orphaned EBS volumes did.

Xtrahh Detailzzz

  • max_session_duration = 3600:
    • one hour, the shortest standard session. A CI run takes minutes.
  • aws_iam_policy_document instead of hand-written JSON:
    • Terraform checks its structure, it’s easier to read, and you can reference other resources’ ARNs directly.
  • Variables with defaults: works with zero configuration today, but you could point it at another repo or branch later without editing code.
  • Local state, no backend: matches your HCP bootstrap. The trade-off is that terraform.tfstate lives only on your machine, so back it up.

Part B: AWS Infra Side – 4 Files Created

versions.tf

  • Sources:
  • This is the same version range as your bootstrap/hcp-terraform module, so both bootstraps behave the same. There’s no backend block, so the state is stored locally in this folder, just like the HCP bootstrap.
  • Your bootstrap/hcp-terraform/versions.tf: the version range was copied from there to keep the two bootstraps consistent.

variables.tf

  • Sources:
  • Every variable has a default, so you don’t need a .tfvars file. github_repository and github_branch are the security boundary: they decide who is allowed to push images.
  • Ensure to add github_owner_id & github_repository_id

main.tf

BlockSyntax source (Terraform Registry)Concept source (AWS / GitHub docs)
aws_iam_openid_connect_provideriam_openid_connect_providerGitHub Docs: OIDC in AWS and the AWS Security Blog: IAM roles for GitHub Actions
aws_ecr_repositoryecr_repositoryAWS: basic scanning in ECR
aws_ecr_lifecycle_policyecr_lifecycle_policyAWS: ECR lifecycle policies and examples
Trust policy (aws_iam_policy_document)iam_policy_documentGitHub Docs: OIDC in AWS (the aud and sub conditions)
aws_iam_roleiam_roleAWS Security Blog
aws_iam_role_policy (ECR push permissions)iam_role_policyAWS: ECR lifecycle and repository docs (the push/pull actions)
BlockWhat it doesWhy it matters
aws_iam_openid_connect_providerTells AWS to trust login tokens that GitHub Actions issuesCI gets temporary credentials (1 hour), so there are no access keys to leak
aws_ecr_repositoryThe image registry for the appscan_on_push runs AWS’s own vulnerability scan. force_delete = false means a stray destroy can’t wipe your images
aws_ecr_lifecycle_policyDeletes untagged leftovers after 14 daysKeeps storage costs near zero
Trust policy (github_actions_trust)Who can use the roleOnly a token from repo:earpjennings37/say_when_app:ref:refs/heads/main. A fork, another repo or a PR branch gets “access denied”
aws_iam_roleThe role CI assumesIts sessions last at most 1 hour
Permissions policy (ecr_push)What the role can doLog in to ECR, plus push and pull this one repository. Nothing else in your account
  • Similar to variables above, add locals to pull values from github_oidc_sub
    • }Why split(): your existing github_repository variable already holds "earpjennings37/say_when_app". split("/", ...) breaks it into [0] = "earpjennings37" and [1] = "say_when_app", so there’s still one place to set the repo name. The two new ID variables slot in next to each part.

outputs.tf

  • Sources:
  • Save the two outputs: github_actions_role_arn and ecr_repository_url.
  • If apply fails with EntityAlreadyExists on the OIDC provider, your account already trusts GitHub from an earlier experiment. Import it and apply again:
============
git checkout main && git pull
============
cd bootstrap/github-actions
============
terraform init
============
terraform plan # expect: 6 to add, 0 to change, 0 to destroy
============
terraform apply
============
terraform output
============

Part C: AWS Infra Side – Github Variables

  • Sources:
  • Tell the app repo which role to use. On GitHub, go to say_when_app → Settings → Secrets and variables → Actions → Variables tab → New repository variable:
    • Name: AWS_ROLE_ARN
    • Value: the github_actions_role_arn output
  • Why a variable and not a secret:
    • An ARN is just an address, not a credential.
    • On its own it grants nothing, because the role only trusts tokens from your repo’s main branch.
      • Keeping it as a variable also means it’s visible in workflow logs, which makes debugging easier.

Part D: The App Repo patch

  • Sources:
  • Watch the real run on main: go to Actions → the run for your merge commit. This time every step runs: push to ECR, SBOM, sign, attest and verify. The run’s Summary shows the image tag and digest.
  • On the PR the Build
    • Scan and sign workflow runs the tests
    • ARM64 build
    • Trivy scan only.
    • Nothing is pushed or signed on a PR.
    • Wait for both jobs to go green, then merge.
  • The empty __init__.py files turn the folders into Python packages, so from app.main import app works in the tests.
============
mkdir -p app tests .github/workflows
============
touch app/__init__.py tests/__init__.py
============

Isolated Python envmt

============
sudo apt install -y python3.10-venv # one-time on Ubuntu/WSL
============
python3 -m venv .venv
============
source .venv/bin/activate # every new terminal
============
which python pip
============

Dependencies: ranges, split runtime vs dev

  • Pin ranges of split runtime vs dev as dependencies
    • Why the ranges?
      • (>=0.115,<1.0): you get bug fixes automatically, but no surprise breaking major versions.
  • Sources:
FilePackagesShips in the image?
requirements.txtfastapi, uvicorn, prometheus-client, PyYAMLYes
requirements-dev.txtpytest, httpxNo, tests only
pip install -r requirements.txt -r requirements-dev.txt

Chores as config, not code

- id: rake-leaves
name: Rake the leaves
expect: [Rake, Leaf]
reject: [Basketball, Sports Ball]
min_confidence: 80
  • app/main.py:
    • exposes 5 endpoints
EndpointPurpose
/healthzKubernetes liveness and readiness probes
/versionWhich build is running (the git SHA, set at build time)
/choresAll chores from the YAML file
/chores/{id}One chore, or a 404 if unknown
/metricsPrometheus counter of requests by endpoint

Testing & Port-Forwarding

============
pytest -q # 5 passed
============
python -m uvicorn app.main:app --port 8080 # open http://localhost:8080/docs
============
http://localhost:8080/docs
============
========
Expand /chores/{chore_id}
→ Try it out → enter rake-leaves → Execute. You’ll get the chore back.
Then try shovel-snow to see the 404 with “No chore called ‘shovel-snow’”.
========

INFO:     Started server process [89667]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8080 (Press CTRL+C to quit)
INFO:     127.0.0.1:37590 - "GET /docs HTTP/1.1" 200 OK
INFO:     127.0.0.1:37590 - "GET /openapi.json HTTP/1.1" 200 OK
INFO:     127.0.0.1:48326 - "GET /chores/rake-leaves HTTP/1.1" 200 OK
INFO:     127.0.0.1:48326 - "GET /chores/rake-leaves HTTP/1.1" 200 OK
INFO:     127.0.0.1:46344 - "GET /chores/shovel-snow HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:46344 - "GET /chores/shovel-snow HTTP/1.1" 404 Not Found
INFO:     127.0.0.1:33238 - "GET /chores/rake-leaves HTTP/1.1" 200 OK
INFO:     127.0.0.1:48234 - "GET /metrics HTTP/1.1" 200 OK
INFO:     127.0.0.1:48234 - "GET /favicon.ico HTTP/1.1" 404 Not Found
========
Open http://localhost:8080/metrics. Look for lines like:
cleanup_http_requests_total{endpoint="chore"} 2.0
========
That’s the counter you just bumped, and it’s exactly what Prometheus will scrape in the cluster later.
========

Part E: The App Repo – Ignoring $tuff

FileKeeps outWhy it matters
.gitignore.venv/, __pycache__/, *.jpg, *.jpeg, *.heic, photos/, sbom*.jsonPhone photos carry GPS coordinates in their EXIF data; one stray git add . could publish my home location. .png is allowed on purpose, for blog screenshots.
.dockerignore.git, .venv, tests, docs, *.md, photosA smaller build context builds faster, and less in the image means less for scanners to flag.

Part F: The App Repo – Docker File-n-Stuff

Build Dockerfile

  • It’s the recipe CI uses to turn your app into the image that runs on the cluster. The image runs as user w/no shell, no pip, & final version after reviewed by Trivy is good to go.
========
docker build --build-arg APP_VERSION=local-test -t say-when-app:local .
========
[+] Building 23.1s (17/17) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 866B 0.0s
=> resolve image config for docker-image://docker.io/docker/dockerfile:1 1.2s
=> [auth] docker/dockerfile:pull token for registry-1.docker.io
========
========
docker run --rm -d -p 8080:8080 --name sw say-when-app:local
========
curl -s localhost:8080/version # → {"version":"local-test"}
========
curl -s localhost:8080/healthz # → {"status":"ok"}
========
docker exec sw id # → uid=10001(app) ... NOT root
========
docker images say-when-app:local # note the SIZE for the blog
========
docker stop sw
========

Port-Forward

========
docker run --rm -d -p 8080:8080 --name sw say-when-app:local
========
http://localhost:8080/docs
========

Part G: The App Repo – Github Workflows Build

.github/workflows/build.yaml

========
wc -l .github/workflows/build.yaml
========
python -c "import yaml; yaml.safe_load(open('.github/workflows/build.yaml')); print('YAML OK')"
========
grep -c '@[0-9a-f]\{40\}' .github/workflows/build.yaml
========
docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \
aquasec/trivy:latest image \
--severity HIGH,CRITICAL --ignore-unfixed \
say-when-app:local
========

Part H: Verify both Infra & App are Werrrrrrkin’

Trivy Gate

  • Any fixable HIGH or CRITICAL vulnerability fails the job. Unfixed ones are skipped because nothing can be done about them yet, and they would block every build.
  • Sources:
severity: HIGH,CRITICAL
ignore-unfixed: true
exit-code: "1"

OIDC

  • id-token: write lets the job request a short-lived token from GitHub.
  • AWS checks it against the role’s trust policy and returns credentials that expire within an hour.
  • The role ARN lives in a repository variable, vars.AWS_ROLE_ARN, because an ARN is not a secret.
  • The image is tagged with the commit SHA and signed by its digest, because tags can move and digests can’t.
  • Sources:

SBOM & Keyless Signing

Final CLI Commands:

======================
aws ecr describe-images --repository-name say-when-app \
--query 'imageDetails[].{tag:imageTags[0],pushed:imagePushedAt,bytes:imageSizeInBytes}' \
--output table
======================
----------------------------------------------------------------------------------------------
| DescribeImages |
+----------+------------------------------------+--------------------------------------------+
| bytes | pushed | tag |
+----------+------------------------------------+--------------------------------------------+
| 10965 | 2026-11-18T12:54:22.807000-06:00 | None |
| 3000000 | 2026-11-18T12:54:25.850000-06:00 | None |
| 50000000| 2026-11-18T12:54:13.037000-06:00 | 6000000000000000000000000000000000000000 |
======================
======================
curl -sSfL -o cosign https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64
chmod +x cosign && sudo mv cosign /usr/local/bin/
cosign version
======================
______ ______ _______. __ _______ .__ __.
/ | / __ \ / || | / _____|| \ | |
| ,----'| | | | | (----`| | | | __ | \| |
| | | | | | \ \ | | | | |_ | | . ` |
| `----.| `--' | .----) | | | | |__| | | |\ |
\______| \______/ |_______/ |__| \______| |__| \__|
cosign: A tool for Container Signing, Verification and Storage in an OCI registry
GitVersion: v3.1.4
GitCommit: 0
GitTreeState: clean
BuildDate: 2026-10-20T17:33:02Z
GoVersion: go1.27.3
Compiler: gc
Platform: linux/amd64
======================
======================
export DOCKER_CONFIG=$(mktemp -d)
aws ecr get-login-password | docker login --username AWS --password-stdin "$REGISTRY"
cosign verify "$REGISTRY/say-when-app:$TAG" \
--certificate-identity "$ID" --certificate-oidc-issuer "$ISSUER" > /dev/null \
&& echo "✅ Signature verified"
cosign verify-attestation --type spdxjson "$REGISTRY/say-when-app:$TAG" \
--certificate-identity "$ID" --certificate-oidc-issuer "$ISSUER" > /dev/null \
&& echo "✅ SBOM attestation verified"
rm -rf "$DOCKER_CONFIG"; unset DOCKER_CONFIG
======================
Verification for --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- Existence of the claims in the transparency log was verified offline
- The code-signing certificate was verified using trusted certificate authority certificates
✅ SBOM attestation verified