
Theme & Goal of Blog Post:
- Turn an empty repo into a signed, scanned container image in Amazon ECR, built by a pipeline that never stores an AWS key.
- The flow includes the following
main: tests → ARM64 build → Trivy scan → push to ECR → SBOM → keyless signature

say_when_infra-github-actions-bootstrap.patch:
- A one-time Terraform module, like your HCP Terraform bootstrap. It creates before any app code:
- 5 AWS resources give CI ability to push w/a way to login w/out any keys
- This is an OIDC trust between GitHub Actions and AWS
- They live in
say_when_infra/bootstrap/github-actions/
- They live in
- This is an OIDC trust between GitHub Actions and AWS
- an say-when-app ECR repository that survives cluster rebuilds
- This is found
under bootstrap/hcp-terraform
- This is found
- Finally an IAM role that only main of say_when_app can use, with push access to that one repository and nothing else
- 5 AWS resources give CI ability to push w/a way to login w/out any keys
say_when_app-fall-01.patch:
- say_when_app repo and an app skeleton + 5 tests
- FastAPI “cleanup” app
- then a CI pipeline that builds on native ARM64
- Multi-stage, non-root Dockerfile (138 MB → 51 MB compressed)
.gitignore.dockerignore- multi-stage non-root
Dockerfile - GitHub Actions workflow
- Multi-stage, non-root Dockerfile (138 MB → 51 MB compressed)
- scans, signs image SBOM and pushes it to ECR.
- Trivy gate
- Trust policy via Cloudtrail w/immutable IDs
- Independently verified w/cosign
- uses OIDC w/no AWS keys stored anywhere.
Part A: AWS Infra Side – 5 Questions i based my approach off of
- How long should this live? Does it come and go with the cluster, or stay forever?
- Who should be able to use it?
- What should they be allowed to do? As little as possible.
- What does it cost if I forget about it?
- What happens if someone runs
destroyby accident?
============aws sso login \ --profile "$AWS_PROFILE" \ --use-device-code============aws sts get-caller-identity============

A separate bootstrap/github-actions module instead of adding to the main Terraform
- Question 1
- Lifetime – destroy the cluster after every session, but the app’s images and their signatures need to survive those rebuilds. Anything in the main workspace gets destroyed with it. Today’s Popeye ECR repo proves the point: it has
force_delete = true, so every teardown wipes its image and you rebuild it each time.
- Lifetime – destroy the cluster after every session, but the app’s images and their signatures need to survive those rebuilds. Anything in the main workspace gets destroyed with it. Today’s Popeye ECR repo proves the point: it has
OIDC & NOT AWS Access Keys in Github Secrets
- Questions 2 and 4.
- Stored keys never expire, and anyone who reads them can use them. With OIDC, GitHub issues a temporary token for each workflow run, AWS swaps it for credentials that expire after an hour, and there’s nothing to leak or rotate.
- When connecting 2 services, OIDC or workload ID help – its the same as IRSA & Pod ID already in my cluster.
- Stored keys never expire, and anyone who reads them can use them. With OIDC, GitHub issues a temporary token for each workflow run, AWS swaps it for credentials that expire after an hour, and there’s nothing to leak or rotate.
Trust policys 2 condition – aud & sub
- Question 2
- Without conditions, any GitHub repo in the world could present a token from GitHub’s OIDC provider and try to use your role.
audchecks the token was meant for AWS.subpins it to your repo, yourmainbranch.
- A fork, a PR branch or someone else’s repo gets denied. IAM itself recommends this: it recommends that users evaluate the IAM condition key, token.actions.githubusercontent.com:sub, in the trust policy of any role that trusts GitHub’s OIDC identity provider (IdP). github
StringEqualsrather thanStringLike:StringLikeallows wildcards likerepo:earpjennings37/*.- That’s handy, but it opens the role to every repo you’ll ever create. An exact match is the safe default.
- Without conditions, any GitHub repo in the world could present a token from GitHub’s OIDC provider and try to use your role.
Permission list (ecr: action)
- Question 3
- Least privilege – CI needs to log in, push layers and read back what it pushed. Nothing else, and only for this one repository.
GetAuthorizationTokenuses"*": some AWS actions can’t be limited to one resource. Logging in to ECR is account-wide by design, so"*"is required there. Every other action is pinned to the repo’s ARN.
ECR repository settings.
- Question 5
scan_on_push = true:- a free second opinion. Trivy scans in CI before the push, and AWS scans again after it.
force_delete = false:- If this module is ever destroyed by accident, AWS refuses to delete a repository that still has images. That’s a deliberate speed bump. (Popeye’s repo has the opposite setting, because it’s meant to be disposable.)
MUTABLEtags:- this one’s a judgment call.
IMMUTABLEsounds safer, but Cosign stores signatures as extra tags, and immutable tags can break re-running a workflow.- The real protection is that Kyverno (Fall #3) will check the signature on the image’s digest, the fingerprint of its exact contents. A moved tag can’t fake that.
- this one’s a judgment call.
Lifecycle policy – untagged images expire after 14 days
- Question 4
- Cost – ECR keeps everything forever unless told otherwise. Every build leaves untagged layers behind, and they’d quietly add up, just like the 33 orphaned EBS volumes did.
Xtrahh Detailzzz
max_session_duration = 3600:- one hour, the shortest standard session. A CI run takes minutes.
aws_iam_policy_documentinstead of hand-written JSON:- Terraform checks its structure, it’s easier to read, and you can reference other resources’ ARNs directly.
- Variables with defaults: works with zero configuration today, but you could point it at another repo or branch later without editing code.
- Local state, no backend: matches your HCP bootstrap. The trade-off is that
terraform.tfstatelives only on your machine, so back it up.
Part B: AWS Infra Side – 4 Files Created
versions.tf
- Sources:
- Terraform Registry: AWS provider:
- the
required_providersandprovider "aws"setup.
- the
- Terraform Registry: AWS provider:
- This is the same version range as your
bootstrap/hcp-terraformmodule, so both bootstraps behave the same. There’s nobackendblock, so the state is stored locally in this folder, just like the HCP bootstrap. - Your
bootstrap/hcp-terraform/versions.tf: the version range was copied from there to keep the two bootstraps consistent.
variables.tf
- Sources:
- Every variable has a default, so you don’t need a
.tfvarsfile.github_repositoryandgithub_branchare the security boundary: they decide who is allowed to push images. - Ensure to add github_owner_id & github_repository_id
main.tf
- Sources:
| Block | Syntax source (Terraform Registry) | Concept source (AWS / GitHub docs) |
|---|---|---|
aws_iam_openid_connect_provider | iam_openid_connect_provider | GitHub Docs: OIDC in AWS and the AWS Security Blog: IAM roles for GitHub Actions |
aws_ecr_repository | ecr_repository | AWS: basic scanning in ECR |
aws_ecr_lifecycle_policy | ecr_lifecycle_policy | AWS: ECR lifecycle policies and examples |
Trust policy (aws_iam_policy_document) | iam_policy_document | GitHub Docs: OIDC in AWS (the aud and sub conditions) |
aws_iam_role | iam_role | AWS Security Blog |
aws_iam_role_policy (ECR push permissions) | iam_role_policy | AWS: ECR lifecycle and repository docs (the push/pull actions) |
MUTABLEtags may look less secure, but trust here comes from the signed digest, not the tag.- Fall #3’s Kyverno policy checks the signature on the exact image content, so retagging can’t sneak anything in.
| Block | What it does | Why it matters |
|---|---|---|
aws_iam_openid_connect_provider | Tells AWS to trust login tokens that GitHub Actions issues | CI gets temporary credentials (1 hour), so there are no access keys to leak |
aws_ecr_repository | The image registry for the app | scan_on_push runs AWS’s own vulnerability scan. force_delete = false means a stray destroy can’t wipe your images |
aws_ecr_lifecycle_policy | Deletes untagged leftovers after 14 days | Keeps storage costs near zero |
Trust policy (github_actions_trust) | Who can use the role | Only a token from repo:earpjennings37/say_when_app:ref:refs/heads/main. A fork, another repo or a PR branch gets “access denied” |
aws_iam_role | The role CI assumes | Its sessions last at most 1 hour |
Permissions policy (ecr_push) | What the role can do | Log in to ECR, plus push and pull this one repository. Nothing else in your account |
- Similar to variables above, add locals to pull values from github_oidc_sub
}Whysplit(): your existinggithub_repositoryvariable already holds"earpjennings37/say_when_app".split("/", ...)breaks it into[0] = "earpjennings37"and[1] = "say_when_app", so there’s still one place to set the repo name. The two new ID variables slot in next to each part.

outputs.tf
- Sources:
- The values come from the attributes listed on the registry pages of –
- Save the two outputs:
github_actions_role_arnandecr_repository_url. - If apply fails with
EntityAlreadyExistson the OIDC provider, your account already trusts GitHub from an earlier experiment. Import it and apply again:
============git checkout main && git pull============cd bootstrap/github-actions============terraform init============terraform plan # expect: 6 to add, 0 to change, 0 to destroy============terraform apply============terraform output============

Part C: AWS Infra Side – Github Variables
- Sources:
- Tell the app repo which role to use. On GitHub, go to
say_when_app→ Settings → Secrets and variables → Actions → Variables tab → New repository variable:- Name:
AWS_ROLE_ARN - Value: the
github_actions_role_arnoutput
- Name:
- Why a variable and not a secret:
- An ARN is just an address, not a credential.
- On its own it grants nothing, because the role only trusts tokens from your repo’s
mainbranch.- Keeping it as a variable also means it’s visible in workflow logs, which makes debugging easier.

Part D: The App Repo patch
- Sources:
- Watch the real run on
main: go to Actions → the run for your merge commit. This time every step runs: push to ECR, SBOM, sign, attest and verify. The run’s Summary shows the image tag and digest. - On the PR the Build
- Scan and sign workflow runs the tests
- ARM64 build
- Trivy scan only.
- Nothing is pushed or signed on a PR.
- Wait for both jobs to go green, then merge.
- The empty
__init__.pyfiles turn the folders into Python packages, sofrom app.main import appworks in the tests.
============mkdir -p app tests .github/workflows============touch app/__init__.py tests/__init__.py============
Isolated Python envmt
============sudo apt install -y python3.10-venv # one-time on Ubuntu/WSL============python3 -m venv .venv============source .venv/bin/activate # every new terminal============which python pip============
Dependencies: ranges, split runtime vs dev
- Pin ranges of split runtime vs dev as dependencies
- Why the ranges?
- (
>=0.115,<1.0): you get bug fixes automatically, but no surprise breaking major versions.
- (
- Why the ranges?
- Sources:
| File | Packages | Ships in the image? |
|---|---|---|
requirements.txt | fastapi, uvicorn, prometheus-client, PyYAML | Yes |
requirements-dev.txt | pytest, httpx | No, tests only |
pip install -r requirements.txt -r requirements-dev.txt
Chores as config, not code
- Sources:
app/chores.yaml:- 10 fall chores,
- each with
expect,rejectandmin_confidence. - A new chore means a YAML edit, not a code change.
- id: rake-leaves name: Rake the leaves expect: [Rake, Leaf] reject: [Basketball, Sports Ball] min_confidence: 80
app/main.py:- exposes 5 endpoints
| Endpoint | Purpose |
|---|---|
/healthz | Kubernetes liveness and readiness probes |
/version | Which build is running (the git SHA, set at build time) |
/chores | All chores from the YAML file |
/chores/{id} | One chore, or a 404 if unknown |
/metrics | Prometheus counter of requests by endpoint |
Testing & Port-Forwarding
- Sources
============pytest -q # 5 passed============python -m uvicorn app.main:app --port 8080 # open http://localhost:8080/docs============http://localhost:8080/docs============
========Expand /chores/{chore_id} → Try it out → enter rake-leaves → Execute. You’ll get the chore back. Then try shovel-snow to see the 404 with “No chore called ‘shovel-snow’”.========

INFO: Started server process [89667]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:8080 (Press CTRL+C to quit)
INFO: 127.0.0.1:37590 - "GET /docs HTTP/1.1" 200 OK
INFO: 127.0.0.1:37590 - "GET /openapi.json HTTP/1.1" 200 OK
INFO: 127.0.0.1:48326 - "GET /chores/rake-leaves HTTP/1.1" 200 OK
INFO: 127.0.0.1:48326 - "GET /chores/rake-leaves HTTP/1.1" 200 OK
INFO: 127.0.0.1:46344 - "GET /chores/shovel-snow HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:46344 - "GET /chores/shovel-snow HTTP/1.1" 404 Not Found
INFO: 127.0.0.1:33238 - "GET /chores/rake-leaves HTTP/1.1" 200 OK
INFO: 127.0.0.1:48234 - "GET /metrics HTTP/1.1" 200 OK
INFO: 127.0.0.1:48234 - "GET /favicon.ico HTTP/1.1" 404 Not Found

========Open http://localhost:8080/metrics. Look for lines like:cleanup_http_requests_total{endpoint="chore"} 2.0========That’s the counter you just bumped, and it’s exactly what Prometheus will scrape in the cluster later.========

Part E: The App Repo – Ignoring $tuff
| File | Keeps out | Why it matters |
|---|---|---|
.gitignore | .venv/, __pycache__/, *.jpg, *.jpeg, *.heic, photos/, sbom*.json | Phone photos carry GPS coordinates in their EXIF data; one stray git add . could publish my home location. .png is allowed on purpose, for blog screenshots. |
.dockerignore | .git, .venv, tests, docs, *.md, photos | A smaller build context builds faster, and less in the image means less for scanners to flag. |
Part F: The App Repo – Docker File-n-Stuff
- Sources:
- Dockerfile reference: FROM, ARG/ENV, USER, EXPOSE, CMD exec form
- Multi-stage builds
- Build cache: ordering layers
- Dockerfile best practices, including the non-root user
- Python official image (slim variants)
- Python command-line env vars:
- Kubernetes: security context and
runAsNonRoot - Uvicorn deployment settings, on
--host 0.0.0.0
Build Dockerfile
- It’s the recipe CI uses to turn your app into the image that runs on the cluster. The image runs as user w/no shell, no pip, & final version after reviewed by Trivy is good to go.
========docker build --build-arg APP_VERSION=local-test -t say-when-app:local .========[+] Building 23.1s (17/17) FINISHED docker:default => [internal] load build definition from Dockerfile 0.1s => => transferring dockerfile: 866B 0.0s => resolve image config for docker-image://docker.io/docker/dockerfile:1 1.2s => [auth] docker/dockerfile:pull token for registry-1.docker.io ========

========docker run --rm -d -p 8080:8080 --name sw say-when-app:local========curl -s localhost:8080/version # → {"version":"local-test"}========curl -s localhost:8080/healthz # → {"status":"ok"}========docker exec sw id # → uid=10001(app) ... NOT root========docker images say-when-app:local # note the SIZE for the blog========docker stop sw========

Port-Forward

========docker run --rm -d -p 8080:8080 --name sw say-when-app:local========http://localhost:8080/docs========

Part G: The App Repo – Github Workflows Build
- Sources
- When it runs
- Tests gate everything
- build & scan
- push to ECR w/no stored aws keys
- sbom, sign, & verify
.github/workflows/build.yaml

========wc -l .github/workflows/build.yaml========python -c "import yaml; yaml.safe_load(open('.github/workflows/build.yaml')); print('YAML OK')"========grep -c '@[0-9a-f]\{40\}' .github/workflows/build.yaml========docker run --rm -v /var/run/docker.sock:/var/run/docker.sock \ aquasec/trivy:latest image \ --severity HIGH,CRITICAL --ignore-unfixed \ say-when-app:local========
Part H: Verify both Infra & App are Werrrrrrkin’

Trivy Gate
- Any fixable HIGH or CRITICAL vulnerability fails the job. Unfixed ones are skipped because nothing can be done about them yet, and they would block every build.
- Sources:
severity: HIGH,CRITICALignore-unfixed: trueexit-code: "1"
OIDC
id-token: writelets the job request a short-lived token from GitHub.- AWS checks it against the role’s trust policy and returns credentials that expire within an hour.
- The role ARN lives in a repository variable,
vars.AWS_ROLE_ARN, because an ARN is not a secret. - The image is tagged with the commit SHA and signed by its digest, because tags can move and digests can’t.
- Sources:
SBOM & Keyless Signing
- Sources:
- The SBOM (Syft, SPDX format) lists every package in the image to find the next big CVE
- Keyless signing = no private key
- Cosign trades the OIDC identity for a short-lived Sigstore certificate and records the signature
- Of course the attestation signs the SBOM too for integrity
cosign verifyproves the image was signed by this workflow, onmain, in this repo. Kyverno will enforce the same check in the cluster in Fall #3.
Final CLI Commands:
======================aws ecr describe-images --repository-name say-when-app \ --query 'imageDetails[].{tag:imageTags[0],pushed:imagePushedAt,bytes:imageSizeInBytes}' \ --output table======================----------------------------------------------------------------------------------------------| DescribeImages |+----------+------------------------------------+--------------------------------------------+| bytes | pushed | tag |+----------+------------------------------------+--------------------------------------------+| 10965 | 2026-11-18T12:54:22.807000-06:00 | None || 3000000 | 2026-11-18T12:54:25.850000-06:00 | None || 50000000| 2026-11-18T12:54:13.037000-06:00 | 6000000000000000000000000000000000000000 |======================
======================curl -sSfL -o cosign https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64chmod +x cosign && sudo mv cosign /usr/local/bin/cosign version====================== ______ ______ _______. __ _______ .__ __. / | / __ \ / || | / _____|| \ | || ,----'| | | | | (----`| | | | __ | \| || | | | | | \ \ | | | | |_ | | . ` || `----.| `--' | .----) | | | | |__| | | |\ | \______| \______/ |_______/ |__| \______| |__| \__|cosign: A tool for Container Signing, Verification and Storage in an OCI registryGitVersion: v3.1.4GitCommit: 0GitTreeState: cleanBuildDate: 2026-10-20T17:33:02ZGoVersion: go1.27.3Compiler: gcPlatform: linux/amd64======================
======================export DOCKER_CONFIG=$(mktemp -d)aws ecr get-login-password | docker login --username AWS --password-stdin "$REGISTRY"cosign verify "$REGISTRY/say-when-app:$TAG" \ --certificate-identity "$ID" --certificate-oidc-issuer "$ISSUER" > /dev/null \ && echo "✅ Signature verified"cosign verify-attestation --type spdxjson "$REGISTRY/say-when-app:$TAG" \ --certificate-identity "$ID" --certificate-oidc-issuer "$ISSUER" > /dev/null \ && echo "✅ SBOM attestation verified"rm -rf "$DOCKER_CONFIG"; unset DOCKER_CONFIG======================Verification for --The following checks were performed on each of these signatures: - The cosign claims were validated - Existence of the claims in the transparency log was verified offline - The code-signing certificate was verified using trusted certificate authority certificates✅ SBOM attestation verified