Skip to content

Live Activation Runbook

This runbook turns the repository from a statically validated reference into a real non-production AWS/EKS environment. It is intentionally separate from local CI because activation changes an AWS account and creates billable resources.

0. Cost and account boundary

Before any apply, confirm the intended AWS account, region, budget owner and teardown window. The dev design can create cost from:

  • the EKS control plane;
  • a NAT gateway and its public IPv4/data processing;
  • managed EC2 worker nodes;
  • EBS volumes and snapshots;
  • load balancers or public IPv4 addresses if enabled later;
  • CloudWatch/data-transfer usage generated by the environment.

Do not treat the OpenCost node run-rate alert as the AWS bill. It measures Kubernetes/node economics, not every AWS service charge.

1. Preflight

./scripts/preflight.sh

The script is read-only. It resolves the current AWS identity when available and prints the cost-sensitive components. A failing identity check is a hard stop.

The repository currently targets EKS Kubernetes 1.36 with STANDARD support policy. Re-check the AWS EKS support calendar immediately before provisioning; do not deploy an extended-support version merely because an old example still references it.

2. Bootstrap state and GitHub OIDC

Bootstrap uses the approved operator session only for the account-level primitives:

cp bootstrap/terraform.tfvars.example bootstrap/terraform.tfvars
terraform -chdir=bootstrap init
terraform -chdir=bootstrap plan -out=bootstrap.plan
terraform -chdir=bootstrap show bootstrap.plan
terraform -chdir=bootstrap apply bootstrap.plan

Then configure the repository variables/environments from the bootstrap outputs:

./scripts/configure-github-repo.sh

No long-lived AWS access key should be added to GitHub.

3. Dev infrastructure plan

Render the dev backend and inspect the plan before apply:

./scripts/render-backend-config.sh dev
terraform -chdir=infra init -reconfigure -backend-config=environments/dev.backend.hcl
terraform -chdir=infra plan -var-file=environments/dev.tfvars.example -out=dev.plan
terraform -chdir=infra show dev.plan

Expected high-cost items deserve explicit review: NAT gateway, EKS cluster, managed node group and any additional public endpoints.

4. Private EKS API access

The default cluster endpoint is private. A GitHub-hosted runner or laptop on the public internet cannot administer that API without a path into the VPC.

Use one approved access pattern before bootstrapping Kubernetes controllers:

  1. a self-hosted runner inside the VPC;
  2. VPN/Tailscale/SSM-based administrative connectivity into the VPC; or
  3. a temporary, tightly CIDR-restricted public EKS endpoint only for bootstrap, followed by immediate reversion to private-only access.

Do not expose Argo CD, Grafana, Prometheus or Kubernetes administrative services publicly just to simplify bootstrap.

5. Argo CD bootstrap

The helper is dry-run by default:

./scripts/bootstrap-argocd.sh

After confirming the kubectl context points to the approved dev cluster:

EXECUTE=1 ./scripts/bootstrap-argocd.sh

Monitoring-scoped resources and workload-scoped resources are kept in separate Argo Applications so the destination namespace matches the manifests under each application path.

6. Runtime evidence checklist

Do not mark the live milestone complete until evidence exists for all applicable controls:

  • GitHub OIDC plan/apply role assumption;
  • encrypted remote state + lock behavior;
  • Argo reconciliation healthy;
  • signed image admitted and an invalid image rejected;
  • Kyverno policy tests confirmed in-cluster;
  • Trivy Operator reports generated;
  • Falco events observed from a benign test;
  • Prometheus target up and SLO rules loaded;
  • controlled game-day alert fires and routes correctly;
  • Velero backup/restore smoke test succeeds;
  • OpenCost returns real allocation data;
  • VPA recommendation report is captured;
  • teardown or scale-down path is documented.

Record links/screenshots in Evidence. Never publish account IDs, private endpoints, credentials or internal network details.

7. Teardown

For an ephemeral dev exercise, destroy workload/infrastructure only after exporting the evidence required for the portfolio and verifying any backup artifacts that should be retained.

terraform -chdir=infra plan -destroy -var-file=environments/dev.tfvars.example
terraform -chdir=infra destroy -var-file=environments/dev.tfvars.example

The bootstrap state/KMS/OIDC layer has a separate lifecycle. Do not destroy it casually if it protects Terraform history or is shared by later dev rebuilds.