Case study · Cloud infrastructure
Production-Ready EKS Cluster with GitOps
A Kubernetes platform that runs the same GitOps stack on a laptop (Minikube) and on AWS EKS, with Argo CD, Vault, Prometheus and Grafana.
How it fits together
- Git
- argo-apps/
- helm-charts/
- terraform/
- CI (GitHub Actions)
- validate
- terraform plan
- security scan
- daily drift check
- Argo CD
- App of apps
- automated sync
- Cluster
- Minikube or EKS (3 AZs)
- Vault
- Prometheus + Grafana
- web app
The problem
Teams want a production-like Kubernetes platform to learn and test on, but a real EKS cluster costs money and takes about an hour to build. The goal was one Git repository that brings up the same stack locally in about 25 minutes and on AWS when it is needed.
What I built
- Terraform modules for a VPC across 3 Availability Zones (public and private subnets, one NAT gateway per zone), an EKS cluster with managed node group and add-ons, IAM roles for service accounts (OIDC), KMS keys and VPC flow logs.
- Argo CD applications for the web app, Prometheus, Grafana and Vault, so the cluster state comes from Git.
- Two Vault modes: single replica with manual unseal on Minikube, and HA with Raft storage and AWS KMS auto-unseal on EKS.
- Pod Security Standards, network policies with explicit allow rules, and least-privilege RBAC.
- 14 GitHub Actions workflows, including Terraform plan and apply, image scanning, secret scanning and a daily drift check at 03:00 UTC.
Decisions
- Local first: Minikube is the default path and AWS is an override (values-minikube.yaml first). Most work happens on a laptop, so the docs are about 80% local.
- Upstream Helm charts for Prometheus, Grafana and Vault with values files in Git, and a custom chart only for the app.
- The Argo CD 3.1 upgrade shipped with a preflight script and a one-command rollback script that backs up Applications and Projects first.
What went wrong: The Prometheus CRD that was too big
Syncing the Prometheus stack failed with "metadata.annotations: Too long: may not be more than 262144 bytes". Kubernetes limits annotations to 256 KB, and large inline Helm values in the Argo CD Application pushed the CRD over it. The fix was to move the values into external values files in Git. It is now in the troubleshooting guide.
What I would improve next
- docs/architecture.md still describes an older layout (environments/, applications/). Update it to match argo-apps/ and helm-charts/.
- Add the Infracost check that cost_monitoring.tf mentions. The workflows do not run it yet.