171 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
MichaelFisher1997	4965017b86	Fix Load Balancer network attachment ... Add hcloud_load_balancer_network resource to attach LB to private network. This is required before targets can use use_private_ip=true. LB gets IP 10.0.1.5 on the private network.	2026-03-23 02:44:35 +00:00
MichaelFisher1997	b2b9c38b91	Fix Load Balancer output attribute - use ipv4 instead of ipv4_address	2026-03-23 02:40:50 +00:00
MichaelFisher1997	ff31cb4e74	Implement HA control plane with Load Balancer (3-3 topology) ... Major changes: - Terraform: Scale to 3 control planes (cx23) + 3 workers (cx33) - Terraform: Add Hetzner Load Balancer (lb11) for Kubernetes API - Terraform: Add kube_api_lb_ip output - Ansible: Add community.network collection to requirements - Ansible: Update inventory to include LB endpoint - Ansible: Configure secondary CPs and workers to join via LB - Ansible: Add k3s_join_endpoint variable for HA joins - Workflow: Add imports for cp-2, cp-3, and worker-3 - Docs: Update STABLE_BASELINE.md with HA topology and phase gates Topology: - 3 control planes (cx23 - 2 vCPU, 8GB RAM each) - 3 workers (cx33 - 4 vCPU, 16GB RAM each) - 1 Load Balancer (lb11) routing to all 3 control planes on port 6443 - Workers and secondary CPs join via LB endpoint for HA Cost impact: +~€26/month (2 extra CPs + 1 extra worker + LB)	2026-03-23 02:39:39 +00:00
MichaelFisher1997	8b4a445b37	Update STABLE_BASELINE.md - CCM/CSI integration achieved ... Document the successful completion of Hetzner CCM and CSI integration: - CCM deployed via Ansible before workers join (fixes uninitialized taint) - CSI provides hcloud-volumes StorageClass for persistent storage - Two consecutive rebuilds passed all phase gates - PVC provisioning tested and working Platform now has full cloud provider integration with persistent volumes.	2026-03-23 02:25:00 +00:00
MichaelFisher1997	e447795395	Install helm binary in ccm-deploy role before using it ... The kubernetes.core.helm module requires helm CLI to be installed on the target node. Added check and install step using the official helm install script.	2026-03-23 00:07:39 +00:00
MichaelFisher1997	31b82c9371	Deploy CCM via Ansible before workers join to fix external cloud provider ... This fixes the chicken-and-egg problem where workers with --kubelet-arg=cloud-provider=external couldn't join because CCM wasn't running yet to remove the node.cloudprovider.kubernetes.io/uninitialized taint. Changes: - Create ansible/roles/ccm-deploy/ to deploy CCM via Helm during Ansible phase - Reorder site.yml: CCM deploys after secrets but before workers join - CCM runs on control_plane[0] with proper tolerations for control plane nodes - Add 10s pause after CCM ready to ensure it can process new nodes - Workers can now successfully join with external cloud provider enabled Flux still manages CCM for updates, but initial install happens in Ansible.	2026-03-22 23:58:03 +00:00
MichaelFisher1997	cadfedacf1	Fix providerID health check - use shell module for piped grep	2026-03-22 22:55:55 +00:00
MichaelFisher1997	561cd67b0c	Enable Hetzner CCM and CSI for cloud provider integration ... - Enable --kubelet-arg=cloud-provider=external on all nodes (control planes and workers) - Activate CCM Kustomization with 10m timeout for Hetzner cloud-controller-manager - Activate CSI Kustomization with dependsOn CCM and 10m timeout for hcloud-csi - Update deploy workflow to wait for CCM/CSI readiness (600s timeout) - Add providerID verification to post-deploy health checks This enables proper cloud provider integration with Hetzner CCM for node labeling and Hetzner CSI for persistent volume provisioning.	2026-03-22 22:26:21 +00:00
MichaelFisher1997	4eebbca648	docs: update README for deferred observability baseline	2026-03-22 01:04:53 +00:00
MichaelFisher1997	7b5d794dfc	fix: update health checks for deferred observability	2026-03-22 01:04:27 +00:00
MichaelFisher1997	8643bbfc12	fix: defer observability to get clean baseline	2026-03-22 01:03:55 +00:00
MichaelFisher1997	84f446c2e6	fix: restore observability timeouts to 5 minutes	2026-03-22 00:43:37 +00:00
MichaelFisher1997	d446e86ece	fix: use static grafana password, remove externalsecret dependency	2026-03-22 00:43:21 +00:00
MichaelFisher1997	90c7f565e0	fix: remove tailscale ingress dependencies from observability	2026-03-22 00:42:35 +00:00
MichaelFisher1997	989848fa89	fix: increase observability timeouts to 10 minutes	2026-03-21 19:34:43 +00:00
MichaelFisher1997	56e5807474	fix: create doppler ClusterSecretStore after ESO is installed	2026-03-21 19:19:43 +00:00
MichaelFisher1997	df0511148c	fix: unsuspend tailscale operator for stable baseline	2026-03-21 19:03:39 +00:00
MichaelFisher1997	894e6275b1	docs: update stable baseline to defer ccm/csi	2026-03-21 18:41:36 +00:00
MichaelFisher1997	a01cf435d4	fix: skip ccm/csi waits for stable baseline - using k3s embedded	2026-03-21 18:40:53 +00:00
MichaelFisher1997	84f77c4a68	fix: use kubectl patch instead of apply for flux controller nodeSelector	2026-03-21 18:05:41 +00:00
MichaelFisher1997	2e4196688c	fix: bootstrap flux in phases - crds first, then resources	2026-03-21 17:42:39 +00:00
MichaelFisher1997	8d1f9f4944	fix: add k3s reset logic for primary control plane	2026-03-21 16:10:17 +00:00
MichaelFisher1997	d4fd43e2f5	refactor: simplify k3s-server bootstrap for	2026-03-21 15:48:33 +00:00
MichaelFisher1997	48a80c362c	fix: disable external cloud-provider kubelet arg for stable baseline	2026-03-21 14:36:54 +00:00
MichaelFisher1997	fcf7f139ff	fix: use public api endpoint for flux bootstrap	2026-03-21 00:07:51 +00:00
MichaelFisher1997	7139ae322d	fix: bootstrap flux during cluster deploy	2026-03-20 10:37:11 +00:00
MichaelFisher1997	528a8dc210	fix: defer doppler store until eso is installed	2026-03-20 09:30:17 +00:00
MichaelFisher1997	349f75729a	fix: bootstrap tailscale namespace before secret	2026-03-20 09:24:35 +00:00
MichaelFisher1997	522626a52b	refactor: simplify stable cluster baseline	2026-03-20 02:24:37 +00:00
MichaelFisher1997	5bd4c41c2d	fix: restore k3s agent bootstrap	2026-03-20 01:50:16 +00:00
MichaelFisher1997	3e41f71b1b	fix: harden terraform destroy workflow	2026-03-19 23:26:03 +00:00
MichaelFisher1997	9d2f30de32	fix: prepare k3s for external cloud provider	2026-03-17 01:21:23 +00:00
MichaelFisher1997	08a3031276	refactor: retire imperative addon roles	2026-03-17 01:04:02 +00:00
MichaelFisher1997	e3ce91db62	fix: align flux ccm with live deployment	2026-03-11 18:17:16 +00:00
MichaelFisher1997	bed8e4afc8	feat: migrate core addons toward flux	2026-03-11 17:43:35 +00:00
MichaelFisher1997	2d4de6cff8	fix: bootstrap doppler store outside flux	2026-03-09 02:58:26 +00:00
MichaelFisher1997	4a83d981c8	fix: skip dry-run validation for doppler store sync	2026-03-09 02:52:08 +00:00
MichaelFisher1997	d188a51ef6	fix: move doppler store manifests out of ignored path	2026-03-09 02:45:46 +00:00
MichaelFisher1997	646ef16258	fix: stabilize flux and external secrets reconciliation	2026-03-09 02:25:27 +00:00
MichaelFisher1997	6f2e056b98	feat: sync runtime secrets from doppler	2026-03-09 00:25:41 +00:00
MichaelFisher1997	e10a70475f	fix: right-size flux observability workloads	2026-03-08 05:17:22 +00:00
MichaelFisher1997	f95e0051a5	feat: automate private tailnet access on cp1	2026-03-08 04:16:06 +00:00
MichaelFisher1997	7c15ac5846	feat: add flux ui on shared tailscale endpoint	2026-03-07 12:30:17 +00:00
MichaelFisher1997	4c104f74e8	feat: route observability through one tailscale endpoint	2026-03-07 01:04:03 +00:00
MichaelFisher1997	be04602bfb	fix: make flux bootstrap reachable from cluster	2026-03-07 00:38:29 +00:00
MichaelFisher1997	06c1356f1e	feat: expose flux observability services over tailscale	2026-03-05 00:43:29 +00:00
MichaelFisher1997	86fb5d5b90	fix: move observability gitops gating to role level	2026-03-05 00:17:25 +00:00
MichaelFisher1997	8b403cd1d6	feat: migrate observability stack to flux gitops	2026-03-04 23:38:40 +00:00
MichaelFisher1997	480a079dc8	fix: fail fast when loki datasource has no labels	2026-03-04 21:00:01 +00:00
MichaelFisher1997	ff8e32daf5	fix: add loki nodeport fallback for grafana datasource reachability	2026-03-04 19:39:16 +00:00