100 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
micqdf	612435c42c	fix: Add Hetzner LB health check config to Traefik	2026-03-26 03:11:10 +00:00
micqdf	ac42f671a2	fix: Remove addon-traefik-config dependency from flux-ui	2026-03-26 03:05:58 +00:00
micqdf	dbe7ec0468	fix: Remove expose boolean from traefik ports config	2026-03-26 03:01:13 +00:00
micqdf	816ac8b3c0	fix: Use official Traefik helm repo instead of rancher-stable	2026-03-26 02:59:00 +00:00
micqdf	6f7998639f	fix: Use standard kustomize API in traefik addon	2026-03-26 02:56:52 +00:00
micqdf	7a14f89ad1	fix: Correct traefik kustomization path and sourceRef	2026-03-26 02:55:37 +00:00
micqdf	786901c5d7	fix: Correct traefik kustomization reference (directory not file)	2026-03-26 02:54:29 +00:00
micqdf	46f3d1130b	feat: Add Flux-managed Traefik HelmRelease with Hetzner LB config	2026-03-26 02:52:49 +00:00
micqdf	2fe5a626d4	fix: Add Hetzner network zone annotation to Traefik LoadBalancer	2026-03-26 02:30:43 +00:00
micqdf	2ef68c8087	fix: Remove deprecated enablePodMonitor field in CNP Cluster	2026-03-26 01:01:53 +00:00
micqdf	e2cae18f5f	fix: Remove backup config for initial deployment - add backup after DB is running	2026-03-26 00:46:50 +00:00
micqdf	e0c1e41ee9	fix: Remove bootstrap recovery - create fresh DB (recovery only needed after first backup)	2026-03-26 00:43:49 +00:00
micqdf	63533de901	fix: Fix retentionPolicy format (14d not keep14)	2026-03-26 00:41:44 +00:00
micqdf	1b39710f63	fix: Move retentionPolicy to correct location in backup spec	2026-03-26 00:39:25 +00:00
micqdf	8c034323dc	fix: Fix Cluster CR with correct barmanObjectStore schema	2026-03-26 00:35:23 +00:00
micqdf	5fa2b411ee	fix: Fix Cluster CR schema - use barmanObjectStore instead of b2	2026-03-26 00:33:04 +00:00
micqdf	3ea28e525f	fix: Fix CNP operator image repository (cloudnative-pg not postgresql)	2026-03-26 00:21:09 +00:00
micqdf	4b95ba113d	fix: Remove LPP helm (already installed by k3s), fix CNP chart version to 0.27.1	2026-03-26 00:13:22 +00:00
micqdf	13627bf81f	fix: Split CNP operator from CNP cluster to fix CRD dependency ... - Move CNP operator HelmRelease to cnpg-operator folder - Create addon-cnpg-operator kustomization (deploys operator first) - Update addon-cnpg to dependOn addon-cnpg-operator - Add addon-cnpg as dependency for addon-rancher (needs database)	2026-03-26 00:06:34 +00:00
micqdf	ef3fb2489a	fix: Convert kustomization-lpp and kustomization-cnpg to Flux Kustomization CRs	2026-03-26 00:03:53 +00:00
micqdf	7097495d72	fix: Add missing metadata.name to kustomization-lpp and kustomization-cnpg	2026-03-25 23:39:45 +00:00
micqdf	9d601dc77c	feat: Add CloudNativePG with B2 backups for persistent Rancher database ... - Add Local Path Provisioner for storage - Add CloudNativePG operator (v1.27.0) via Flux - Create PostgreSQL cluster with B2 (Backblaze) auto-backup/restore - Update Rancher to use external PostgreSQL via CATTLE_DB_CATTLE_* env vars - Add weekly pg_dump CronJob to B2 (Sundays 2AM) - Add pre-destroy backup hook to destroy workflow - Add B2 credentials to Doppler (B2_ACCOUNT_ID, B2_APPLICATION_KEY) - Generate RANCHER_DB_PASSWORD in Doppler Backup location: HetznerTerra/rancher-backups/ Retention: 14 backups	2026-03-25 23:06:45 +00:00
micqdf	f36445d99a	Fix CNI: configure flannel to use private network interface (enp7s0) instead of public	2026-03-25 01:44:33 +00:00
micqdf	89c2c99963	Fix Rancher: remove conflicting LoadBalancer, add HTTPS port-forward, use tailscale serve only	2026-03-25 00:59:16 +00:00
micqdf	4a35cfb549	Fix Rancher: use correct targetPort 444 for HTTPS	2026-03-24 23:30:58 +00:00
micqdf	3d50bfc534	Fix Rancher service selector: use cattle-system-rancher label	2026-03-24 23:25:36 +00:00
micqdf	ab2f287bfb	Fix Rancher: use correct service name cattle-system-rancher	2026-03-24 22:30:49 +00:00
micqdf	dcb2675b67	Upgrade Rancher to 2.13.3 for K8s 1.34 compatibility	2026-03-24 21:42:51 +00:00
micqdf	b40bec7e0e	Fix Rancher: use Doppler secret instead of hardcoded password	2026-03-24 21:13:23 +00:00
micqdf	efe0c0cfd5	Fix Rancher: upgrade to 2.10.3 for K8s 1.34 compatibility	2026-03-24 20:29:38 +00:00
micqdf	c61d9f9c1d	Remove traefik-config dependency from Rancher	2026-03-24 20:02:08 +00:00
micqdf	60ceac4624	Fix Rancher access: add kubectl port-forward + tailscale serve setup	2026-03-24 20:01:57 +00:00
micqdf	47b384a337	Fix Rancher access: add Tailscale service for Traefik with port 9442, fix deployment order	2026-03-24 19:40:37 +00:00
micqdf	ecf17113fb	Fix Rancher deployment: add cattle-system namespace, fix Traefik config with port 9442	2026-03-24 19:09:28 +00:00
micqdf	4ffbcfa312	Add Rancher management UI	2026-03-24 01:53:04 +00:00
micqdf	8745bcda47	Fix Weave GitOps image tag - remove invalid v0.41.0 ... The version v0.41.0 doesn't exist in the registry. Removing explicit image tag to let the chart use its default compatible version.	2026-03-24 01:39:48 +00:00
micqdf	e47ec2a3e7	Update Weave GitOps to v0.41.0 to support HelmRelease v2 API ... Fixes error: 'no matches for kind HelmRelease in version v2beta1' The cluster uses HelmRelease v2 API but Weave GitOps v0.38.0 was looking for the old v2beta1 API. Updated image tag to v0.41.0 which supports the newer API version.	2026-03-24 01:33:10 +00:00
micqdf	45c899d2bd	Configure Weave GitOps to use Doppler-managed admin credentials ... Changes: - Enable adminUser creation but disable Helm-managed secret - Use ExternalSecret (cluster-user-auth) from Doppler instead - Doppler secrets: WEAVE_GITOPS_ADMIN_USERNAME and WEAVE_GITOPS_ADMIN_PASSWORD_BCRYPT_HASH - Added cluster-user-auth to viewSecretsResourceNames for RBAC Login credentials are now managed via Doppler and External Secrets Operator.	2026-03-24 01:01:30 +00:00
micqdf	0e52d8f159	Use Tailscale DNS names instead of IPs for TLS SANs ... Changed from hardcoded Tailscale IPs to DNS names: - k8s-cluster-cp-1.silverside-gopher.ts.net - k8s-cluster-cp-2.silverside-gopher.ts.net - k8s-cluster-cp-3.silverside-gopher.ts.net This is more robust since Tailscale IPs change on rebuild, but DNS names remain consistent. After next rebuild, cluster accessible via: - kubectl --server=https://k8s-cluster-cp-1.silverside-gopher.ts.net:6443	2026-03-23 23:50:48 +00:00
micqdf	4726db2b5b	Add Tailscale IPs to k3s TLS SANs for secure tailnet access ... Changes: - Add tailscale_control_plane_ips list to k3s-server defaults - Include all 3 control plane Tailscale IPs (100.120.55.97, 100.108.90.123, 100.92.149.85) - Update primary k3s install to add Tailscale IPs to TLS certificates - Enables kubectl access via Tailscale without certificate errors After next deploy, cluster will be accessible via: - kubectl --server=https://100.120.55.97:6443 (or any CP tailscale IP) - kubectl --server=https://k8s-cluster-cp-1:6443 (via tailscale DNS)	2026-03-23 23:04:00 +00:00
micqdf	90d105e5ea	Fix kube_api_endpoint variable passing for HA cluster ... - Remove circular variable reference in site.yml - Add kube_api_endpoint default to k3s-server role - Variable is set via inventory group_vars and passed to role - Primary CP now correctly adds LB IP to TLS SANs Note: Existing cluster needs destroy/rebuild to regenerate certificates.	2026-03-23 03:01:53 +00:00
micqdf	952a80a742	Fix HA cluster join via Load Balancer private IP ... Changes: - Use LB private IP (10.0.1.5) instead of public IP for cluster joins - Add LB private IP to k3s TLS SANs on primary control plane - This allows secondary CPs and workers to verify certificates when joining via LB Fixes x509 certificate validation error when joining via LB public IP.	2026-03-23 02:56:41 +00:00
micqdf	4965017b86	Fix Load Balancer network attachment ... Add hcloud_load_balancer_network resource to attach LB to private network. This is required before targets can use use_private_ip=true. LB gets IP 10.0.1.5 on the private network.	2026-03-23 02:44:35 +00:00
micqdf	b2b9c38b91	Fix Load Balancer output attribute - use ipv4 instead of ipv4_address	2026-03-23 02:40:50 +00:00
micqdf	ff31cb4e74	Implement HA control plane with Load Balancer (3-3 topology) ... Major changes: - Terraform: Scale to 3 control planes (cx23) + 3 workers (cx33) - Terraform: Add Hetzner Load Balancer (lb11) for Kubernetes API - Terraform: Add kube_api_lb_ip output - Ansible: Add community.network collection to requirements - Ansible: Update inventory to include LB endpoint - Ansible: Configure secondary CPs and workers to join via LB - Ansible: Add k3s_join_endpoint variable for HA joins - Workflow: Add imports for cp-2, cp-3, and worker-3 - Docs: Update STABLE_BASELINE.md with HA topology and phase gates Topology: - 3 control planes (cx23 - 2 vCPU, 8GB RAM each) - 3 workers (cx33 - 4 vCPU, 16GB RAM each) - 1 Load Balancer (lb11) routing to all 3 control planes on port 6443 - Workers and secondary CPs join via LB endpoint for HA Cost impact: +~€26/month (2 extra CPs + 1 extra worker + LB)	2026-03-23 02:39:39 +00:00
micqdf	8b4a445b37	Update STABLE_BASELINE.md - CCM/CSI integration achieved ... Document the successful completion of Hetzner CCM and CSI integration: - CCM deployed via Ansible before workers join (fixes uninitialized taint) - CSI provides hcloud-volumes StorageClass for persistent storage - Two consecutive rebuilds passed all phase gates - PVC provisioning tested and working Platform now has full cloud provider integration with persistent volumes.	2026-03-23 02:25:00 +00:00
micqdf	e447795395	Install helm binary in ccm-deploy role before using it ... The kubernetes.core.helm module requires helm CLI to be installed on the target node. Added check and install step using the official helm install script.	2026-03-23 00:07:39 +00:00
micqdf	31b82c9371	Deploy CCM via Ansible before workers join to fix external cloud provider ... This fixes the chicken-and-egg problem where workers with --kubelet-arg=cloud-provider=external couldn't join because CCM wasn't running yet to remove the node.cloudprovider.kubernetes.io/uninitialized taint. Changes: - Create ansible/roles/ccm-deploy/ to deploy CCM via Helm during Ansible phase - Reorder site.yml: CCM deploys after secrets but before workers join - CCM runs on control_plane[0] with proper tolerations for control plane nodes - Add 10s pause after CCM ready to ensure it can process new nodes - Workers can now successfully join with external cloud provider enabled Flux still manages CCM for updates, but initial install happens in Ansible.	2026-03-22 23:58:03 +00:00
micqdf	cadfedacf1	Fix providerID health check - use shell module for piped grep	2026-03-22 22:55:55 +00:00
micqdf	561cd67b0c	Enable Hetzner CCM and CSI for cloud provider integration ... - Enable --kubelet-arg=cloud-provider=external on all nodes (control planes and workers) - Activate CCM Kustomization with 10m timeout for Hetzner cloud-controller-manager - Activate CSI Kustomization with dependsOn CCM and 10m timeout for hcloud-csi - Update deploy workflow to wait for CCM/CSI readiness (600s timeout) - Add providerID verification to post-deploy health checks This enables proper cloud provider integration with Hetzner CCM for node labeling and Hetzner CSI for persistent volume provisioning.	2026-03-22 22:26:21 +00:00
micqdf	4eebbca648	docs: update README for deferred observability baseline	2026-03-22 01:04:53 +00:00
micqdf	7b5d794dfc	fix: update health checks for deferred observability	2026-03-22 01:04:27 +00:00
micqdf	8643bbfc12	fix: defer observability to get clean baseline	2026-03-22 01:03:55 +00:00
micqdf	84f446c2e6	fix: restore observability timeouts to 5 minutes	2026-03-22 00:43:37 +00:00
micqdf	d446e86ece	fix: use static grafana password, remove externalsecret dependency	2026-03-22 00:43:21 +00:00
micqdf	90c7f565e0	fix: remove tailscale ingress dependencies from observability	2026-03-22 00:42:35 +00:00
micqdf	989848fa89	fix: increase observability timeouts to 10 minutes	2026-03-21 19:34:43 +00:00
micqdf	56e5807474	fix: create doppler ClusterSecretStore after ESO is installed	2026-03-21 19:19:43 +00:00
micqdf	df0511148c	fix: unsuspend tailscale operator for stable baseline	2026-03-21 19:03:39 +00:00
micqdf	894e6275b1	docs: update stable baseline to defer ccm/csi	2026-03-21 18:41:36 +00:00
micqdf	a01cf435d4	fix: skip ccm/csi waits for stable baseline - using k3s embedded	2026-03-21 18:40:53 +00:00
micqdf	84f77c4a68	fix: use kubectl patch instead of apply for flux controller nodeSelector	2026-03-21 18:05:41 +00:00
micqdf	2e4196688c	fix: bootstrap flux in phases - crds first, then resources	2026-03-21 17:42:39 +00:00
micqdf	8d1f9f4944	fix: add k3s reset logic for primary control plane	2026-03-21 16:10:17 +00:00
micqdf	d4fd43e2f5	refactor: simplify k3s-server bootstrap for	2026-03-21 15:48:33 +00:00
micqdf	48a80c362c	fix: disable external cloud-provider kubelet arg for stable baseline	2026-03-21 14:36:54 +00:00
micqdf	fcf7f139ff	fix: use public api endpoint for flux bootstrap	2026-03-21 00:07:51 +00:00
micqdf	7139ae322d	fix: bootstrap flux during cluster deploy	2026-03-20 10:37:11 +00:00
micqdf	528a8dc210	fix: defer doppler store until eso is installed	2026-03-20 09:30:17 +00:00
micqdf	349f75729a	fix: bootstrap tailscale namespace before secret	2026-03-20 09:24:35 +00:00
micqdf	522626a52b	refactor: simplify stable cluster baseline	2026-03-20 02:24:37 +00:00
micqdf	5bd4c41c2d	fix: restore k3s agent bootstrap	2026-03-20 01:50:16 +00:00
micqdf	3e41f71b1b	fix: harden terraform destroy workflow	2026-03-19 23:26:03 +00:00
micqdf	9d2f30de32	fix: prepare k3s for external cloud provider	2026-03-17 01:21:23 +00:00
micqdf	08a3031276	refactor: retire imperative addon roles	2026-03-17 01:04:02 +00:00
micqdf	e3ce91db62	fix: align flux ccm with live deployment	2026-03-11 18:17:16 +00:00
micqdf	bed8e4afc8	feat: migrate core addons toward flux	2026-03-11 17:43:35 +00:00
micqdf	2d4de6cff8	fix: bootstrap doppler store outside flux	2026-03-09 02:58:26 +00:00
micqdf	4a83d981c8	fix: skip dry-run validation for doppler store sync	2026-03-09 02:52:08 +00:00
micqdf	d188a51ef6	fix: move doppler store manifests out of ignored path	2026-03-09 02:45:46 +00:00
micqdf	646ef16258	fix: stabilize flux and external secrets reconciliation	2026-03-09 02:25:27 +00:00
micqdf	6f2e056b98	feat: sync runtime secrets from doppler	2026-03-09 00:25:41 +00:00
micqdf	e10a70475f	fix: right-size flux observability workloads	2026-03-08 05:17:22 +00:00
micqdf	f95e0051a5	feat: automate private tailnet access on cp1	2026-03-08 04:16:06 +00:00
micqdf	7c15ac5846	feat: add flux ui on shared tailscale endpoint	2026-03-07 12:30:17 +00:00
micqdf	4c104f74e8	feat: route observability through one tailscale endpoint	2026-03-07 01:04:03 +00:00
micqdf	be04602bfb	fix: make flux bootstrap reachable from cluster	2026-03-07 00:38:29 +00:00
micqdf	06c1356f1e	feat: expose flux observability services over tailscale	2026-03-05 00:43:29 +00:00
micqdf	86fb5d5b90	fix: move observability gitops gating to role level	2026-03-05 00:17:25 +00:00
micqdf	8b403cd1d6	feat: migrate observability stack to flux gitops	2026-03-04 23:38:40 +00:00
micqdf	480a079dc8	fix: fail fast when loki datasource has no labels	2026-03-04 21:00:01 +00:00
micqdf	ff8e32daf5	fix: add loki nodeport fallback for grafana datasource reachability	2026-03-04 19:39:16 +00:00
micqdf	eb1ad0bea7	fix: make grafana prometheus datasource resilient with nodeport fallback	2026-03-04 19:22:31 +00:00
micqdf	9ff9d1e633	fix: clear stale helm pending revisions before kube-prometheus upgrade	2026-03-04 18:35:55 +00:00
micqdf	6177b581e4	fix: correct dashboard verification checks and retry helm upgrade lock	2026-03-04 08:48:30 +00:00
micqdf	b1e21c4a4b	fix: speed up dashboards workflow firewall apply and set TF_VAR env	2026-03-04 03:54:56 +00:00
micqdf	2f166ed9e7	feat: manage grafana content as code with fast dashboard workflow	2026-03-04 03:36:01 +00:00
micqdf	1c39274df7	feat: stabilize tailscale observability exposure with declarative proxy class	2026-03-04 01:37:00 +00:00
micqdf	28eaa36ec4	fix: use tag:k8s for tailscale operator default tags	2026-03-04 00:57:33 +00:00
micqdf	02fa71c0aa	fix: use tag:k8 for tailscale operator default tag	2026-03-04 00:27:47 +00:00