100 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
micqdf	89e53d9ec9	fix: Handle restricted B2 keys and safe JSON parsing in restore step	2026-03-31 01:43:04 +00:00
micqdf	5a2551f40a	fix: Fix flux CLI download URL - use correct GitHub URL with v prefix on version	2026-03-30 03:11:40 +00:00
micqdf	8c7b62c024	feat: Automate Rancher backup restore in CI pipeline ... - Wait for Rancher and rancher-backup operator to be ready - Patch default SA in cattle-resources-system (fixes post-install hook failure) - Clean up failed patch-sa jobs - Force reconcile rancher-backup HelmRelease - Find latest backup from B2 using Backblaze API - Create Restore CR to restore Rancher state from latest backup - Wait for restore to complete before continuing	2026-03-30 01:56:29 +00:00
micqdf	a1f07f863a	docs: Update restore template with real Backup CR format ... Include actual restore CR spec and note the latest backup filename for reference.	2026-03-30 00:09:53 +00:00
micqdf	2c3a49c2e0	fix: Rename B2 secret keys to match rancher-backup operator expectations ... The operator expects accessKey/secretKey, not aws_access_key_id/aws_secret_access_key.	2026-03-30 00:05:13 +00:00
micqdf	a7ce3dcc1a	fix: Remove s3 block from rancher-backup HelmRelease values ... The S3 config caused the operator to try downloading kubectl, which fails in the container. S3 credentials are correctly configured in the Backup CR and ExternalSecret instead.	2026-03-29 23:47:21 +00:00
micqdf	0ab9418458	fix: Re-add HTTPS port to Tailscale LB for Rancher ... Rancher now manages its own TLS (no longer tls:external), so it serves HTTPS on port 443. The Tailscale LoadBalancer needs to expose both HTTP (80) and HTTPS (443) targeting the corresponding container ports.	2026-03-29 23:04:49 +00:00
micqdf	c251672618	fix: Configure S3 bucketName for rancher-backup operator	2026-03-29 23:01:18 +00:00
micqdf	89364e8f37	fix: Add dependsOn for rancher-backup operator to wait for CRDs	2026-03-29 22:57:22 +00:00
micqdf	20d7a6f777	fix: Install rancher-backup CRD chart before operator ... The rancher-backup operator requires CRDs from the rancher-backup-crd chart to be installed first.	2026-03-29 22:51:34 +00:00
micqdf	22ce5fd6f4	feat: Add cert-manager as dependency for Rancher ... Rancher requires cert-manager when managing its own TLS (not tls:external). Added cert-manager HelmRelease with CRDs enabled.	2026-03-29 22:36:30 +00:00
micqdf	afb1782d38	fix: Separate Backup CRs into their own kustomization ... The Backup and Restore CRs need the rancher-backup CRDs to exist first. Moved them to a separate kustomization that depends on the operator being ready.	2026-03-29 22:22:29 +00:00
micqdf	48870433bf	fix: Remove tls:external from Rancher HelmRelease ... With Tailscale LoadBalancer, TLS is not actually terminated at the edge. The Tailscale proxy does TCP passthrough, so Rancher must serve its own TLS certs. Setting tls: external caused Rancher to listen HTTP-only, which broke HTTPS access through Tailscale.	2026-03-29 22:19:23 +00:00
micqdf	f2c506b350	refactor: Replace CNPG external DB with rancher-backup operator ... Rancher 2.x uses embedded etcd, not an external PostgreSQL database. The CATTLE_DB_CATTLE_* env vars are Rancher v1 only and were ignored. - Remove all CNPG (CloudNativePG) cluster, operator, and related configs - Remove external DB env vars from Rancher HelmRelease - Remove rancher-db-password ExternalSecret - Add rancher-backup operator HelmRelease (v106.0.2+up8.1.0) - Add B2 credentials ExternalSecret for backup storage - Add recurring Backup CR (daily at 03:00, 7 day retention) - Add commented-out Restore CR for rebuild recovery - Update Flux dependency graph accordingly	2026-03-29 21:53:16 +00:00
micqdf	efdf13976a	fix: Handle missing 'online' field in Tailscale API response	2026-03-29 13:52:23 +00:00
micqdf	5269884408	feat: Auto-cleanup stale Tailscale devices before cluster boot ... Adds tailscale-cleanup Ansible role that uses the Tailscale API to delete offline devices matching reserved hostnames (e.g. rancher). Runs during site.yml before Finalize to prevent hostname collisions like rancher-1 on rebuild. Requires TAILSCALE_API_KEY (API access token) passed as extra var.	2026-03-29 11:47:53 +00:00
micqdf	6e5b0518be	feat: Add kubeconfig refresh script and fix Ansible Finalize to use public IP ... - scripts/refresh-kubeconfig.sh fetches a fresh kubeconfig from CP1 - Ansible site.yml Finalize step now uses public IP instead of Tailscale hostname for the kubeconfig server address - Updated AGENTS.md with kubeconfig refresh instructions	2026-03-29 03:31:36 +00:00
micqdf	905d069e91	fix: Add serverName to CNPG externalClusters for B2 recovery ... CNPG uses the external cluster name (b2-backup) as the barman server name by default, but the backups were stored under server name rancher-db.	2026-03-29 03:22:19 +00:00
micqdf	25ba4b7115	fix: Add skipEmptyWalArchiveCheck annotation and B2 secret healthcheck to CNPG ... - Skip WAL archive emptiness check so recovery works when restoring over an existing backup archive in B2 - Add healthCheck for b2-credentials secret in CNPG kustomization to prevent recovery from starting before ExternalSecret has synced	2026-03-29 03:15:23 +00:00
micqdf	6a593fd559	feat: Add B2 recovery bootstrap to CNPG cluster	2026-03-29 00:22:24 +00:00
micqdf	936f54a1b5	fix: Restore canonical Rancher tailnet hostname	2026-03-29 00:00:39 +00:00
micqdf	c9df11e65f	fix: Align Rancher tailnet hostname with live proxy	2026-03-28 23:47:09 +00:00
micqdf	a3c238fda9	fix: Apply Rancher server URL after chart install	2026-03-28 23:12:59 +00:00
micqdf	a15fa50302	fix: Use Doppler-backed Rancher bootstrap password	2026-03-28 22:51:38 +00:00
micqdf	0f4f0b09fb	fix: Add Rancher DB password ExternalSecret	2026-03-28 22:42:05 +00:00
micqdf	4c002a870c	fix: Remove invalid Rancher server-url manifest	2026-03-28 22:39:31 +00:00
micqdf	43d11ac7e6	docs: Add agent guidance and sync Rancher docs	2026-03-28 22:13:37 +00:00
micqdf	8c5edcf0a1	fix: Set Rancher server URL to tailnet hostname	2026-03-28 04:07:44 +00:00
micqdf	a81da0d178	feat: Expose Rancher via Tailscale hostname	2026-03-28 03:59:02 +00:00
micqdf	2a72527c79	fix: Switch Traefik from LoadBalancer to NodePort, remove unused Hetzner LB	2026-03-28 03:21:19 +00:00
micqdf	7cb3b84ecb	feat: Replace custom pgdump job with CNPG ScheduledBackup	2026-03-28 03:15:39 +00:00
micqdf	d4930235fa	fix: Point CNPG backups at the existing B2 bucket	2026-03-26 23:35:19 +00:00
micqdf	ee8dc4b451	fix: Add Role for B2 credentials access	2026-03-26 23:04:40 +00:00
micqdf	144d40e7ac	feat: Add RBAC for CNP to read B2 credentials secret	2026-03-26 22:56:00 +00:00
micqdf	cc14e32572	fix: Use gzip instead of lzop for backup compression	2026-03-26 22:51:10 +00:00
micqdf	a207a5a7fd	fix: Remove invalid encryption field from CNP backup config	2026-03-26 22:49:29 +00:00
micqdf	4e1772c175	feat: Add B2 backup configuration to CNP Cluster	2026-03-26 22:47:31 +00:00
micqdf	ff70b12084	chore: Add HTTP/HTTPS firewall rules for Load Balancer	2026-03-26 22:36:13 +00:00
micqdf	a3963c56e6	cleanup: Remove traefik-config, simplify traefik helmrelease	2026-03-26 03:16:56 +00:00
micqdf	612435c42c	fix: Add Hetzner LB health check config to Traefik	2026-03-26 03:11:10 +00:00
micqdf	ac42f671a2	fix: Remove addon-traefik-config dependency from flux-ui	2026-03-26 03:05:58 +00:00
micqdf	dbe7ec0468	fix: Remove expose boolean from traefik ports config	2026-03-26 03:01:13 +00:00
micqdf	816ac8b3c0	fix: Use official Traefik helm repo instead of rancher-stable	2026-03-26 02:59:00 +00:00
micqdf	6f7998639f	fix: Use standard kustomize API in traefik addon	2026-03-26 02:56:52 +00:00
micqdf	7a14f89ad1	fix: Correct traefik kustomization path and sourceRef	2026-03-26 02:55:37 +00:00
micqdf	786901c5d7	fix: Correct traefik kustomization reference (directory not file)	2026-03-26 02:54:29 +00:00
micqdf	46f3d1130b	feat: Add Flux-managed Traefik HelmRelease with Hetzner LB config	2026-03-26 02:52:49 +00:00
micqdf	2fe5a626d4	fix: Add Hetzner network zone annotation to Traefik LoadBalancer	2026-03-26 02:30:43 +00:00
micqdf	2ef68c8087	fix: Remove deprecated enablePodMonitor field in CNP Cluster	2026-03-26 01:01:53 +00:00
micqdf	e2cae18f5f	fix: Remove backup config for initial deployment - add backup after DB is running	2026-03-26 00:46:50 +00:00
micqdf	e0c1e41ee9	fix: Remove bootstrap recovery - create fresh DB (recovery only needed after first backup)	2026-03-26 00:43:49 +00:00
micqdf	63533de901	fix: Fix retentionPolicy format (14d not keep14)	2026-03-26 00:41:44 +00:00
micqdf	1b39710f63	fix: Move retentionPolicy to correct location in backup spec	2026-03-26 00:39:25 +00:00
micqdf	8c034323dc	fix: Fix Cluster CR with correct barmanObjectStore schema	2026-03-26 00:35:23 +00:00
micqdf	5fa2b411ee	fix: Fix Cluster CR schema - use barmanObjectStore instead of b2	2026-03-26 00:33:04 +00:00
micqdf	3ea28e525f	fix: Fix CNP operator image repository (cloudnative-pg not postgresql)	2026-03-26 00:21:09 +00:00
micqdf	4b95ba113d	fix: Remove LPP helm (already installed by k3s), fix CNP chart version to 0.27.1	2026-03-26 00:13:22 +00:00
micqdf	13627bf81f	fix: Split CNP operator from CNP cluster to fix CRD dependency ... - Move CNP operator HelmRelease to cnpg-operator folder - Create addon-cnpg-operator kustomization (deploys operator first) - Update addon-cnpg to dependOn addon-cnpg-operator - Add addon-cnpg as dependency for addon-rancher (needs database)	2026-03-26 00:06:34 +00:00
micqdf	ef3fb2489a	fix: Convert kustomization-lpp and kustomization-cnpg to Flux Kustomization CRs	2026-03-26 00:03:53 +00:00
micqdf	7097495d72	fix: Add missing metadata.name to kustomization-lpp and kustomization-cnpg	2026-03-25 23:39:45 +00:00
micqdf	9d601dc77c	feat: Add CloudNativePG with B2 backups for persistent Rancher database ... - Add Local Path Provisioner for storage - Add CloudNativePG operator (v1.27.0) via Flux - Create PostgreSQL cluster with B2 (Backblaze) auto-backup/restore - Update Rancher to use external PostgreSQL via CATTLE_DB_CATTLE_* env vars - Add weekly pg_dump CronJob to B2 (Sundays 2AM) - Add pre-destroy backup hook to destroy workflow - Add B2 credentials to Doppler (B2_ACCOUNT_ID, B2_APPLICATION_KEY) - Generate RANCHER_DB_PASSWORD in Doppler Backup location: HetznerTerra/rancher-backups/ Retention: 14 backups	2026-03-25 23:06:45 +00:00
micqdf	f36445d99a	Fix CNI: configure flannel to use private network interface (enp7s0) instead of public	2026-03-25 01:44:33 +00:00
micqdf	89c2c99963	Fix Rancher: remove conflicting LoadBalancer, add HTTPS port-forward, use tailscale serve only	2026-03-25 00:59:16 +00:00
micqdf	4a35cfb549	Fix Rancher: use correct targetPort 444 for HTTPS	2026-03-24 23:30:58 +00:00
micqdf	3d50bfc534	Fix Rancher service selector: use cattle-system-rancher label	2026-03-24 23:25:36 +00:00
micqdf	ab2f287bfb	Fix Rancher: use correct service name cattle-system-rancher	2026-03-24 22:30:49 +00:00
micqdf	dcb2675b67	Upgrade Rancher to 2.13.3 for K8s 1.34 compatibility	2026-03-24 21:42:51 +00:00
micqdf	b40bec7e0e	Fix Rancher: use Doppler secret instead of hardcoded password	2026-03-24 21:13:23 +00:00
micqdf	efe0c0cfd5	Fix Rancher: upgrade to 2.10.3 for K8s 1.34 compatibility	2026-03-24 20:29:38 +00:00
micqdf	c61d9f9c1d	Remove traefik-config dependency from Rancher	2026-03-24 20:02:08 +00:00
micqdf	60ceac4624	Fix Rancher access: add kubectl port-forward + tailscale serve setup	2026-03-24 20:01:57 +00:00
micqdf	47b384a337	Fix Rancher access: add Tailscale service for Traefik with port 9442, fix deployment order	2026-03-24 19:40:37 +00:00
micqdf	ecf17113fb	Fix Rancher deployment: add cattle-system namespace, fix Traefik config with port 9442	2026-03-24 19:09:28 +00:00
micqdf	4ffbcfa312	Add Rancher management UI	2026-03-24 01:53:04 +00:00
micqdf	8745bcda47	Fix Weave GitOps image tag - remove invalid v0.41.0 ... The version v0.41.0 doesn't exist in the registry. Removing explicit image tag to let the chart use its default compatible version.	2026-03-24 01:39:48 +00:00
micqdf	e47ec2a3e7	Update Weave GitOps to v0.41.0 to support HelmRelease v2 API ... Fixes error: 'no matches for kind HelmRelease in version v2beta1' The cluster uses HelmRelease v2 API but Weave GitOps v0.38.0 was looking for the old v2beta1 API. Updated image tag to v0.41.0 which supports the newer API version.	2026-03-24 01:33:10 +00:00
micqdf	45c899d2bd	Configure Weave GitOps to use Doppler-managed admin credentials ... Changes: - Enable adminUser creation but disable Helm-managed secret - Use ExternalSecret (cluster-user-auth) from Doppler instead - Doppler secrets: WEAVE_GITOPS_ADMIN_USERNAME and WEAVE_GITOPS_ADMIN_PASSWORD_BCRYPT_HASH - Added cluster-user-auth to viewSecretsResourceNames for RBAC Login credentials are now managed via Doppler and External Secrets Operator.	2026-03-24 01:01:30 +00:00
micqdf	0e52d8f159	Use Tailscale DNS names instead of IPs for TLS SANs ... Changed from hardcoded Tailscale IPs to DNS names: - k8s-cluster-cp-1.silverside-gopher.ts.net - k8s-cluster-cp-2.silverside-gopher.ts.net - k8s-cluster-cp-3.silverside-gopher.ts.net This is more robust since Tailscale IPs change on rebuild, but DNS names remain consistent. After next rebuild, cluster accessible via: - kubectl --server=https://k8s-cluster-cp-1.silverside-gopher.ts.net:6443	2026-03-23 23:50:48 +00:00
micqdf	4726db2b5b	Add Tailscale IPs to k3s TLS SANs for secure tailnet access ... Changes: - Add tailscale_control_plane_ips list to k3s-server defaults - Include all 3 control plane Tailscale IPs (100.120.55.97, 100.108.90.123, 100.92.149.85) - Update primary k3s install to add Tailscale IPs to TLS certificates - Enables kubectl access via Tailscale without certificate errors After next deploy, cluster will be accessible via: - kubectl --server=https://100.120.55.97:6443 (or any CP tailscale IP) - kubectl --server=https://k8s-cluster-cp-1:6443 (via tailscale DNS)	2026-03-23 23:04:00 +00:00
micqdf	90d105e5ea	Fix kube_api_endpoint variable passing for HA cluster ... - Remove circular variable reference in site.yml - Add kube_api_endpoint default to k3s-server role - Variable is set via inventory group_vars and passed to role - Primary CP now correctly adds LB IP to TLS SANs Note: Existing cluster needs destroy/rebuild to regenerate certificates.	2026-03-23 03:01:53 +00:00
micqdf	952a80a742	Fix HA cluster join via Load Balancer private IP ... Changes: - Use LB private IP (10.0.1.5) instead of public IP for cluster joins - Add LB private IP to k3s TLS SANs on primary control plane - This allows secondary CPs and workers to verify certificates when joining via LB Fixes x509 certificate validation error when joining via LB public IP.	2026-03-23 02:56:41 +00:00
micqdf	4965017b86	Fix Load Balancer network attachment ... Add hcloud_load_balancer_network resource to attach LB to private network. This is required before targets can use use_private_ip=true. LB gets IP 10.0.1.5 on the private network.	2026-03-23 02:44:35 +00:00
micqdf	b2b9c38b91	Fix Load Balancer output attribute - use ipv4 instead of ipv4_address	2026-03-23 02:40:50 +00:00
micqdf	ff31cb4e74	Implement HA control plane with Load Balancer (3-3 topology) ... Major changes: - Terraform: Scale to 3 control planes (cx23) + 3 workers (cx33) - Terraform: Add Hetzner Load Balancer (lb11) for Kubernetes API - Terraform: Add kube_api_lb_ip output - Ansible: Add community.network collection to requirements - Ansible: Update inventory to include LB endpoint - Ansible: Configure secondary CPs and workers to join via LB - Ansible: Add k3s_join_endpoint variable for HA joins - Workflow: Add imports for cp-2, cp-3, and worker-3 - Docs: Update STABLE_BASELINE.md with HA topology and phase gates Topology: - 3 control planes (cx23 - 2 vCPU, 8GB RAM each) - 3 workers (cx33 - 4 vCPU, 16GB RAM each) - 1 Load Balancer (lb11) routing to all 3 control planes on port 6443 - Workers and secondary CPs join via LB endpoint for HA Cost impact: +~€26/month (2 extra CPs + 1 extra worker + LB)	2026-03-23 02:39:39 +00:00
micqdf	8b4a445b37	Update STABLE_BASELINE.md - CCM/CSI integration achieved ... Document the successful completion of Hetzner CCM and CSI integration: - CCM deployed via Ansible before workers join (fixes uninitialized taint) - CSI provides hcloud-volumes StorageClass for persistent storage - Two consecutive rebuilds passed all phase gates - PVC provisioning tested and working Platform now has full cloud provider integration with persistent volumes.	2026-03-23 02:25:00 +00:00
micqdf	e447795395	Install helm binary in ccm-deploy role before using it ... The kubernetes.core.helm module requires helm CLI to be installed on the target node. Added check and install step using the official helm install script.	2026-03-23 00:07:39 +00:00
micqdf	31b82c9371	Deploy CCM via Ansible before workers join to fix external cloud provider ... This fixes the chicken-and-egg problem where workers with --kubelet-arg=cloud-provider=external couldn't join because CCM wasn't running yet to remove the node.cloudprovider.kubernetes.io/uninitialized taint. Changes: - Create ansible/roles/ccm-deploy/ to deploy CCM via Helm during Ansible phase - Reorder site.yml: CCM deploys after secrets but before workers join - CCM runs on control_plane[0] with proper tolerations for control plane nodes - Add 10s pause after CCM ready to ensure it can process new nodes - Workers can now successfully join with external cloud provider enabled Flux still manages CCM for updates, but initial install happens in Ansible.	2026-03-22 23:58:03 +00:00
micqdf	cadfedacf1	Fix providerID health check - use shell module for piped grep	2026-03-22 22:55:55 +00:00
micqdf	561cd67b0c	Enable Hetzner CCM and CSI for cloud provider integration ... - Enable --kubelet-arg=cloud-provider=external on all nodes (control planes and workers) - Activate CCM Kustomization with 10m timeout for Hetzner cloud-controller-manager - Activate CSI Kustomization with dependsOn CCM and 10m timeout for hcloud-csi - Update deploy workflow to wait for CCM/CSI readiness (600s timeout) - Add providerID verification to post-deploy health checks This enables proper cloud provider integration with Hetzner CCM for node labeling and Hetzner CSI for persistent volume provisioning.	2026-03-22 22:26:21 +00:00
micqdf	4eebbca648	docs: update README for deferred observability baseline	2026-03-22 01:04:53 +00:00
micqdf	7b5d794dfc	fix: update health checks for deferred observability	2026-03-22 01:04:27 +00:00
micqdf	8643bbfc12	fix: defer observability to get clean baseline	2026-03-22 01:03:55 +00:00
micqdf	84f446c2e6	fix: restore observability timeouts to 5 minutes	2026-03-22 00:43:37 +00:00
micqdf	d446e86ece	fix: use static grafana password, remove externalsecret dependency	2026-03-22 00:43:21 +00:00
micqdf	90c7f565e0	fix: remove tailscale ingress dependencies from observability	2026-03-22 00:42:35 +00:00
micqdf	989848fa89	fix: increase observability timeouts to 10 minutes	2026-03-21 19:34:43 +00:00
micqdf	56e5807474	fix: create doppler ClusterSecretStore after ESO is installed	2026-03-21 19:19:43 +00:00
micqdf	df0511148c	fix: unsuspend tailscale operator for stable baseline	2026-03-21 19:03:39 +00:00
micqdf	894e6275b1	docs: update stable baseline to defer ccm/csi	2026-03-21 18:41:36 +00:00
micqdf	a01cf435d4	fix: skip ccm/csi waits for stable baseline - using k3s embedded	2026-03-21 18:40:53 +00:00