100 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
micqdf	ecf17113fb	Fix Rancher deployment: add cattle-system namespace, fix Traefik config with port 9442	2026-03-24 19:09:28 +00:00
micqdf	4ffbcfa312	Add Rancher management UI	2026-03-24 01:53:04 +00:00
micqdf	8745bcda47	Fix Weave GitOps image tag - remove invalid v0.41.0 ... The version v0.41.0 doesn't exist in the registry. Removing explicit image tag to let the chart use its default compatible version.	2026-03-24 01:39:48 +00:00
micqdf	e47ec2a3e7	Update Weave GitOps to v0.41.0 to support HelmRelease v2 API ... Fixes error: 'no matches for kind HelmRelease in version v2beta1' The cluster uses HelmRelease v2 API but Weave GitOps v0.38.0 was looking for the old v2beta1 API. Updated image tag to v0.41.0 which supports the newer API version.	2026-03-24 01:33:10 +00:00
micqdf	45c899d2bd	Configure Weave GitOps to use Doppler-managed admin credentials ... Changes: - Enable adminUser creation but disable Helm-managed secret - Use ExternalSecret (cluster-user-auth) from Doppler instead - Doppler secrets: WEAVE_GITOPS_ADMIN_USERNAME and WEAVE_GITOPS_ADMIN_PASSWORD_BCRYPT_HASH - Added cluster-user-auth to viewSecretsResourceNames for RBAC Login credentials are now managed via Doppler and External Secrets Operator.	2026-03-24 01:01:30 +00:00
micqdf	0e52d8f159	Use Tailscale DNS names instead of IPs for TLS SANs ... Changed from hardcoded Tailscale IPs to DNS names: - k8s-cluster-cp-1.silverside-gopher.ts.net - k8s-cluster-cp-2.silverside-gopher.ts.net - k8s-cluster-cp-3.silverside-gopher.ts.net This is more robust since Tailscale IPs change on rebuild, but DNS names remain consistent. After next rebuild, cluster accessible via: - kubectl --server=https://k8s-cluster-cp-1.silverside-gopher.ts.net:6443	2026-03-23 23:50:48 +00:00
micqdf	4726db2b5b	Add Tailscale IPs to k3s TLS SANs for secure tailnet access ... Changes: - Add tailscale_control_plane_ips list to k3s-server defaults - Include all 3 control plane Tailscale IPs (100.120.55.97, 100.108.90.123, 100.92.149.85) - Update primary k3s install to add Tailscale IPs to TLS certificates - Enables kubectl access via Tailscale without certificate errors After next deploy, cluster will be accessible via: - kubectl --server=https://100.120.55.97:6443 (or any CP tailscale IP) - kubectl --server=https://k8s-cluster-cp-1:6443 (via tailscale DNS)	2026-03-23 23:04:00 +00:00
micqdf	90d105e5ea	Fix kube_api_endpoint variable passing for HA cluster ... - Remove circular variable reference in site.yml - Add kube_api_endpoint default to k3s-server role - Variable is set via inventory group_vars and passed to role - Primary CP now correctly adds LB IP to TLS SANs Note: Existing cluster needs destroy/rebuild to regenerate certificates.	2026-03-23 03:01:53 +00:00
micqdf	952a80a742	Fix HA cluster join via Load Balancer private IP ... Changes: - Use LB private IP (10.0.1.5) instead of public IP for cluster joins - Add LB private IP to k3s TLS SANs on primary control plane - This allows secondary CPs and workers to verify certificates when joining via LB Fixes x509 certificate validation error when joining via LB public IP.	2026-03-23 02:56:41 +00:00
micqdf	4965017b86	Fix Load Balancer network attachment ... Add hcloud_load_balancer_network resource to attach LB to private network. This is required before targets can use use_private_ip=true. LB gets IP 10.0.1.5 on the private network.	2026-03-23 02:44:35 +00:00
micqdf	b2b9c38b91	Fix Load Balancer output attribute - use ipv4 instead of ipv4_address	2026-03-23 02:40:50 +00:00
micqdf	ff31cb4e74	Implement HA control plane with Load Balancer (3-3 topology) ... Major changes: - Terraform: Scale to 3 control planes (cx23) + 3 workers (cx33) - Terraform: Add Hetzner Load Balancer (lb11) for Kubernetes API - Terraform: Add kube_api_lb_ip output - Ansible: Add community.network collection to requirements - Ansible: Update inventory to include LB endpoint - Ansible: Configure secondary CPs and workers to join via LB - Ansible: Add k3s_join_endpoint variable for HA joins - Workflow: Add imports for cp-2, cp-3, and worker-3 - Docs: Update STABLE_BASELINE.md with HA topology and phase gates Topology: - 3 control planes (cx23 - 2 vCPU, 8GB RAM each) - 3 workers (cx33 - 4 vCPU, 16GB RAM each) - 1 Load Balancer (lb11) routing to all 3 control planes on port 6443 - Workers and secondary CPs join via LB endpoint for HA Cost impact: +~€26/month (2 extra CPs + 1 extra worker + LB)	2026-03-23 02:39:39 +00:00
micqdf	8b4a445b37	Update STABLE_BASELINE.md - CCM/CSI integration achieved ... Document the successful completion of Hetzner CCM and CSI integration: - CCM deployed via Ansible before workers join (fixes uninitialized taint) - CSI provides hcloud-volumes StorageClass for persistent storage - Two consecutive rebuilds passed all phase gates - PVC provisioning tested and working Platform now has full cloud provider integration with persistent volumes.	2026-03-23 02:25:00 +00:00
micqdf	e447795395	Install helm binary in ccm-deploy role before using it ... The kubernetes.core.helm module requires helm CLI to be installed on the target node. Added check and install step using the official helm install script.	2026-03-23 00:07:39 +00:00
micqdf	31b82c9371	Deploy CCM via Ansible before workers join to fix external cloud provider ... This fixes the chicken-and-egg problem where workers with --kubelet-arg=cloud-provider=external couldn't join because CCM wasn't running yet to remove the node.cloudprovider.kubernetes.io/uninitialized taint. Changes: - Create ansible/roles/ccm-deploy/ to deploy CCM via Helm during Ansible phase - Reorder site.yml: CCM deploys after secrets but before workers join - CCM runs on control_plane[0] with proper tolerations for control plane nodes - Add 10s pause after CCM ready to ensure it can process new nodes - Workers can now successfully join with external cloud provider enabled Flux still manages CCM for updates, but initial install happens in Ansible.	2026-03-22 23:58:03 +00:00
micqdf	cadfedacf1	Fix providerID health check - use shell module for piped grep	2026-03-22 22:55:55 +00:00
micqdf	561cd67b0c	Enable Hetzner CCM and CSI for cloud provider integration ... - Enable --kubelet-arg=cloud-provider=external on all nodes (control planes and workers) - Activate CCM Kustomization with 10m timeout for Hetzner cloud-controller-manager - Activate CSI Kustomization with dependsOn CCM and 10m timeout for hcloud-csi - Update deploy workflow to wait for CCM/CSI readiness (600s timeout) - Add providerID verification to post-deploy health checks This enables proper cloud provider integration with Hetzner CCM for node labeling and Hetzner CSI for persistent volume provisioning.	2026-03-22 22:26:21 +00:00
micqdf	4eebbca648	docs: update README for deferred observability baseline	2026-03-22 01:04:53 +00:00
micqdf	7b5d794dfc	fix: update health checks for deferred observability	2026-03-22 01:04:27 +00:00
micqdf	8643bbfc12	fix: defer observability to get clean baseline	2026-03-22 01:03:55 +00:00
micqdf	84f446c2e6	fix: restore observability timeouts to 5 minutes	2026-03-22 00:43:37 +00:00
micqdf	d446e86ece	fix: use static grafana password, remove externalsecret dependency	2026-03-22 00:43:21 +00:00
micqdf	90c7f565e0	fix: remove tailscale ingress dependencies from observability	2026-03-22 00:42:35 +00:00
micqdf	989848fa89	fix: increase observability timeouts to 10 minutes	2026-03-21 19:34:43 +00:00
micqdf	56e5807474	fix: create doppler ClusterSecretStore after ESO is installed	2026-03-21 19:19:43 +00:00
micqdf	df0511148c	fix: unsuspend tailscale operator for stable baseline	2026-03-21 19:03:39 +00:00
micqdf	894e6275b1	docs: update stable baseline to defer ccm/csi	2026-03-21 18:41:36 +00:00
micqdf	a01cf435d4	fix: skip ccm/csi waits for stable baseline - using k3s embedded	2026-03-21 18:40:53 +00:00
micqdf	84f77c4a68	fix: use kubectl patch instead of apply for flux controller nodeSelector	2026-03-21 18:05:41 +00:00
micqdf	2e4196688c	fix: bootstrap flux in phases - crds first, then resources	2026-03-21 17:42:39 +00:00
micqdf	8d1f9f4944	fix: add k3s reset logic for primary control plane	2026-03-21 16:10:17 +00:00
micqdf	d4fd43e2f5	refactor: simplify k3s-server bootstrap for	2026-03-21 15:48:33 +00:00
micqdf	48a80c362c	fix: disable external cloud-provider kubelet arg for stable baseline	2026-03-21 14:36:54 +00:00
micqdf	fcf7f139ff	fix: use public api endpoint for flux bootstrap	2026-03-21 00:07:51 +00:00
micqdf	7139ae322d	fix: bootstrap flux during cluster deploy	2026-03-20 10:37:11 +00:00
micqdf	528a8dc210	fix: defer doppler store until eso is installed	2026-03-20 09:30:17 +00:00
micqdf	349f75729a	fix: bootstrap tailscale namespace before secret	2026-03-20 09:24:35 +00:00
micqdf	522626a52b	refactor: simplify stable cluster baseline	2026-03-20 02:24:37 +00:00
micqdf	5bd4c41c2d	fix: restore k3s agent bootstrap	2026-03-20 01:50:16 +00:00
micqdf	3e41f71b1b	fix: harden terraform destroy workflow	2026-03-19 23:26:03 +00:00
micqdf	9d2f30de32	fix: prepare k3s for external cloud provider	2026-03-17 01:21:23 +00:00
micqdf	08a3031276	refactor: retire imperative addon roles	2026-03-17 01:04:02 +00:00
micqdf	e3ce91db62	fix: align flux ccm with live deployment	2026-03-11 18:17:16 +00:00
micqdf	bed8e4afc8	feat: migrate core addons toward flux	2026-03-11 17:43:35 +00:00
micqdf	2d4de6cff8	fix: bootstrap doppler store outside flux	2026-03-09 02:58:26 +00:00
micqdf	4a83d981c8	fix: skip dry-run validation for doppler store sync	2026-03-09 02:52:08 +00:00
micqdf	d188a51ef6	fix: move doppler store manifests out of ignored path	2026-03-09 02:45:46 +00:00
micqdf	646ef16258	fix: stabilize flux and external secrets reconciliation	2026-03-09 02:25:27 +00:00
micqdf	6f2e056b98	feat: sync runtime secrets from doppler	2026-03-09 00:25:41 +00:00
micqdf	e10a70475f	fix: right-size flux observability workloads	2026-03-08 05:17:22 +00:00
micqdf	f95e0051a5	feat: automate private tailnet access on cp1	2026-03-08 04:16:06 +00:00
micqdf	7c15ac5846	feat: add flux ui on shared tailscale endpoint	2026-03-07 12:30:17 +00:00
micqdf	4c104f74e8	feat: route observability through one tailscale endpoint	2026-03-07 01:04:03 +00:00
micqdf	be04602bfb	fix: make flux bootstrap reachable from cluster	2026-03-07 00:38:29 +00:00
micqdf	06c1356f1e	feat: expose flux observability services over tailscale	2026-03-05 00:43:29 +00:00
micqdf	86fb5d5b90	fix: move observability gitops gating to role level	2026-03-05 00:17:25 +00:00
micqdf	8b403cd1d6	feat: migrate observability stack to flux gitops	2026-03-04 23:38:40 +00:00
micqdf	480a079dc8	fix: fail fast when loki datasource has no labels	2026-03-04 21:00:01 +00:00
micqdf	ff8e32daf5	fix: add loki nodeport fallback for grafana datasource reachability	2026-03-04 19:39:16 +00:00
micqdf	eb1ad0bea7	fix: make grafana prometheus datasource resilient with nodeport fallback	2026-03-04 19:22:31 +00:00
micqdf	9ff9d1e633	fix: clear stale helm pending revisions before kube-prometheus upgrade	2026-03-04 18:35:55 +00:00
micqdf	6177b581e4	fix: correct dashboard verification checks and retry helm upgrade lock	2026-03-04 08:48:30 +00:00
micqdf	b1e21c4a4b	fix: speed up dashboards workflow firewall apply and set TF_VAR env	2026-03-04 03:54:56 +00:00
micqdf	2f166ed9e7	feat: manage grafana content as code with fast dashboard workflow	2026-03-04 03:36:01 +00:00
micqdf	1c39274df7	feat: stabilize tailscale observability exposure with declarative proxy class	2026-03-04 01:37:00 +00:00
micqdf	28eaa36ec4	fix: use tag:k8s for tailscale operator default tags	2026-03-04 00:57:33 +00:00
micqdf	02fa71c0aa	fix: use tag:k8 for tailscale operator default tag	2026-03-04 00:27:47 +00:00
micqdf	2bbf05cdca	fix: make tailscale operator non-blocking by default and gate observability patching on readiness	2026-03-03 21:47:16 +00:00
micqdf	213c1fb4e4	fix: detect tailscale tag permission errors and clean access output	2026-03-03 08:51:25 +00:00
micqdf	414ac73c25	fix: fail fast on tailscale oauth 403 with actionable message	2026-03-02 23:57:53 +00:00
micqdf	542d7a6be5	fix: align tailscale proxy tags with operator tags	2026-03-02 23:36:18 +00:00
micqdf	210b617cc9	fix: pin tailscale operator to control-plane node for DNS stability	2026-03-02 23:32:36 +00:00
micqdf	3686249e31	chore: remove blocking Tailscale endpoint retries in observability	2026-03-02 22:47:55 +00:00
micqdf	f56d1447c1	fix: make Tailscale endpoint wait non-blocking in observability	2026-03-02 22:08:36 +00:00
micqdf	63247b79a6	fix: harden Tailscale operator rollout with preflight and diagnostics	2026-03-02 21:39:47 +00:00
micqdf	f6e159406a	ci: retrigger with correct chart name	2026-03-02 21:15:44 +00:00
micqdf	0ae1c9395c	fix: use correct chart name tailscale/tailscale-operator	2026-03-02 21:15:37 +00:00
micqdf	272c5ddc6e	ci: retrigger with fixed Tailscale operator version	2026-03-02 21:04:01 +00:00
micqdf	eb6bf3862a	fix: update Tailscale operator chart version to 1.95.91	2026-03-02 21:03:53 +00:00
micqdf	5a3f7550fe	docs: add Gitea secrets setup guide for Tailscale operator	2026-03-02 20:29:19 +00:00
micqdf	a0ed6523ec	feat: add Tailscale Kubernetes Operator for Grafana/Prometheus access	2026-03-02 20:28:51 +00:00
micqdf	4f61a840c7	ci: retrigger with non-blocking Loki install	2026-03-02 19:41:55 +00:00
micqdf	d876430703	fix: remove Helm wait flag, check Loki rollout separately	2026-03-02 19:41:46 +00:00
micqdf	56b6216257	ci: retrigger after Helm lock cleanup	2026-03-02 19:18:29 +00:00
micqdf	91fe2e658c	fix: clear stuck Helm lock before Loki install	2026-03-02 19:18:23 +00:00
micqdf	13cec1aa28	ci: retrigger with YAML fix	2026-03-02 18:29:25 +00:00
micqdf	bc133e65d3	fix: quote failed_when expression for YAML parsing	2026-03-02 18:29:16 +00:00
micqdf	df4fdb5496	ci: retrigger with Loki fixes	2026-03-02 18:21:23 +00:00
micqdf	cec7c42efb	fix: disable Loki caches and canary, use chart v6.10.0	2026-03-02 18:21:22 +00:00
micqdf	ee692620b5	ci: retrigger with Loki v6.10.0	2026-03-02 17:59:37 +00:00
micqdf	a6d327fa1f	fix: re-enable Loki with v6.10.0 and minimal working config	2026-03-02 17:59:31 +00:00
micqdf	fe6cb39eaf	ci: retrigger with Loki disabled	2026-03-02 17:07:59 +00:00
micqdf	feaefd28a1	fix: disable Loki to unblock pipeline - will re-enable separately	2026-03-02 17:07:51 +00:00
micqdf	80ab59e22d	ci: retrigger with enhanced Loki diagnostics	2026-03-02 14:44:31 +00:00
micqdf	6c0282e9d5	fix: add more Loki diagnostics - values content and Helm releases	2026-03-02 14:44:12 +00:00
micqdf	45aa616741	ci: retrigger for Loki diagnostics	2026-03-02 14:42:42 +00:00
micqdf	b595c1738a	fix: show detailed Loki template and resource diagnostics	2026-03-02 14:41:40 +00:00
micqdf	1c4dfd7fae	ci: retrigger with fixed Loki values	2026-03-02 13:58:31 +00:00
micqdf	6b9fc1f6b8	fix: add all required replica=0 settings for Loki v6 singleBinary	2026-03-02 13:58:15 +00:00
micqdf	2b5cad9d15	ci: retrigger for Loki template debug	2026-03-02 13:57:03 +00:00