100 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
micqdf	5cf68771dd	fix: wait longer for flux health reconciles	2026-04-30 17:26:16 +00:00
micqdf	6d6e3e8371	fix: import runner image archives during prepull	2026-04-30 09:08:44 +00:00
micqdf	353a408dac	fix: support bullseye pip in gitea runner	2026-04-30 07:57:55 +00:00
micqdf	b3612083ad	fix: disable terraform wrapper in gitea workflows	2026-04-30 07:54:48 +00:00
micqdf	8c0dbd997d	fix: simplify terraform deploy job lifecycle	2026-04-30 07:52:49 +00:00
micqdf	3a975a323c	fix: remove deploy pr comment post hook	2026-04-30 07:49:15 +00:00
micqdf	d126de4dc4	fix: align terraform ci version with provider lock	2026-04-30 07:45:10 +00:00
micqdf	a33a993867	fix: harden cluster rebuild determinism	2026-04-30 07:36:27 +00:00
micqdf	f52e657f9f	docs	2026-04-30 07:03:21 +00:00
micqdf	f49b08f50c	fix: reinstall k3s on version drift	2026-04-30 06:03:53 +00:00
micqdf	327bb860b7	fix: pin k3s below rancher limit	2026-04-30 05:23:37 +00:00
micqdf	fd5451a5ef	fix: wait for ssh before gathering facts	2026-04-30 03:44:13 +00:00
micqdf	7333cb2780	test	2026-04-30 02:59:47 +00:00
micqdf	feecf97cd5	test	2026-04-30 02:43:30 +00:00
micqdf	b5bcec2663	fix: use kubeconfig for observability reset	2026-04-27 02:28:38 +00:00
micqdf	0ad56405ee	fix: seed grafana observability images	2026-04-27 01:50:41 +00:00
micqdf	d050e8962a	fix: seed cert-manager images before flux	2026-04-27 00:04:19 +00:00
micqdf	d925eeac3f	fix: remove Rancher backup workflow	2026-04-26 22:13:20 +00:00
micqdf	2bde45e106	fix: allow intentional destroy without backup	2026-04-26 22:01:39 +00:00
micqdf	50752ca4b0	fix: allow initial deploy without Rancher backup	2026-04-26 21:27:57 +00:00
micqdf	a2ed9555c0	fix: vendor critical bootstrap charts	2026-04-26 21:01:01 +00:00
micqdf	14462dd870	fix: avoid resetting healthy observability	2026-04-26 20:25:42 +00:00
micqdf	0625eee297	fix: uninstall failed observability upgrades	2026-04-26 18:46:07 +00:00
micqdf	2dc4ab6329	fix: make observability image seeding non-fatal	2026-04-26 12:34:02 +00:00
micqdf	bbec0dfff4	fix: skip traefik in observability seeding	2026-04-26 12:06:41 +00:00
micqdf	6de826e030	fix: allow cached OCI chart artifacts	2026-04-26 11:44:24 +00:00
micqdf	bdba2b7af2	fix: defer observability image seeding	2026-04-26 11:13:22 +00:00
micqdf	499a3462e7	fix: seed observability dependencies	2026-04-26 10:32:25 +00:00
micqdf	daf6ccd0e4	fix: retry bootstrap image imports	2026-04-26 09:43:31 +00:00
micqdf	a6a630000a	fix: vendor Tailscale operator chart	2026-04-26 09:17:44 +00:00
micqdf	ff9e58d44f	fix: remove NFS chart fetch dependency	2026-04-26 07:48:11 +00:00
micqdf	8b94e4dd06	fix: import bootstrap images from runner	2026-04-26 06:13:37 +00:00
micqdf	547a29e000	fix: require kube-vip image archive	2026-04-26 05:04:39 +00:00
micqdf	760f0482d4	fix: pass Proxmox delete params in query	2026-04-26 04:32:01 +00:00
micqdf	440e268e4f	fix: seed kube-vip image from runner	2026-04-26 04:28:21 +00:00
micqdf	24851f5a9b	fix: retry transient Proxmox apply failures	2026-04-26 04:02:14 +00:00
micqdf	ded8efe7fb	test	2026-04-26 03:52:54 +00:00
micqdf	c10646d228	fix: harden tailnet smoke script	2026-04-26 03:09:18 +00:00
micqdf	50d97209e6	fix: ignore Rancher Turtles cleanup hook pod	2026-04-26 02:33:21 +00:00
micqdf	46b2ff7d19	fix: harden final health checks	2026-04-26 02:14:02 +00:00
micqdf	a4f1d179e9	fix: use Rancher registry for webhook image	2026-04-26 01:35:16 +00:00
micqdf	9879de5a86	fix: stop pre-pulling Rancher child images	2026-04-26 00:57:49 +00:00
micqdf	195e9bce25	fix: parallelize Rancher child image warmup	2026-04-26 00:02:12 +00:00
micqdf	4796606432	fix: warm Rancher child images on all nodes	2026-04-25 23:30:20 +00:00
micqdf	b1eab6a0fa	fix: vendor Rancher chart for bootstrap	2026-04-25 23:08:26 +00:00
micqdf	f3c96b65d2	fix: shorten Rancher chart retry windows	2026-04-25 22:30:07 +00:00
micqdf	c7a375758f	fix: retry Rancher chart pulls during waits	2026-04-25 22:03:09 +00:00
micqdf	d0be48b65c	fix: gate Tailscale addon on Helm release	2026-04-25 21:21:34 +00:00
micqdf	40647318b4	fix: tolerate cached Helm repository artifacts	2026-04-25 20:44:03 +00:00
micqdf	cdb26904d2	fix: retry Tailscale chart pulls during bootstrap	2026-04-25 20:11:43 +00:00
micqdf	3c06e046c2	fix: warm External Secrets image before install	2026-04-25 19:46:21 +00:00
micqdf	17f1815e7f	fix: use CRI pulls for Flux image warmup	2026-04-25 19:28:29 +00:00
micqdf	66e86e55ea	fix: require Flux image warmup before bootstrap	2026-04-25 19:02:32 +00:00
micqdf	43df412243	fix: handle missing Proxmox VM config during cleanup	2026-04-25 17:40:51 +00:00
micqdf	383ef9e9ac	fix: clean orphan Proxmox cloud-init volumes	2026-04-25 17:38:57 +00:00
micqdf	18abc5073b	fix: keep concurrent Terraform apply	2026-04-25 17:30:59 +00:00
micqdf	f8da2594ca	fix: serialize Proxmox VM apply	2026-04-25 17:27:59 +00:00
micqdf	e0359f0097	tes	2026-04-25 17:22:12 +00:00
micqdf	003333a061	fix: make health checks observe Flux readiness	2026-04-25 03:52:43 +00:00
micqdf	a6071c504b	fix: point Promtail at Loki service	2026-04-25 03:43:23 +00:00
micqdf	08123457f1	fix: ignore stale install hook pods in health check	2026-04-25 03:41:00 +00:00
micqdf	757d88ed52	fix: use cached Promtail images when available	2026-04-25 03:25:44 +00:00
micqdf	15defc686f	fix: allow slow Promtail image pulls	2026-04-25 03:10:47 +00:00
micqdf	abb7578328	fix: run post-deploy checks with bash	2026-04-25 02:42:54 +00:00
micqdf	bc87a7ca43	fix: avoid immutable observability PVC changes	2026-04-25 02:25:40 +00:00
micqdf	045880bdd6	fix: ignore stale Rancher helm operation pods	2026-04-25 02:23:30 +00:00
micqdf	bfcf57bcc5	fix: enforce post-deploy health checks	2026-04-25 02:22:16 +00:00
micqdf	7e3ebec95b	fix: wait for Rancher resources before rollout checks	2026-04-25 01:54:21 +00:00
micqdf	0c31c3b1d5	fix: fail fast on stalled Flux Helm releases	2026-04-25 01:40:42 +00:00
micqdf	5523feb563	fix: wait for Rancher Flux resources before rollout	2026-04-25 00:59:16 +00:00
micqdf	cafa2fa0b3	fix: reset stalled bootstrap Helm releases	2026-04-25 00:48:33 +00:00
micqdf	a7fd4c0b97	fix: wait on actual ESO deployment names	2026-04-25 00:07:48 +00:00
micqdf	e56a3a6c38	fix: wait for ESO webhook before ClusterSecretStore	2026-04-24 23:13:03 +00:00
micqdf	7b2eca07ab	fix: pull external-secrets chart from OCI	2026-04-24 15:24:58 +00:00
micqdf	347ca041ba	fix: reduce rerun bootstrap pre-pull delays	2026-04-24 12:09:34 +00:00
micqdf	3f52bad854	fix: make Ansible reruns faster and idempotent	2026-04-24 11:44:11 +00:00
micqdf	c89c31adea	fix: clean up Ansible bootstrap warnings	2026-04-24 11:07:13 +00:00
micqdf	68b293efe4	fix: qualify Flux HelmChart bootstrap resources	2026-04-24 10:47:13 +00:00
micqdf	1f465cc0c1	fix: force reconcile bootstrap Helm charts	2026-04-24 10:17:49 +00:00
micqdf	6e22bd26b3	fix: wait directly on ESO Helm readiness	2026-04-23 22:09:45 +00:00
micqdf	869880c152	fix: wait for ESO resources before CRD conditions	2026-04-23 21:17:44 +00:00
micqdf	31e95eb227	fix: pre-pull Flux controllers before bootstrap rollout	2026-04-23 20:36:57 +00:00
micqdf	12675417bd	fix: use correct namespace and deployment name for ESO rollout check ... The ESO deployment is named external-secrets-external-secrets in the external-secrets namespace, not external-secrets in kube-system.	2026-04-23 19:00:15 +00:00
micqdf	8e081ddfda	fix: wait on ESO deployment directly instead of Flux Kustomization status ... The addon-external-secrets Flux Kustomization was timing out during bootstrap because image pulls on fresh Proxmox VMs are slow. The critical dependency is the ESO deployment being available for the Doppler ClusterSecretStore. Replace the Kustomization readiness check with direct checks for ESO CRD establishment and deployment rollout, which are the actual prerequisites for the next step.	2026-04-23 07:32:19 +00:00
micqdf	4b7517c9c5	fix: health-check external-secrets addon via HelmRelease only ... The external-secrets Kustomization was still using wait=true, which makes Flux hold the addon in a failed state when the HelmRepository has transient fetch errors even though the HelmRelease and runtime controller deployments are healthy. Switch it to an explicit HelmRelease health check like the other helm-backed addons.	2026-04-23 07:11:21 +00:00
micqdf	f9bc53723f	fix: make image pre-pull roles fully best effort ... The pre-pull roles were still blocking the playbook because they retried until success and exhausted their retry budget during registry TLS timeouts. Keep the image pulls as opportunistic cache warmers, but never let them fail the bootstrap; log any missed images instead.	2026-04-23 06:41:21 +00:00
micqdf	ee6417c18e	fix: pre-pull core bootstrap images on cp1 before Flux bootstrap ... Fresh clusters were repeatedly timing out while kubelet pulled the pause image, k3s packaged component images, and Flux controller images onto the first control plane. Pre-pull the core control-plane bootstrap images into containerd on cp-1 so Flux and packaged addons start from a warm cache instead of racing registry TLS timeouts.	2026-04-23 05:55:14 +00:00
micqdf	1156dc0203	fix: pre-pull kube-vip images before waiting for VIP ... The primary control plane was stalling because kubelet still had to pull both the Rancher pause image and the kube-vip image before the DaemonSet pod could become Ready. Pre-pull those images into containerd, extend the readiness wait, and emit pod diagnostics if kube-vip still does not come up.	2026-04-23 03:55:52 +00:00
micqdf	4151027e01	fix: clean stale Tailscale node devices before bootstrap ... Run the Tailscale cleanup role against the cluster hostnames before any node reconnects to the tailnet. This removes stale offline cp/worker devices from previous rebuilds so replacement VMs can reclaim their original hostnames instead of getting -1 suffixes.	2026-04-23 03:25:17 +00:00
micqdf	9269e9df1b	docs: add guide for deploying app repos to the cluster ... Document the recommended two-repo model for application delivery, including Flux attachment objects, Doppler/ExternalSecret wiring, Tailscale service exposure, and the steps for enabling the suspended apps layer.	2026-04-23 02:43:00 +00:00
micqdf	d9374bc209	fix: remove duplicate wait keys from helm addon kustomizations ... The repo-only Kustomization healthCheck change accidentally left the original wait:true keys in the Rancher and Rancher backup Kustomizations, which broke the infrastructure kustomize build. Remove the duplicate keys so Flux can apply the HelmRelease-only health checks cleanly.	2026-04-23 02:20:57 +00:00
micqdf	c570a476b5	fix: make helm-based addon kustomizations health-check HelmReleases only ... These addon Kustomizations were using wait=true, which made Flux treat transient HelmRepository fetch timeouts as addon failures even when the HelmRelease and runtime workloads were healthy. Switch the affected Kustomizations to explicit HelmRelease healthChecks so readiness reflects the actual deployed platform state instead of repository fetch flakiness.	2026-04-23 02:15:45 +00:00
micqdf	a7f11ccf94	fix: give Rancher more time to pass startup probe during upgrades ... Rancher needs longer than the chart default 2-minute startup probe budget on this cluster while it restores local catalogs and finishes API startup. Extend the startup probe failure threshold so Helm upgrades can complete instead of restarting the new pod before it becomes ready.	2026-04-23 01:44:25 +00:00
micqdf	a7d540ca65	fix: stop forcing Flux releases during deploy bootstrap ... Remove the HelmRelease reset/force annotations from the deploy workflow now that the cluster can converge on its own. The runtime waits remain, but CI no longer re-triggers Rancher and NFS churn on every bootstrap attempt.	2026-04-23 00:35:31 +00:00
micqdf	098bd98876	fix: wait on Rancher and storage runtime objects during bootstrap ... Flux can leave HelmRelease and Kustomization conditions stale after transient chart fetch or image pull failures even when the underlying workloads recover. Switch the deploy workflow to wait on the concrete runtime resources we care about: the NFS provisioner deployment and StorageClass, Rancher deployment, webhook, cert-manager issuer/certificate, and the rancher-backup deployment.	2026-04-22 18:41:09 +00:00
micqdf	55d7b8201e	fix: make Rancher image pre-pull best effort and disable managed SUC ... Docker Hub TLS handshakes are too flaky to make pre-pulling a hard bootstrap requirement. Treat image pre-pull as opportunistic and disable Rancher's managed system-upgrade-controller feature so that image is removed from the critical install path while Rancher and its webhook converge.	2026-04-22 11:33:13 +00:00
micqdf	9c0523e880	fix: pre-pull Rancher images and reset Rancher release during bootstrap ... Rancher installs were stalling on transient Docker Hub TLS handshake timeouts for rancher shell, webhook, and system-upgrade-controller images. Pre-pull the required images onto all nodes after k3s comes up, extend the Rancher HelmRelease timeout, and reset/force the Rancher HelmRelease before waiting on addon-rancher so bootstrap can recover from stale failed remediation state.	2026-04-22 11:00:54 +00:00
micqdf	8372d562ad	fix: reset and force nfs helmrelease during bootstrap ... When the NFS storage HelmRelease has already entered a failed remediation state, a plain reconcile request is not enough to clear the stale failure counters. Send requestedAt, resetAt, and forceAt together so helm-controller retries the release cleanly before the workflow waits on addon-nfs-storage.	2026-04-22 10:35:32 +00:00
micqdf	1bb11dfe3a	fix: force nfs storage reconcile during flux bootstrap ... The NFS HelmRelease can remain in a failed state from an earlier bootstrap attempt even after the backing NFS export is corrected and the pod becomes healthy. Request a fresh reconcile of the HelmRelease and addon kustomization before waiting on addon-nfs-storage so the bootstrap step can observe the recovered state.	2026-04-22 10:08:20 +00:00
micqdf	624cd5aab6	fix: point NFS provisioner at active Proxmox host export ... The cluster nodes can reach the exported NFS path on 10.27.27.239, not 10.27.27.22. Update the storage addon and repo note so the NFS provisioner mounts the live export and Flux health checks can converge.	2026-04-22 09:46:01 +00:00