338 Commits

This Branch

This Branch

All Branches

Author	SHA1	Message	Date
micqdf	b1eab6a0fa	fix: vendor Rancher chart for bootstrap	2026-04-25 23:08:26 +00:00
micqdf	f3c96b65d2	fix: shorten Rancher chart retry windows	2026-04-25 22:30:07 +00:00
micqdf	c7a375758f	fix: retry Rancher chart pulls during waits	2026-04-25 22:03:09 +00:00
micqdf	d0be48b65c	fix: gate Tailscale addon on Helm release	2026-04-25 21:21:34 +00:00
micqdf	40647318b4	fix: tolerate cached Helm repository artifacts	2026-04-25 20:44:03 +00:00
micqdf	cdb26904d2	fix: retry Tailscale chart pulls during bootstrap	2026-04-25 20:11:43 +00:00
micqdf	3c06e046c2	fix: warm External Secrets image before install	2026-04-25 19:46:21 +00:00
micqdf	17f1815e7f	fix: use CRI pulls for Flux image warmup	2026-04-25 19:28:29 +00:00
micqdf	66e86e55ea	fix: require Flux image warmup before bootstrap	2026-04-25 19:02:32 +00:00
micqdf	43df412243	fix: handle missing Proxmox VM config during cleanup	2026-04-25 17:40:51 +00:00
micqdf	383ef9e9ac	fix: clean orphan Proxmox cloud-init volumes	2026-04-25 17:38:57 +00:00
micqdf	18abc5073b	fix: keep concurrent Terraform apply	2026-04-25 17:30:59 +00:00
micqdf	f8da2594ca	fix: serialize Proxmox VM apply	2026-04-25 17:27:59 +00:00
micqdf	e0359f0097	tes	2026-04-25 17:22:12 +00:00
micqdf	003333a061	fix: make health checks observe Flux readiness	2026-04-25 03:52:43 +00:00
micqdf	a6071c504b	fix: point Promtail at Loki service	2026-04-25 03:43:23 +00:00
micqdf	08123457f1	fix: ignore stale install hook pods in health check	2026-04-25 03:41:00 +00:00
micqdf	757d88ed52	fix: use cached Promtail images when available	2026-04-25 03:25:44 +00:00
micqdf	15defc686f	fix: allow slow Promtail image pulls	2026-04-25 03:10:47 +00:00
micqdf	abb7578328	fix: run post-deploy checks with bash	2026-04-25 02:42:54 +00:00
micqdf	bc87a7ca43	fix: avoid immutable observability PVC changes	2026-04-25 02:25:40 +00:00
micqdf	045880bdd6	fix: ignore stale Rancher helm operation pods	2026-04-25 02:23:30 +00:00
micqdf	bfcf57bcc5	fix: enforce post-deploy health checks	2026-04-25 02:22:16 +00:00
micqdf	7e3ebec95b	fix: wait for Rancher resources before rollout checks	2026-04-25 01:54:21 +00:00
micqdf	0c31c3b1d5	fix: fail fast on stalled Flux Helm releases	2026-04-25 01:40:42 +00:00
micqdf	5523feb563	fix: wait for Rancher Flux resources before rollout	2026-04-25 00:59:16 +00:00
micqdf	cafa2fa0b3	fix: reset stalled bootstrap Helm releases	2026-04-25 00:48:33 +00:00
micqdf	a7fd4c0b97	fix: wait on actual ESO deployment names	2026-04-25 00:07:48 +00:00
micqdf	e56a3a6c38	fix: wait for ESO webhook before ClusterSecretStore	2026-04-24 23:13:03 +00:00
micqdf	7b2eca07ab	fix: pull external-secrets chart from OCI	2026-04-24 15:24:58 +00:00
micqdf	347ca041ba	fix: reduce rerun bootstrap pre-pull delays	2026-04-24 12:09:34 +00:00
micqdf	3f52bad854	fix: make Ansible reruns faster and idempotent	2026-04-24 11:44:11 +00:00
micqdf	c89c31adea	fix: clean up Ansible bootstrap warnings	2026-04-24 11:07:13 +00:00
micqdf	68b293efe4	fix: qualify Flux HelmChart bootstrap resources	2026-04-24 10:47:13 +00:00
micqdf	1f465cc0c1	fix: force reconcile bootstrap Helm charts	2026-04-24 10:17:49 +00:00
micqdf	6e22bd26b3	fix: wait directly on ESO Helm readiness	2026-04-23 22:09:45 +00:00
micqdf	869880c152	fix: wait for ESO resources before CRD conditions	2026-04-23 21:17:44 +00:00
micqdf	31e95eb227	fix: pre-pull Flux controllers before bootstrap rollout	2026-04-23 20:36:57 +00:00
micqdf	12675417bd	fix: use correct namespace and deployment name for ESO rollout check ... The ESO deployment is named external-secrets-external-secrets in the external-secrets namespace, not external-secrets in kube-system.	2026-04-23 19:00:15 +00:00
micqdf	8e081ddfda	fix: wait on ESO deployment directly instead of Flux Kustomization status ... The addon-external-secrets Flux Kustomization was timing out during bootstrap because image pulls on fresh Proxmox VMs are slow. The critical dependency is the ESO deployment being available for the Doppler ClusterSecretStore. Replace the Kustomization readiness check with direct checks for ESO CRD establishment and deployment rollout, which are the actual prerequisites for the next step.	2026-04-23 07:32:19 +00:00
micqdf	4b7517c9c5	fix: health-check external-secrets addon via HelmRelease only ... The external-secrets Kustomization was still using wait=true, which makes Flux hold the addon in a failed state when the HelmRepository has transient fetch errors even though the HelmRelease and runtime controller deployments are healthy. Switch it to an explicit HelmRelease health check like the other helm-backed addons.	2026-04-23 07:11:21 +00:00
micqdf	f9bc53723f	fix: make image pre-pull roles fully best effort ... The pre-pull roles were still blocking the playbook because they retried until success and exhausted their retry budget during registry TLS timeouts. Keep the image pulls as opportunistic cache warmers, but never let them fail the bootstrap; log any missed images instead.	2026-04-23 06:41:21 +00:00
micqdf	ee6417c18e	fix: pre-pull core bootstrap images on cp1 before Flux bootstrap ... Fresh clusters were repeatedly timing out while kubelet pulled the pause image, k3s packaged component images, and Flux controller images onto the first control plane. Pre-pull the core control-plane bootstrap images into containerd on cp-1 so Flux and packaged addons start from a warm cache instead of racing registry TLS timeouts.	2026-04-23 05:55:14 +00:00
micqdf	1156dc0203	fix: pre-pull kube-vip images before waiting for VIP ... The primary control plane was stalling because kubelet still had to pull both the Rancher pause image and the kube-vip image before the DaemonSet pod could become Ready. Pre-pull those images into containerd, extend the readiness wait, and emit pod diagnostics if kube-vip still does not come up.	2026-04-23 03:55:52 +00:00
micqdf	4151027e01	fix: clean stale Tailscale node devices before bootstrap ... Run the Tailscale cleanup role against the cluster hostnames before any node reconnects to the tailnet. This removes stale offline cp/worker devices from previous rebuilds so replacement VMs can reclaim their original hostnames instead of getting -1 suffixes.	2026-04-23 03:25:17 +00:00
micqdf	9269e9df1b	docs: add guide for deploying app repos to the cluster ... Document the recommended two-repo model for application delivery, including Flux attachment objects, Doppler/ExternalSecret wiring, Tailscale service exposure, and the steps for enabling the suspended apps layer.	2026-04-23 02:43:00 +00:00
micqdf	d9374bc209	fix: remove duplicate wait keys from helm addon kustomizations ... The repo-only Kustomization healthCheck change accidentally left the original wait:true keys in the Rancher and Rancher backup Kustomizations, which broke the infrastructure kustomize build. Remove the duplicate keys so Flux can apply the HelmRelease-only health checks cleanly.	2026-04-23 02:20:57 +00:00
micqdf	c570a476b5	fix: make helm-based addon kustomizations health-check HelmReleases only ... These addon Kustomizations were using wait=true, which made Flux treat transient HelmRepository fetch timeouts as addon failures even when the HelmRelease and runtime workloads were healthy. Switch the affected Kustomizations to explicit HelmRelease healthChecks so readiness reflects the actual deployed platform state instead of repository fetch flakiness.	2026-04-23 02:15:45 +00:00
micqdf	a7f11ccf94	fix: give Rancher more time to pass startup probe during upgrades ... Rancher needs longer than the chart default 2-minute startup probe budget on this cluster while it restores local catalogs and finishes API startup. Extend the startup probe failure threshold so Helm upgrades can complete instead of restarting the new pod before it becomes ready.	2026-04-23 01:44:25 +00:00
micqdf	a7d540ca65	fix: stop forcing Flux releases during deploy bootstrap ... Remove the HelmRelease reset/force annotations from the deploy workflow now that the cluster can converge on its own. The runtime waits remain, but CI no longer re-triggers Rancher and NFS churn on every bootstrap attempt.	2026-04-23 00:35:31 +00:00