98 Commits

Author	SHA1	Message	Date
micqdf	9879de5a86	fix: stop pre-pulling Rancher child images	2026-04-26 00:57:49 +00:00
micqdf	195e9bce25	fix: parallelize Rancher child image warmup	2026-04-26 00:02:12 +00:00
micqdf	4796606432	fix: warm Rancher child images on all nodes	2026-04-25 23:30:20 +00:00
micqdf	f3c96b65d2	fix: shorten Rancher chart retry windows	2026-04-25 22:30:07 +00:00
micqdf	c7a375758f	fix: retry Rancher chart pulls during waits	2026-04-25 22:03:09 +00:00
micqdf	40647318b4	fix: tolerate cached Helm repository artifacts	2026-04-25 20:44:03 +00:00
micqdf	cdb26904d2	fix: retry Tailscale chart pulls during bootstrap	2026-04-25 20:11:43 +00:00
micqdf	3c06e046c2	fix: warm External Secrets image before install	2026-04-25 19:46:21 +00:00
micqdf	17f1815e7f	fix: use CRI pulls for Flux image warmup	2026-04-25 19:28:29 +00:00
micqdf	66e86e55ea	fix: require Flux image warmup before bootstrap	2026-04-25 19:02:32 +00:00
micqdf	43df412243	fix: handle missing Proxmox VM config during cleanup	2026-04-25 17:40:51 +00:00
micqdf	383ef9e9ac	fix: clean orphan Proxmox cloud-init volumes	2026-04-25 17:38:57 +00:00
micqdf	18abc5073b	fix: keep concurrent Terraform apply	2026-04-25 17:30:59 +00:00
micqdf	f8da2594ca	fix: serialize Proxmox VM apply	2026-04-25 17:27:59 +00:00
micqdf	003333a061	fix: make health checks observe Flux readiness	2026-04-25 03:52:43 +00:00
micqdf	a6071c504b	fix: point Promtail at Loki service	2026-04-25 03:43:23 +00:00
micqdf	08123457f1	fix: ignore stale install hook pods in health check	2026-04-25 03:41:00 +00:00
micqdf	15defc686f	fix: allow slow Promtail image pulls	2026-04-25 03:10:47 +00:00
micqdf	abb7578328	fix: run post-deploy checks with bash	2026-04-25 02:42:54 +00:00
micqdf	045880bdd6	fix: ignore stale Rancher helm operation pods	2026-04-25 02:23:30 +00:00
micqdf	bfcf57bcc5	fix: enforce post-deploy health checks	2026-04-25 02:22:16 +00:00
micqdf	7e3ebec95b	fix: wait for Rancher resources before rollout checks	2026-04-25 01:54:21 +00:00
micqdf	0c31c3b1d5	fix: fail fast on stalled Flux Helm releases	2026-04-25 01:40:42 +00:00
micqdf	5523feb563	fix: wait for Rancher Flux resources before rollout	2026-04-25 00:59:16 +00:00
micqdf	cafa2fa0b3	fix: reset stalled bootstrap Helm releases	2026-04-25 00:48:33 +00:00
micqdf	a7fd4c0b97	fix: wait on actual ESO deployment names	2026-04-25 00:07:48 +00:00
micqdf	e56a3a6c38	fix: wait for ESO webhook before ClusterSecretStore	2026-04-24 23:13:03 +00:00
micqdf	7b2eca07ab	fix: pull external-secrets chart from OCI	2026-04-24 15:24:58 +00:00
micqdf	347ca041ba	fix: reduce rerun bootstrap pre-pull delays	2026-04-24 12:09:34 +00:00
micqdf	68b293efe4	fix: qualify Flux HelmChart bootstrap resources	2026-04-24 10:47:13 +00:00
micqdf	1f465cc0c1	fix: force reconcile bootstrap Helm charts	2026-04-24 10:17:49 +00:00
micqdf	6e22bd26b3	fix: wait directly on ESO Helm readiness	2026-04-23 22:09:45 +00:00
micqdf	869880c152	fix: wait for ESO resources before CRD conditions	2026-04-23 21:17:44 +00:00
micqdf	31e95eb227	fix: pre-pull Flux controllers before bootstrap rollout	2026-04-23 20:36:57 +00:00
micqdf	12675417bd	fix: use correct namespace and deployment name for ESO rollout check ... The ESO deployment is named external-secrets-external-secrets in the external-secrets namespace, not external-secrets in kube-system.	2026-04-23 19:00:15 +00:00
micqdf	8e081ddfda	fix: wait on ESO deployment directly instead of Flux Kustomization status ... The addon-external-secrets Flux Kustomization was timing out during bootstrap because image pulls on fresh Proxmox VMs are slow. The critical dependency is the ESO deployment being available for the Doppler ClusterSecretStore. Replace the Kustomization readiness check with direct checks for ESO CRD establishment and deployment rollout, which are the actual prerequisites for the next step.	2026-04-23 07:32:19 +00:00
micqdf	a7d540ca65	fix: stop forcing Flux releases during deploy bootstrap ... Remove the HelmRelease reset/force annotations from the deploy workflow now that the cluster can converge on its own. The runtime waits remain, but CI no longer re-triggers Rancher and NFS churn on every bootstrap attempt.	2026-04-23 00:35:31 +00:00
micqdf	098bd98876	fix: wait on Rancher and storage runtime objects during bootstrap ... Flux can leave HelmRelease and Kustomization conditions stale after transient chart fetch or image pull failures even when the underlying workloads recover. Switch the deploy workflow to wait on the concrete runtime resources we care about: the NFS provisioner deployment and StorageClass, Rancher deployment, webhook, cert-manager issuer/certificate, and the rancher-backup deployment.	2026-04-22 18:41:09 +00:00
micqdf	9c0523e880	fix: pre-pull Rancher images and reset Rancher release during bootstrap ... Rancher installs were stalling on transient Docker Hub TLS handshake timeouts for rancher shell, webhook, and system-upgrade-controller images. Pre-pull the required images onto all nodes after k3s comes up, extend the Rancher HelmRelease timeout, and reset/force the Rancher HelmRelease before waiting on addon-rancher so bootstrap can recover from stale failed remediation state.	2026-04-22 11:00:54 +00:00
micqdf	8372d562ad	fix: reset and force nfs helmrelease during bootstrap ... When the NFS storage HelmRelease has already entered a failed remediation state, a plain reconcile request is not enough to clear the stale failure counters. Send requestedAt, resetAt, and forceAt together so helm-controller retries the release cleanly before the workflow waits on addon-nfs-storage.	2026-04-22 10:35:32 +00:00
micqdf	1bb11dfe3a	fix: force nfs storage reconcile during flux bootstrap ... The NFS HelmRelease can remain in a failed state from an earlier bootstrap attempt even after the backing NFS export is corrected and the pod becomes healthy. Request a fresh reconcile of the HelmRelease and addon kustomization before waiting on addon-nfs-storage so the bootstrap step can observe the recovered state.	2026-04-22 10:08:20 +00:00
micqdf	71bdc6a709	fix: extend Flux bootstrap timeouts on fresh clusters ... Fresh Proxmox clusters need longer for the Flux controller rollouts and first GitRepository/Kustomization reconciliations, especially while images are still being pulled onto the control plane. Increase the bootstrap wait windows so CI does not fail while the controllers are still converging.	2026-04-22 08:36:27 +00:00
micqdf	714f20417b	fix: tolerate control-plane taint when pinning Flux to cp1 ... Flux bootstrap patches the controllers onto k8s-cluster-cp-1, but the control-plane node is tainted NoSchedule. Add the matching toleration in both the checked-in patch manifest and the bootstrap workflow so the controllers can actually schedule and roll out on cp-1.	2026-04-22 05:05:15 +00:00
micqdf	5c53b8e06e	fix: normalize Proxmox endpoint and stop dashboards self-trigger ... Accept Proxmox API endpoints with or without /api2/json in CI and local tfvars, and avoid running the dashboards workflow just because its own workflow file changed during platform migrations.	2026-04-22 03:13:22 +00:00
micqdf	b1dae28aa5	feat: migrate cluster baseline from Hetzner to Proxmox ... Replace Hetzner infrastructure and cloud-provider assumptions with Proxmox VM clones, kube-vip API HA, and NFS-backed storage. Update bootstrap, Flux addons, CI workflows, and docs to target the new private Proxmox baseline while preserving the existing Tailscale, Doppler, Flux, Rancher, and B2 backup flows.	2026-04-22 03:02:13 +00:00
micqdf	7385c2263e	fix: add tailnet smoke checks and move Tailscale operator to stable ... Add a post-deploy smoke test that validates Tailscale DNS, proxy readiness, reachability, and service responses for Rancher, Grafana, and Prometheus. Move the operator to the stable Helm repo/version and align the baseline docs with the current HA private-only architecture.	2026-04-18 19:59:13 +00:00
micqdf	2ba6b6a896	fix: remove unused Flux CLI install from deploy workflow ... The deploy pipeline never uses the flux binary after installation, so the GitHub release download only adds a flaky failure point. Remove the step and keep the bootstrap path kubectl-only.	2026-04-18 17:45:59 +00:00
micqdf	ceefcc3b29	cleanup: Remove obsolete port-forwarding, deferred Traefik files, and CI workaround ... - Remove ansible/roles/private-access/ (replaced by Tailscale LB services) - Remove deferred observability ingress/traefik files (replaced by direct Tailscale LBs) - Remove orphaned kustomization-traefik-config.yaml (no backing directory) - Simplify CI: remove SA patch + job deletion workaround for rancher-backup (now handled by postRenderer in HelmRelease) - Update AGENTS.md to reflect current architecture	2026-04-02 01:21:23 +00:00
micqdf	89e53d9ec9	fix: Handle restricted B2 keys and safe JSON parsing in restore step	2026-03-31 01:43:04 +00:00
micqdf	5a2551f40a	fix: Fix flux CLI download URL - use correct GitHub URL with v prefix on version	2026-03-30 03:11:40 +00:00