OpenStaticFish/HetznerTerra

Fork 0

Files

T

micqdf c3a2f25c94

Deploy Cluster / Terraform (push) Successful in 2m11s

Details

Deploy Cluster / Ansible (push) Successful in 10m9s

Details

docs: record validated Rancher restore drill

Update the baseline to treat Rancher backup and restore validation as part
of the accepted platform state, and capture the successful live drill run
performed on 2026-04-18.

2026-04-18 21:27:42 +00:00

2.9 KiB

Raw Blame History

Stable Private-Only Baseline

This document defines the current engineering target for this repository.

Topology

3 control planes (HA etcd cluster)
3 workers
Hetzner Load Balancer for Kubernetes API
private Hetzner network
Tailscale operator access and service exposure
Rancher exposed through Tailscale (rancher.silverside-gopher.ts.net)
Grafana exposed through Tailscale (grafana.silverside-gopher.ts.net)
Prometheus exposed through Tailscale (prometheus.silverside-gopher.ts.net:9090)
apps Kustomization suspended by default

In Scope

Terraform infrastructure bootstrap
Ansible k3s bootstrap with external cloud provider
HA control plane (3 nodes with etcd quorum)
Hetzner Load Balancer for Kubernetes API
Hetzner CCM deployed via Ansible (before workers join)
Hetzner CSI for persistent volumes (via Flux)
Flux core reconciliation
External Secrets Operator with Doppler
Tailscale private access and smoke-check validation
cert-manager
Rancher and rancher-backup
Rancher backup/restore validation
Observability stack (Grafana, Prometheus, Loki, Promtail)
Persistent volume provisioning validated

Deferred for Later Phases

app workloads in apps/

Out of Scope

public ingress or DNS
public TLS
app workloads
cross-region / multi-cluster disaster recovery strategy
upgrade strategy

Phase Gates

Terraform apply completes for HA topology (3 CP, 3 workers, 1 LB).
Load Balancer is healthy with all 3 control plane targets.
Primary control plane bootstraps with --cluster-init.
Secondary control planes join via Load Balancer endpoint.
CCM deployed via Ansible before workers join (fixes uninitialized taint issue).
Workers join successfully via Load Balancer and all nodes show proper providerID.
etcd reports 3 healthy members.
Flux source and infrastructure reconciliation are healthy.
CSI deploys and creates hcloud-volumes StorageClass.
PVC provisioning tested and working.
External Secrets sync required secrets.
Tailscale private access works for Rancher, Grafana, and Prometheus.
CI smoke checks pass for Tailscale DNS resolution, tailscale ping, and HTTP reachability.
A fresh Rancher backup can be created and restored successfully.
Terraform destroy succeeds cleanly or via workflow retry.

Success Criteria

Success requires two consecutive HA rebuilds passing all phase gates with no manual fixes, no manual kubectl patching, and no manual Tailscale proxy recreation.

Validated Drills

2026-04-18: live Rancher backup/restore drill succeeded on the current cluster.
A fresh one-time backup was created, restored back onto the same cluster, and post-restore validation confirmed:
- all nodes remained Ready
- Flux infrastructure stayed healthy
- Rancher backup/restore resources reported Completed
- Rancher, Grafana, and Prometheus remained reachable through the Tailscale smoke checks

2.9 KiB Raw Blame History