From f52e657f9f70e410618c0386735736892207d4b3 Mon Sep 17 00:00:00 2001
From: MichaelFisher1997 <contact@michaelfisher.tech>
Date: Thu, 30 Apr 2026 07:03:21 +0000
Subject: [PATCH] docs

---
 AGENTS.md              |  77 ++++---
 README.md              | 490 +++++++++++++++--------------------------
 ansible/inventory.tmpl |   1 -
 3 files changed, 221 insertions(+), 347 deletions(-)

diff --git a/AGENTS.md b/AGENTS.md
index 5c4bc8f..4105c07 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -1,52 +1,57 @@
 # AGENTS.md
 
-Repository guide for OpenCode sessions in this repo.
+Compact repo guidance for OpenCode sessions. Trust executable sources over docs when they conflict.
 
 ## Read First
 
-- Trust manifests and workflows over prose when they conflict.
-- Highest-value sources: `terraform/main.tf`, `terraform/variables.tf`, `ansible/site.yml`, `clusters/prod/flux-system/`, `infrastructure/addons/kustomization.yaml`, `.gitea/workflows/deploy.yml`, `.gitea/workflows/destroy.yml`, `README.md`, `STABLE_BASELINE.md`, `scripts/refresh-kubeconfig.sh`, `scripts/smoke-check-tailnet-services.sh`.
+- Highest-value sources: `.gitea/workflows/deploy.yml`, `.gitea/workflows/destroy.yml`, `terraform/main.tf`, `terraform/variables.tf`, `terraform/servers.tf`, `ansible/site.yml`, `ansible/inventory.tmpl`, `clusters/prod/flux-system/`, `infrastructure/addons/kustomization.yaml`.
+- `STABLE_BASELINE.md` still contains stale Rancher backup/restore references; current workflows and addon manifests do not deploy or restore `rancher-backup`.
 
-## Current Baseline
+## Baseline
 
-- HA private cluster: 3 control planes, 5 workers on Proxmox.
-- Proxmox clones come from template `9000` on node `flex`; API VIP is `10.27.27.40` via kube-vip.
-- Storage is `nfs-subdir-external-provisioner` backed by `10.27.27.239:/TheFlash/k8s-nfs` with StorageClass `flash-nfs`.
-- Tailscale is the private access path for Rancher and shared services.
-- Rancher, Grafana, and Prometheus are exposed through Tailscale; Flux UI / Weave GitOps is removed.
-- `apps/` is suspended by default.
-- Rancher stores state in embedded etcd; backup/restore uses `rancher-backup` to B2.
+- Proxmox HA K3s cluster: 3 control planes, 5 workers, VMIDs `200-202` and `210-214`, node `flex`, template VMID `9000`, datastore `Flash`.
+- API HA is kube-vip at `10.27.27.40`; control planes are `10.27.27.30-32`, workers are `10.27.27.41-45`.
+- SSH user is `ubuntu`; Ansible derives the flannel iface from `ansible_default_ipv4.interface` with `eth0` fallback, so do not hard-code `ens18`.
+- Storage is raw-manifest `nfs-subdir-external-provisioner` using `10.27.27.239:/TheFlash/k8s-nfs` and default StorageClass `flash-nfs`.
+- Tailscale is the private access path. Rancher, Grafana, and Prometheus are exposed only through Tailscale services.
+- `apps` is intentionally suspended in `clusters/prod/flux-system/kustomization-apps.yaml`.
 
-## Common Commands
+## Commands
 
-- Terraform: `terraform -chdir=terraform fmt -recursive`, `terraform -chdir=terraform validate`, `terraform -chdir=terraform plan -var-file=../terraform.tfvars`, `terraform -chdir=terraform apply -var-file=../terraform.tfvars`
-- Ansible: `ansible-galaxy collection install -r ansible/requirements.yml`, `cd ansible && python3 generate_inventory.py`, `ansible-playbook -i ansible/inventory.ini ansible/site.yml --syntax-check`, `ansible-playbook ansible/site.yml`
-- Flux/Kustomize: `kubectl kustomize infrastructure/addons/<addon>`, `kubectl kustomize clusters/prod/flux-system`
-- Kubeconfig refresh: `scripts/refresh-kubeconfig.sh <cp1-ip>`
-- Tailnet smoke check: `ssh ubuntu@<cp1-ip> 'bash -s' < scripts/smoke-check-tailnet-services.sh`
+- Terraform: `terraform -chdir=terraform fmt -recursive`, `terraform -chdir=terraform validate`, `terraform -chdir=terraform plan -var-file=../terraform.tfvars`, `terraform -chdir=terraform apply -var-file=../terraform.tfvars`.
+- Ansible setup: `ansible-galaxy collection install -r ansible/requirements.yml`, then from `ansible/` run `python3 generate_inventory.py` and `ansible-playbook site.yml --syntax-check`.
+- Flux/Kustomize checks: `kubectl kustomize infrastructure/addons/<addon>`, `kubectl kustomize infrastructure/addons`, `kubectl kustomize clusters/prod/flux-system`.
+- Kubeconfig refresh: `scripts/refresh-kubeconfig.sh <cp1-ip>`; use this if local `kubectl` falls back to `localhost:8080` after rebuilds.
+- Tailnet smoke check from cp1: `ssh ubuntu@<cp1-ip> 'bash -s' < scripts/smoke-check-tailnet-services.sh`.
+- Fast Grafana content iteration uses `.gitea/workflows/dashboards.yml` and `ansible/dashboards.yml`, not a full cluster rebuild.
 
-## Workflow Rules
+## Deploy Flow
 
-- Keep diffs small and validate only the directory you edited.
-- Update manifests and docs together when behavior changes.
-- Use `set -euo pipefail` in workflow shell blocks.
-- CI deploy order is Terraform -> Ansible -> Flux bootstrap -> Rancher restore -> health checks.
-- One object per Kubernetes YAML file; keep filenames kebab-case.
-- If `kubectl` points at `localhost:8080` after a rebuild, refresh kubeconfig from the primary control-plane IP.
-- Bootstrap assumptions that matter: SSH user is `ubuntu`, NIC is `ens18`, API join endpoint is the kube-vip address.
+- Pushes to `main` run Gitea CI: Terraform fmt/init/validate/plan/apply, Proxmox cleanup/retry, Ansible bootstrap, Flux bootstrap, addon gates, Rancher gate, observability image seeding, health checks, tailnet smoke checks.
+- Deploy and destroy workflows share `concurrency.group: prod-cluster`; destroy only requires workflow input `confirm: destroy` and has no backup gate.
+- Keep `set -euo pipefail` in workflow shell blocks.
+- Terraform retry cleanup has hard-coded target VMIDs/names in `.gitea/workflows/deploy.yml`; update it when changing node counts, names, or VMIDs.
+- Fresh VMs have unreliable registry/chart egress, so critical images are prepared by `skopeo` on the runner and imported with `k3s ctr`; update the workflow archive lists when adding bootstrap-time images.
+- CI applies `clusters/prod/flux-system/gotk-components.yaml` directly and then patches Flux controller deployments inline; changes only in `gotk-controller-cp1-patches.yaml` do not affect CI bootstrap.
 
-## Repo-Specific Gotchas
+## GitOps Addons
 
-- `rancher-backup` uses a postRenderer to swap the broken hook image to `rancher/kubectl:v1.34.0`; do not put S3 config in HelmRelease values. Put it in the Backup CR.
-- Tailscale cleanup only runs before service proxies exist; it removes stale offline `rancher`/`grafana`/`prometheus`/`flux` devices, then must stop so live proxies are not deleted.
-- Keep the vendored Tailscale operator chart at `infrastructure/charts/tailscale-operator` pinned to `1.96.5`; do not restore the remote HelmRepository unless cluster-side chart fetches are reliable.
-- The repo no longer uses a cloud controller manager. If you see `providerID` or Hetzner-specific logic, it is stale.
-- Current private URLs:
-  - Rancher: `https://rancher.silverside-gopher.ts.net/`
-  - Grafana: `http://grafana.silverside-gopher.ts.net/`
-  - Prometheus: `http://prometheus.silverside-gopher.ts.net:9090/`
+- Vendored charts are intentional: `infrastructure/charts/{cert-manager,traefik,kube-prometheus-stack,tailscale-operator,rancher}`. Do not restore remote `HelmRepository` objects unless cluster-side chart fetch reliability is intentionally changed.
+- External Secrets and Loki/Promtail use Flux `OCIRepository`; Rancher, Tailscale, cert-manager, Traefik, and kube-prometheus-stack use `GitRepository` chart paths.
+- Use fully qualified `helmchart.source.toolkit.fluxcd.io/...` in scripts; K3s also has `helmcharts.helm.cattle.io`, so `helmchart/...` can target the wrong resource.
+- `doppler-bootstrap` only creates the `external-secrets` namespace and Doppler token secret. The deploy workflow creates `ClusterSecretStore/doppler-hetznerterra` after ESO CRDs and webhook endpoints exist.
+- The checked-in `infrastructure/addons/external-secrets/clustersecretstore-doppler-hetznerterra.yaml` is not included by that addon kustomization; do not assume Flux applies it.
+- Keep Kubernetes manifests one object per file with kebab-case filenames.
+
+## Gotchas
+
+- Rancher chart `2.13.3` requires Kubernetes `<1.35.0-0`; K3s `latest` can break Rancher. Role defaults pin `v1.34.6+k3s1`; do not reintroduce a generated-inventory `k3s_version=latest` override.
+- The repo no longer uses a cloud controller manager. `providerID`, Hetzner CCM/CSI, or Hetzner firewall/load-balancer logic is stale.
+- Tailscale cleanup must only remove stale offline reserved hostnames before live service proxies exist; do not delete active `rancher`, `grafana`, `prometheus`, or `flux` devices.
+- Proxmox endpoint should be the base URL, for example `https://100.105.0.115:8006/`; provider/workflow code strips `/api2/json` when needed.
+- Current private URLs: Rancher `https://rancher.silverside-gopher.ts.net/`, Grafana `http://grafana.silverside-gopher.ts.net/`, Prometheus `http://prometheus.silverside-gopher.ts.net:9090/`.
 
 ## Secrets
 
-- Runtime secrets live in Doppler + External Secrets.
-- Bootstrap and CI secrets stay in Gitea; never commit secrets, kubeconfigs, or private keys.
+- Runtime secrets are Doppler + External Secrets; Terraform/bootstrap/CI secrets stay in Gitea Actions secrets.
+- Never commit secrets, kubeconfigs, private keys, `terraform.tfvars`, or generated `outputs/` artifacts.
diff --git a/README.md b/README.md
index 4da6083..8fd324d 100644
--- a/README.md
+++ b/README.md
@@ -1,291 +1,268 @@
 # Proxmox Kubernetes Cluster
 
-Production-ready private Kubernetes cluster on Proxmox using Terraform, Ansible, and Flux.
+Private HA K3s cluster on Proxmox, provisioned by Terraform, bootstrapped by Ansible, and reconciled by Flux.
 
 ## Architecture
 
-| Component | Details |
-|-----------|---------|
-| **Control Plane** | 3x Proxmox VMs (2 vCPU / 4 GiB / 32 GiB) |
-| **Workers** | 5x Proxmox VMs (4 vCPU / 8 GiB / 64 GiB) |
-| **K8s** | k3s (latest, HA) |
-| **Addons** | NFS provisioner + Prometheus + Grafana + Loki + Rancher |
-| **Access** | SSH/API and private services restricted to Tailnet |
-| **Bootstrap** | Terraform + Ansible + Flux |
+| Component | Current Baseline |
+|-----------|------------------|
+| **Control plane** | 3 Proxmox VMs, VMIDs `200-202`, IPs `10.27.27.30-32`, 2 vCPU / 4 GiB / 32 GiB |
+| **Workers** | 5 Proxmox VMs, VMIDs `210-214`, IPs `10.27.27.41-45`, 4 vCPU / 8 GiB / 64 GiB |
+| **Kubernetes** | K3s `v1.34.6+k3s1`, HA embedded etcd, kube-vip API VIP `10.27.27.40` |
+| **Proxmox** | Node `flex`, template VMID `9000`, datastore `Flash`, bridge `vmbr0` |
+| **Storage** | Raw-manifest `nfs-subdir-external-provisioner`, `10.27.27.239:/TheFlash/k8s-nfs`, default StorageClass `flash-nfs` |
+| **GitOps** | Flux source `platform` on branch `main`; `apps` Kustomization is intentionally suspended |
+| **Private access** | Tailscale operator exposes Rancher, Grafana, and Prometheus; no public ingress baseline |
+| **Runtime secrets** | Doppler service token bootstraps External Secrets Operator |
+
+K3s is pinned because Rancher chart `2.13.3` requires Kubernetes `<1.35.0-0`.
 
 ## Prerequisites
 
-### 1. Proxmox API Token
+- Terraform `>= 1.0`.
+- Ansible with Python `jinja2` and `pyyaml`.
+- `kubectl` for local verification.
+- Proxmox API token for the `bpg/proxmox` provider.
+- S3-compatible bucket for Terraform state, currently Backblaze B2.
+- SSH key pair available to Terraform and Ansible, defaulting to `~/.ssh/infra` and `~/.ssh/infra.pub`.
 
-Create an API token for the Proxmox VE user used by Terraform. The repo expects the `bpg/proxmox` provider with:
+Expected Proxmox inputs:
 
-- endpoint: `https://100.105.0.115:8006/`
-- node: `flex`
-- clone source: template `9000` (`ubuntu-2404-k8s-template`)
-- auth: API token
+| Setting | Value |
+|---------|-------|
+| Endpoint | `https://100.105.0.115:8006/` |
+| Node | `flex` |
+| Clone source | Template VMID `9000` (`ubuntu-2404-k8s-template`) |
+| Storage | `Flash` |
 
-### 2. Backblaze B2 Bucket (for Terraform State)
+## Local Setup
 
-1. Go to [Backblaze B2](https://secure.backblaze.com/b2_buckets.htm)
-2. Click **Create a Bucket**
-3. Set bucket name: `k8s-terraform-state` (must be globally unique)
-4. Choose **Private** access
-5. Click **Create Bucket**
-6. Create application key:
-   - Go to **App Keys** → **Add a New Application Key**
-   - Name: `terraform-state`
-   - Allow access to: `k8s-terraform-state` bucket only
-   - Type: **Read and Write**
-   - Copy **keyID** (access key) and **applicationKey** (secret key)
-7. Note your bucket's S3 endpoint (e.g., `https://s3.eu-central-003.backblazeb2.com`)
-
-### 3. SSH Key Pair
-
-```bash
-ssh-keygen -t ed25519 -C "k8s@proxmox" -f ~/.ssh/infra
-```
-
-### 4. Local Tools
-
-- [Terraform](https://terraform.io/downloads) >= 1.0
-- [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html) >= 2.9
-- Python 3 with `jinja2` and `pyyaml`
-
-## Setup
-
-### 1. Clone Repository
-
-```bash
-git clone <your-gitea-repo>/HetznerTerra.git
-cd HetznerTerra
-```
-
-### 2. Configure Variables
+Create local variables from the example:
 
 ```bash
 cp terraform.tfvars.example terraform.tfvars
 ```
 
-Edit `terraform.tfvars`:
+Important defaults in `terraform.tfvars.example`:
 
 ```hcl
- proxmox_endpoint         = "https://100.105.0.115:8006/"
- proxmox_api_token_id     = "terraform-prov@pve!k8s-cluster"
- proxmox_api_token_secret = "your-proxmox-token-secret"
+proxmox_endpoint         = "https://100.105.0.115:8006/"
+proxmox_api_token_id     = "terraform-prov@pve!k8s-cluster"
+proxmox_api_token_secret = "your-proxmox-api-token-secret"
 
- ssh_public_key  = "~/.ssh/infra.pub"
- ssh_private_key = "~/.ssh/infra"
+ssh_public_key  = "~/.ssh/infra.pub"
+ssh_private_key = "~/.ssh/infra"
 
 s3_access_key = "your-backblaze-key-id"
 s3_secret_key = "your-backblaze-application-key"
 s3_endpoint   = "https://s3.eu-central-003.backblazeb2.com"
 s3_bucket     = "k8s-terraform-state"
 
-tailscale_auth_key = "tskey-auth-..."
-tailscale_tailnet  = "yourtailnet.ts.net"
-
- kube_api_vip = "10.27.27.40"
+tailscale_tailnet = "yourtailnet.ts.net"
+kube_api_vip     = "10.27.27.40"
 ```
 
-### 3. Initialize Terraform
+Initialize Terraform with backend credentials:
 
 ```bash
-cd terraform
-
-# Create backend config file (or use CLI args)
-cat > backend.hcl << EOF
-endpoint                    = "https://s3.eu-central-003.backblazeb2.com"
-bucket                      = "k8s-terraform-state"
-access_key                  = "your-backblaze-key-id"
-secret_key                  = "your-backblaze-application-key"
-skip_requesting_account_id  = true
-EOF
-
-terraform init -backend-config=backend.hcl
+terraform -chdir=terraform init \
+  -backend-config="endpoint=<s3-endpoint>" \
+  -backend-config="bucket=<s3-bucket>" \
+  -backend-config="region=auto" \
+  -backend-config="access_key=<s3-access-key>" \
+  -backend-config="secret_key=<s3-secret-key>" \
+  -backend-config="skip_requesting_account_id=true"
 ```
 
-### 4. Plan and Apply
+## Common Commands
+
+Terraform:
 
 ```bash
-terraform plan -var-file=../terraform.tfvars
-terraform apply -var-file=../terraform.tfvars
+terraform -chdir=terraform fmt -recursive
+terraform -chdir=terraform validate
+terraform -chdir=terraform plan -var-file=../terraform.tfvars
+terraform -chdir=terraform apply -var-file=../terraform.tfvars
 ```
 
-### 5. Generate Ansible Inventory
+Ansible setup:
 
 ```bash
-cd ../ansible
+ansible-galaxy collection install -r ansible/requirements.yml
+cd ansible
 python3 generate_inventory.py
+ansible-playbook site.yml --syntax-check
 ```
 
-### 6. Bootstrap Cluster
+Manual Ansible bootstrap uses the same extra vars as the deploy workflow:
 
 ```bash
-ansible-playbook site.yml
+cd ansible
+ansible-playbook site.yml \
+  -e "tailscale_auth_key=$TAILSCALE_AUTH_KEY" \
+  -e "tailscale_tailnet=$TAILSCALE_TAILNET" \
+  -e "tailscale_oauth_client_id=$TAILSCALE_OAUTH_CLIENT_ID" \
+  -e "tailscale_oauth_client_secret=$TAILSCALE_OAUTH_CLIENT_SECRET" \
+  -e "doppler_hetznerterra_service_token=$DOPPLER_HETZNERTERRA_SERVICE_TOKEN" \
+  -e "tailscale_api_key=${TAILSCALE_API_KEY:-}" \
+  -e "grafana_admin_password=${GRAFANA_ADMIN_PASSWORD:-}" \
+  -e "cluster_name=k8s-cluster"
 ```
 
-### 7. Get Kubeconfig
+Flux/Kustomize verification:
 
 ```bash
+kubectl kustomize infrastructure/addons/<addon>
+kubectl kustomize infrastructure/addons
+kubectl kustomize clusters/prod/flux-system
+```
+
+Refresh kubeconfig after rebuilds:
+
+```bash
+scripts/refresh-kubeconfig.sh 10.27.27.30
 export KUBECONFIG=$(pwd)/outputs/kubeconfig
 kubectl get nodes
 ```
 
-Use `scripts/refresh-kubeconfig.sh <cp1-public-ip>` to refresh kubeconfig against the primary control-plane public IP after rebuilds.
+Run the tailnet smoke check from cp1:
+
+```bash
+ssh ubuntu@10.27.27.30 'bash -s' < scripts/smoke-check-tailnet-services.sh
+```
 
 ## Gitea CI/CD
 
-This repository includes Gitea workflows for:
+The supported full rebuild path is the Gitea deploy workflow.
 
-- **deploy**: End-to-end Terraform + Ansible + Flux bootstrap + restore + health checks
-- **destroy**: Cluster teardown with backup-aware cleanup
-- **dashboards**: Fast workflow that updates Grafana datasources/dashboards only
+| Workflow | Trigger | Purpose |
+|----------|---------|---------|
+| `.gitea/workflows/deploy.yml` | PR to `main`, push to `main`, manual dispatch | PRs run Terraform plan; pushes run Terraform apply, Ansible bootstrap, Flux bootstrap, addon gates, health checks, and tailnet smoke checks |
+| `.gitea/workflows/destroy.yml` | Manual dispatch with `confirm: destroy` | Terraform destroy with retries; no Rancher backup gate |
+| `.gitea/workflows/dashboards.yml` | Grafana content changes or manual dispatch | Fast Grafana datasource/dashboard update through `ansible/dashboards.yml` |
 
-### Required Gitea Secrets
+Deploy and destroy share `concurrency.group: prod-cluster` so they do not run at the same time.
 
-Set these in your Gitea repository settings (**Settings** → **Secrets** → **Actions**):
+Deploy sequence on push to `main`:
+
+1. Terraform fmt/init/validate/plan/apply.
+2. Cleanup/retry around known transient Proxmox clone and disk-update failures.
+3. Generate Ansible inventory from Terraform outputs.
+4. Prepare critical image archives with `skopeo` on the runner.
+5. Run `ansible/site.yml` to bootstrap nodes, K3s, kube-vip, prerequisite secrets, and kubeconfig.
+6. Apply Flux CRDs/controllers and the `clusters/prod/flux-system` graph.
+7. Gate cert-manager, External Secrets, Tailscale, NFS, Rancher, and observability.
+8. Run post-deploy health checks and Tailscale service smoke checks.
+
+Required Gitea secrets:
 
 | Secret | Description |
 |--------|-------------|
-| `PROXMOX_ENDPOINT` | Proxmox API endpoint (for example `https://100.105.0.115:8006/`) |
+| `PROXMOX_ENDPOINT` | Proxmox API endpoint, for example `https://100.105.0.115:8006/` |
 | `PROXMOX_API_TOKEN_ID` | Proxmox API token ID |
 | `PROXMOX_API_TOKEN_SECRET` | Proxmox API token secret |
-| `S3_ACCESS_KEY` | Backblaze B2 keyID |
-| `S3_SECRET_KEY` | Backblaze B2 applicationKey |
-| `S3_ENDPOINT` | Backblaze S3 endpoint (e.g., `https://s3.eu-central-003.backblazeb2.com`) |
-| `S3_BUCKET` | S3 bucket name (e.g., `k8s-terraform-state`) |
+| `S3_ACCESS_KEY` | S3/Backblaze access key for Terraform state |
+| `S3_SECRET_KEY` | S3/Backblaze secret key for Terraform state |
+| `S3_ENDPOINT` | S3 endpoint, for example `https://s3.eu-central-003.backblazeb2.com` |
+| `S3_BUCKET` | Terraform state bucket, for example `k8s-terraform-state` |
 | `TAILSCALE_AUTH_KEY` | Tailscale auth key for node bootstrap |
-| `TAILSCALE_TAILNET` | Tailnet domain (e.g., `yourtailnet.ts.net`) |
-| `TAILSCALE_OAUTH_CLIENT_ID` | Tailscale OAuth client ID for Kubernetes Operator |
-| `TAILSCALE_OAUTH_CLIENT_SECRET` | Tailscale OAuth client secret for Kubernetes Operator |
-| `DOPPLER_HETZNERTERRA_SERVICE_TOKEN` | Doppler service token for `hetznerterra` runtime secrets |
-| `GRAFANA_ADMIN_PASSWORD` | Optional admin password for Grafana (auto-generated if unset) |
+| `TAILSCALE_TAILNET` | Tailnet domain, for example `silverside-gopher.ts.net` |
+| `TAILSCALE_OAUTH_CLIENT_ID` | Tailscale OAuth client ID for the Kubernetes operator |
+| `TAILSCALE_OAUTH_CLIENT_SECRET` | Tailscale OAuth client secret for the Kubernetes operator |
+| `TAILSCALE_API_KEY` | Optional API key used to delete stale offline reserved devices before service proxies exist |
+| `DOPPLER_HETZNERTERRA_SERVICE_TOKEN` | Doppler service token for runtime cluster secrets |
+| `GRAFANA_ADMIN_PASSWORD` | Optional Grafana admin password |
 | `SSH_PUBLIC_KEY` | SSH public key content |
 | `SSH_PRIVATE_KEY` | SSH private key content |
 
-## GitOps (Flux)
+## GitOps Graph
 
-This repo uses Flux for continuous reconciliation after Terraform + Ansible bootstrap.
+Flux entrypoint:
 
-### Stable private-only baseline
+```text
+clusters/prod/flux-system/
+├── gotk-components.yaml
+├── gitrepository-platform.yaml
+├── kustomization-infrastructure.yaml
+└── kustomization-apps.yaml  # suspend: true
+```
 
-The current default target is the HA private baseline:
+Active infrastructure addons from `infrastructure/addons/kustomization.yaml`:
 
-- `3` control plane nodes
-- `5` worker nodes
-- private Proxmox network only
-- Tailscale for operator and service access
-- Flux-managed platform addons with `apps` suspended by default
+- `addon-nfs-storage`
+- `addon-external-secrets`
+- `addon-cert-manager`
+- `addon-tailscale-operator`
+- `addon-tailscale-proxyclass`
+- `traefik` HelmRelease manifests applied directly by the top-level infrastructure Kustomization
+- `addon-observability`
+- `addon-observability-content`
+- `addon-rancher`
+- `addon-rancher-config`
 
-Detailed phase gates and success criteria live in `STABLE_BASELINE.md`.
+Chart/source strategy:
 
-This is the default until rebuilds are consistently green. High availability, public ingress, and app-layer expansion come later.
+- Vendored charts are intentional: `cert-manager`, `traefik`, `kube-prometheus-stack`, `tailscale-operator`, and `rancher` live under `infrastructure/charts/`.
+- External Secrets, Loki, and Promtail use Flux `OCIRepository` sources.
+- NFS storage is raw Kubernetes manifests, not a Helm chart.
+- Rancher backup/restore is not part of the current live graph.
 
-### Runtime secrets
+Doppler bootstrap details:
 
-Runtime cluster secrets are moving to Doppler + External Secrets Operator.
+- `ansible/roles/doppler-bootstrap` creates the `external-secrets` namespace and the Doppler token secret only.
+- The deploy workflow creates `ClusterSecretStore/doppler-hetznerterra` after ESO CRDs and webhook endpoints exist.
+- The checked-in `infrastructure/addons/external-secrets/clustersecretstore-doppler-hetznerterra.yaml` is not included by the addon kustomization.
 
-- Doppler project: `hetznerterra`
-- Initial auth: service token via `DOPPLER_HETZNERTERRA_SERVICE_TOKEN`
-- First synced secrets:
-  - `GRAFANA_ADMIN_PASSWORD`
+## Access URLs
 
-Terraform/bootstrap secrets remain in Gitea Actions secrets and are not managed by Doppler.
+| Service | URL |
+|---------|-----|
+| Rancher | `https://rancher.silverside-gopher.ts.net/` |
+| Grafana | `http://grafana.silverside-gopher.ts.net/` |
+| Prometheus | `http://prometheus.silverside-gopher.ts.net:9090/` |
 
-### Repository layout
-
-- `clusters/prod/`: cluster entrypoint and Flux reconciliation objects
-- `clusters/prod/flux-system/`: `GitRepository` source and top-level `Kustomization` graph
-- `infrastructure/`: infrastructure addon reconciliation graph
-- `infrastructure/addons/*`: per-addon manifests for Flux-managed cluster addons
-- `apps/`: application workload layer (currently scaffolded)
-
-### Reconciliation graph
-
-- `infrastructure` (top-level)
-  - `addon-nfs-storage`
-  - `addon-tailscale-operator`
-  - `addon-observability`
-  - `addon-observability-content` depends on `addon-observability`
-- `apps` depends on `infrastructure`
-
-### Bootstrap notes
-
-1. Install Flux controllers in `flux-system`.
-2. Create the Flux deploy key/secret named `flux-system` in `flux-system` namespace.
-3. Apply `clusters/prod/flux-system/` once to establish source + reconciliation graph.
-4. Bootstrap-only Ansible creates prerequisite secrets; Flux manages addon lifecycle after bootstrap.
-
-### Current addon status
-
-- Core infrastructure addons are Flux-managed from `infrastructure/addons/`.
-- Active Flux addons for the current baseline: `addon-nfs-storage`, `addon-cert-manager`, `addon-external-secrets`, `addon-tailscale-operator`, `addon-tailscale-proxyclass`, `addon-observability`, `addon-observability-content`, `addon-rancher`, `addon-rancher-config`, `addon-rancher-backup`, `addon-rancher-backup-config`.
-- `apps` remains suspended until workload rollout is explicitly enabled.
-- Ansible is limited to cluster bootstrap, prerequisite secret creation, pre-proxy Tailscale cleanup, and kubeconfig finalization.
-- Weave GitOps / Flux UI is no longer deployed; use Rancher or the `flux` CLI for Flux operations.
-
-### Rancher access
-
-- Rancher is private-only and exposed through Tailscale at `https://rancher.silverside-gopher.ts.net/`.
-- Rancher and the Kubernetes API stay private; kube-vip provides the API VIP on the LAN.
-- Rancher stores state in embedded etcd; no external database is used.
-
-### Stable baseline acceptance
-
-A rebuild is considered successful only when all of the following pass without manual intervention:
-
-- Terraform create succeeds for the default `3` control planes and `5` workers.
-- Ansible bootstrap succeeds end-to-end.
-- All nodes become `Ready`.
-- Flux core reconciliation is healthy.
-- External Secrets Operator is ready.
-- Tailscale operator is ready.
-- Tailnet smoke checks pass for Rancher, Grafana, and Prometheus.
-- Terraform destroy succeeds cleanly or succeeds after workflow retries.
-
-## Observability Stack
-
-Flux deploys a lightweight observability stack in the `observability` namespace:
-
-- `kube-prometheus-stack` (Prometheus + Grafana)
-- `loki`
-- `promtail`
-
-Grafana content is managed as code via ConfigMaps in `infrastructure/addons/observability-content/`.
-
-Grafana and Prometheus are exposed through dedicated Tailscale LoadBalancer services when the Tailscale Kubernetes Operator is healthy.
-
-### Access Grafana and Prometheus
-
-Preferred private access:
-
-- Grafana: `http://grafana.silverside-gopher.ts.net/`
-- Prometheus: `http://prometheus.silverside-gopher.ts.net:9090/`
-
-Fallback (port-forward from a tailnet-connected machine):
-
-Run from a tailnet-connected machine:
+Fallback port-forward from a tailnet-connected machine:
 
 ```bash
 export KUBECONFIG=$(pwd)/outputs/kubeconfig
-
 kubectl -n observability port-forward svc/kube-prometheus-stack-grafana 3000:80
 kubectl -n observability port-forward svc/kube-prometheus-stack-prometheus 9090:9090
 ```
 
-Then open:
+Grafana user is `admin`; password comes from the `GRAFANA_ADMIN_PASSWORD` Doppler secret or the workflow-provided fallback.
 
-- Grafana: http://127.0.0.1:3000
-- Prometheus: http://127.0.0.1:9090
+## Operations
 
-Grafana user: `admin`
-Grafana password: value of `GRAFANA_ADMIN_PASSWORD` secret (or the generated value shown by Ansible output)
+Scale workers by updating `terraform.tfvars` counts, IP lists, and VMID lists together. If node names or VMIDs change, also update the hard-coded retry cleanup target map in `.gitea/workflows/deploy.yml`.
 
-### Verify Tailscale exposure
+Upgrade K3s by changing the role defaults in `ansible/roles/k3s-server/defaults/main.yml` and `ansible/roles/k3s-agent/defaults/main.yml`. Check Rancher chart compatibility before moving to a Kubernetes minor outside `<1.35.0-0`.
+
+Destroy through the Gitea `Destroy` workflow with `confirm: destroy`, or locally with:
 
 ```bash
-export KUBECONFIG=$(pwd)/outputs/kubeconfig
+terraform -chdir=terraform destroy -var-file=../terraform.tfvars
+```
 
+## Troubleshooting
+
+Check K3s from cp1:
+
+```bash
+ssh ubuntu@10.27.27.30 'sudo k3s kubectl get nodes -o wide'
+ssh ubuntu@10.27.27.30 'sudo journalctl -u k3s -n 120 --no-pager'
+```
+
+Check Flux and Rancher:
+
+```bash
+kubectl -n flux-system get gitrepositories,kustomizations,helmreleases,ocirepositories
+kubectl -n flux-system describe helmrelease rancher
+kubectl -n cattle-system get pods,deploy -o wide
+```
+
+Check Tailscale services:
+
+```bash
 kubectl -n tailscale-system get pods
 kubectl -n cattle-system get svc rancher-tailscale
 kubectl -n observability get svc grafana-tailscale prometheus-tailscale
@@ -294,121 +271,14 @@ kubectl -n observability describe svc grafana-tailscale | grep TailscaleProxyRea
 kubectl -n observability describe svc prometheus-tailscale | grep TailscaleProxyReady
 ```
 
-If `TailscaleProxyReady=False`, check:
-
-```bash
-kubectl -n tailscale-system logs deployment/operator --tail=100
-```
-
-Common cause: OAuth client missing tag/scopes permissions.
-
-### Fast dashboard iteration workflow
-
-Use the `Deploy Grafana Content` workflow when changing dashboard/data source templates.
-It avoids full cluster provisioning and only applies Grafana content resources:
-
-- `ansible/roles/observability-content/templates/grafana-datasources.yaml.j2`
-- `ansible/roles/observability-content/templates/grafana-dashboard-k8s-overview.yaml.j2`
-- `ansible/dashboards.yml`
-
-## File Structure
-
-```
-.
-├── terraform/
-│   ├── main.tf
-│   ├── variables.tf
-│   ├── servers.tf
-│   ├── outputs.tf
-│   └── backend.tf
-├── ansible/
-│   ├── inventory.tmpl
-│   ├── generate_inventory.py
-│   ├── site.yml
-│   ├── roles/
-│   │   ├── common/
-│   │   ├── k3s-server/
-│   │   ├── k3s-agent/
-│   │   ├── addon-secrets-bootstrap/
-│   │   ├── observability-content/
-│   │   └── observability/
-│   └── ansible.cfg
-├── .gitea/
-│   └── workflows/
-│       ├── terraform.yml
-│       ├── ansible.yml
-│       └── dashboards.yml
-├── outputs/
-├── terraform.tfvars.example
-└── README.md
-```
-
-## Firewall Rules
-
-This repo no longer manages cloud firewalls. Access control is expected to be handled on your LAN infrastructure and through Tailscale.
-
-Important cluster-local ports still in use:
-
-| Port | Source | Purpose |
-|------|--------|---------|
-| 22 | Admin hosts / CI | SSH |
-| 6443 | 10.27.27.0/24 + VIP | Kubernetes API |
-| 9345 | 10.27.27.0/24 | k3s Supervisor |
-| 2379 | 10.27.27.0/24 | etcd Client |
-| 2380 | 10.27.27.0/24 | etcd Peer |
-| 8472/udp | 10.27.27.0/24 | Flannel VXLAN |
-| 10250 | 10.27.27.0/24 | Kubelet |
-
-## Operations
-
-### Scale Workers
-
-Edit `terraform.tfvars`:
-
-```hcl
-worker_count = 5
-```
-
-Then:
-
-```bash
-terraform apply
-ansible-playbook site.yml
-```
-
-### Upgrade k3s
-
-```bash
-ansible-playbook site.yml -t upgrade
-```
-
-### Destroy Cluster
-
-```bash
-terraform destroy
-```
-
-## Troubleshooting
-
-### Check k3s Logs
-
-```bash
-ssh ubuntu@<control-plane-ip> sudo journalctl -u k3s -f
-```
-
-### Reset k3s
-
-```bash
-ansible-playbook site.yml -t reset
-```
+If local `kubectl` falls back to `localhost:8080`, refresh `outputs/kubeconfig` with `scripts/refresh-kubeconfig.sh 10.27.27.30`.
 
 ## Security Notes
 
-- Control plane has HA (3 nodes, can survive 1 failure)
-- Kubernetes API HA is provided by kube-vip on `10.27.27.40`
-- Rotate API tokens regularly
-- Use network policies in Kubernetes
-- Enable audit logging for production
+- Never commit `terraform.tfvars`, kubeconfigs, private keys, `outputs/`, or real secret values.
+- Terraform/bootstrap/CI secrets stay in Gitea Actions secrets.
+- Runtime cluster secrets are sourced from Doppler through External Secrets.
+- This repo does not manage Proxmox/LAN firewalls or public ingress.
 
 ## License
 
diff --git a/ansible/inventory.tmpl b/ansible/inventory.tmpl
index eac23b9..cb6574a 100644
--- a/ansible/inventory.tmpl
+++ b/ansible/inventory.tmpl
@@ -16,5 +16,4 @@ workers
 ansible_user=ubuntu
 ansible_python_interpreter=/usr/bin/python3
 ansible_ssh_private_key_file={{ private_key_file }}
-k3s_version=latest
 kube_api_endpoint={{ kube_api_lb_ip }}