OpenStaticFish/HetznerTerra
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs

Browse Source
This commit is contained in:
MichaelFisher1997
2026-04-30 07:03:21 +00:00
parent f49b08f50c
commit f52e657f9f
3 changed files with 221 additions and 347 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+41 -36
View File
@@ -1,52 +1,57 @@
# AGENTS.md # AGENTS.md
Repository guide for OpenCode sessions in this repo. Compact repo guidance for OpenCode sessions. Trust executable sources over docs when they conflict.
## Read First ## Read First
- Trust manifests and workflows over prose when they conflict. - Highest-value sources: `.gitea/workflows/deploy.yml`, `.gitea/workflows/destroy.yml`, `terraform/main.tf`, `terraform/variables.tf`, `terraform/servers.tf`, `ansible/site.yml`, `ansible/inventory.tmpl`, `clusters/prod/flux-system/`, `infrastructure/addons/kustomization.yaml`.
- Highest-value sources: `terraform/main.tf`, `terraform/variables.tf`, `ansible/site.yml`, `clusters/prod/flux-system/`, `infrastructure/addons/kustomization.yaml`, `.gitea/workflows/deploy.yml`, `.gitea/workflows/destroy.yml`, `README.md`, `STABLE_BASELINE.md`, `scripts/refresh-kubeconfig.sh`, `scripts/smoke-check-tailnet-services.sh`. - `STABLE_BASELINE.md` still contains stale Rancher backup/restore references; current workflows and addon manifests do not deploy or restore `rancher-backup`.
## Current Baseline ## Baseline
- HA private cluster: 3 control planes, 5 workers on Proxmox. - Proxmox HA K3s cluster: 3 control planes, 5 workers, VMIDs `200-202` and `210-214`, node `flex`, template VMID `9000`, datastore `Flash`.
- Proxmox clones come from template `9000` on node `flex`; API VIP is `10.27.27.40` via kube-vip. - API HA is kube-vip at `10.27.27.40`; control planes are `10.27.27.30-32`, workers are `10.27.27.41-45`.
- Storage is `nfs-subdir-external-provisioner` backed by `10.27.27.239:/TheFlash/k8s-nfs` with StorageClass `flash-nfs`. - SSH user is `ubuntu`; Ansible derives the flannel iface from `ansible_default_ipv4.interface` with `eth0` fallback, so do not hard-code `ens18`.
- Tailscale is the private access path for Rancher and shared services. - Storage is raw-manifest `nfs-subdir-external-provisioner` using `10.27.27.239:/TheFlash/k8s-nfs` and default StorageClass `flash-nfs`.
- Rancher, Grafana, and Prometheus are exposed through Tailscale; Flux UI / Weave GitOps is removed. - Tailscale is the private access path. Rancher, Grafana, and Prometheus are exposed only through Tailscale services.
- `apps/` is suspended by default. - `apps` is intentionally suspended in `clusters/prod/flux-system/kustomization-apps.yaml`.
- Rancher stores state in embedded etcd; backup/restore uses `rancher-backup` to B2.
## Common Commands ## Commands
- Terraform: `terraform -chdir=terraform fmt -recursive`, `terraform -chdir=terraform validate`, `terraform -chdir=terraform plan -var-file=../terraform.tfvars`, `terraform -chdir=terraform apply -var-file=../terraform.tfvars` - Terraform: `terraform -chdir=terraform fmt -recursive`, `terraform -chdir=terraform validate`, `terraform -chdir=terraform plan -var-file=../terraform.tfvars`, `terraform -chdir=terraform apply -var-file=../terraform.tfvars`.
- Ansible: `ansible-galaxy collection install -r ansible/requirements.yml`, `cd ansible && python3 generate_inventory.py`, `ansible-playbook -i ansible/inventory.ini ansible/site.yml --syntax-check`, `ansible-playbook ansible/site.yml` - Ansible setup: `ansible-galaxy collection install -r ansible/requirements.yml`, then from `ansible/` run `python3 generate_inventory.py` and `ansible-playbook site.yml --syntax-check`.
- Flux/Kustomize: `kubectl kustomize infrastructure/addons/<addon>`, `kubectl kustomize clusters/prod/flux-system` - Flux/Kustomize checks: `kubectl kustomize infrastructure/addons/<addon>`, `kubectl kustomize infrastructure/addons`, `kubectl kustomize clusters/prod/flux-system`.
- Kubeconfig refresh: `scripts/refresh-kubeconfig.sh <cp1-ip>` - Kubeconfig refresh: `scripts/refresh-kubeconfig.sh <cp1-ip>`; use this if local `kubectl` falls back to `localhost:8080` after rebuilds.
- Tailnet smoke check: `ssh ubuntu@<cp1-ip> 'bash -s' < scripts/smoke-check-tailnet-services.sh` - Tailnet smoke check from cp1: `ssh ubuntu@<cp1-ip> 'bash -s' < scripts/smoke-check-tailnet-services.sh`.
- Fast Grafana content iteration uses `.gitea/workflows/dashboards.yml` and `ansible/dashboards.yml`, not a full cluster rebuild.
## Workflow Rules ## Deploy Flow
- Keep diffs small and validate only the directory you edited. - Pushes to `main` run Gitea CI: Terraform fmt/init/validate/plan/apply, Proxmox cleanup/retry, Ansible bootstrap, Flux bootstrap, addon gates, Rancher gate, observability image seeding, health checks, tailnet smoke checks.
- Update manifests and docs together when behavior changes. - Deploy and destroy workflows share `concurrency.group: prod-cluster`; destroy only requires workflow input `confirm: destroy` and has no backup gate.
- Use `set -euo pipefail` in workflow shell blocks. - Keep `set -euo pipefail` in workflow shell blocks.
- CI deploy order is Terraform -> Ansible -> Flux bootstrap -> Rancher restore -> health checks. - Terraform retry cleanup has hard-coded target VMIDs/names in `.gitea/workflows/deploy.yml`; update it when changing node counts, names, or VMIDs.
- One object per Kubernetes YAML file; keep filenames kebab-case. - Fresh VMs have unreliable registry/chart egress, so critical images are prepared by `skopeo` on the runner and imported with `k3s ctr`; update the workflow archive lists when adding bootstrap-time images.
- If `kubectl` points at `localhost:8080` after a rebuild, refresh kubeconfig from the primary control-plane IP. - CI applies `clusters/prod/flux-system/gotk-components.yaml` directly and then patches Flux controller deployments inline; changes only in `gotk-controller-cp1-patches.yaml` do not affect CI bootstrap.
- Bootstrap assumptions that matter: SSH user is `ubuntu`, NIC is `ens18`, API join endpoint is the kube-vip address.
## Repo-Specific Gotchas ## GitOps Addons
- `rancher-backup` uses a postRenderer to swap the broken hook image to `rancher/kubectl:v1.34.0`; do not put S3 config in HelmRelease values. Put it in the Backup CR. - Vendored charts are intentional: `infrastructure/charts/{cert-manager,traefik,kube-prometheus-stack,tailscale-operator,rancher}`. Do not restore remote `HelmRepository` objects unless cluster-side chart fetch reliability is intentionally changed.
- Tailscale cleanup only runs before service proxies exist; it removes stale offline `rancher`/`grafana`/`prometheus`/`flux` devices, then must stop so live proxies are not deleted. - External Secrets and Loki/Promtail use Flux `OCIRepository`; Rancher, Tailscale, cert-manager, Traefik, and kube-prometheus-stack use `GitRepository` chart paths.
- Keep the vendored Tailscale operator chart at `infrastructure/charts/tailscale-operator` pinned to `1.96.5`; do not restore the remote HelmRepository unless cluster-side chart fetches are reliable. - Use fully qualified `helmchart.source.toolkit.fluxcd.io/...` in scripts; K3s also has `helmcharts.helm.cattle.io`, so `helmchart/...` can target the wrong resource.
- The repo no longer uses a cloud controller manager. If you see `providerID` or Hetzner-specific logic, it is stale. - `doppler-bootstrap` only creates the `external-secrets` namespace and Doppler token secret. The deploy workflow creates `ClusterSecretStore/doppler-hetznerterra` after ESO CRDs and webhook endpoints exist.
- Current private URLs: - The checked-in `infrastructure/addons/external-secrets/clustersecretstore-doppler-hetznerterra.yaml` is not included by that addon kustomization; do not assume Flux applies it.
- Rancher: `https://rancher.silverside-gopher.ts.net/` - Keep Kubernetes manifests one object per file with kebab-case filenames.
- Grafana: `http://grafana.silverside-gopher.ts.net/`
- Prometheus: `http://prometheus.silverside-gopher.ts.net:9090/` ## Gotchas
- Rancher chart `2.13.3` requires Kubernetes `<1.35.0-0`; K3s `latest` can break Rancher. Role defaults pin `v1.34.6+k3s1`; do not reintroduce a generated-inventory `k3s_version=latest` override.
- The repo no longer uses a cloud controller manager. `providerID`, Hetzner CCM/CSI, or Hetzner firewall/load-balancer logic is stale.
- Tailscale cleanup must only remove stale offline reserved hostnames before live service proxies exist; do not delete active `rancher`, `grafana`, `prometheus`, or `flux` devices.
- Proxmox endpoint should be the base URL, for example `https://100.105.0.115:8006/`; provider/workflow code strips `/api2/json` when needed.
- Current private URLs: Rancher `https://rancher.silverside-gopher.ts.net/`, Grafana `http://grafana.silverside-gopher.ts.net/`, Prometheus `http://prometheus.silverside-gopher.ts.net:9090/`.
## Secrets ## Secrets
- Runtime secrets live in Doppler + External Secrets. - Runtime secrets are Doppler + External Secrets; Terraform/bootstrap/CI secrets stay in Gitea Actions secrets.
- Bootstrap and CI secrets stay in Gitea; never commit secrets, kubeconfigs, or private keys. - Never commit secrets, kubeconfigs, private keys, `terraform.tfvars`, or generated `outputs/` artifacts.
README.md
+179 -309
View File
@@ -1,291 +1,268 @@
# Proxmox Kubernetes Cluster # Proxmox Kubernetes Cluster
Production-ready private Kubernetes cluster on Proxmox using Terraform, Ansible, and Flux. Private HA K3s cluster on Proxmox, provisioned by Terraform, bootstrapped by Ansible, and reconciled by Flux.
## Architecture ## Architecture
| Component | Details | | Component | Current Baseline |
|-----------|---------| |-----------|------------------|
| **Control Plane** | 3x Proxmox VMs (2 vCPU / 4 GiB / 32 GiB) | | **Control plane** | 3 Proxmox VMs, VMIDs `200-202`, IPs `10.27.27.30-32`, 2 vCPU / 4 GiB / 32 GiB |
| **Workers** | 5x Proxmox VMs (4 vCPU / 8 GiB / 64 GiB) | | **Workers** | 5 Proxmox VMs, VMIDs `210-214`, IPs `10.27.27.41-45`, 4 vCPU / 8 GiB / 64 GiB |
| **K8s** | k3s (latest, HA) | | **Kubernetes** | K3s `v1.34.6+k3s1`, HA embedded etcd, kube-vip API VIP `10.27.27.40` |
| **Addons** | NFS provisioner + Prometheus + Grafana + Loki + Rancher | | **Proxmox** | Node `flex`, template VMID `9000`, datastore `Flash`, bridge `vmbr0` |
| **Access** | SSH/API and private services restricted to Tailnet | | **Storage** | Raw-manifest `nfs-subdir-external-provisioner`, `10.27.27.239:/TheFlash/k8s-nfs`, default StorageClass `flash-nfs` |
| **Bootstrap** | Terraform + Ansible + Flux | | **GitOps** | Flux source `platform` on branch `main`; `apps` Kustomization is intentionally suspended |
| **Private access** | Tailscale operator exposes Rancher, Grafana, and Prometheus; no public ingress baseline |
| **Runtime secrets** | Doppler service token bootstraps External Secrets Operator |
K3s is pinned because Rancher chart `2.13.3` requires Kubernetes `<1.35.0-0`.
## Prerequisites ## Prerequisites
### 1. Proxmox API Token - Terraform `>= 1.0`.
- Ansible with Python `jinja2` and `pyyaml`.
- `kubectl` for local verification.
- Proxmox API token for the `bpg/proxmox` provider.
- S3-compatible bucket for Terraform state, currently Backblaze B2.
- SSH key pair available to Terraform and Ansible, defaulting to `~/.ssh/infra` and `~/.ssh/infra.pub`.
Create an API token for the Proxmox VE user used by Terraform. The repo expects the `bpg/proxmox` provider with: Expected Proxmox inputs:
- endpoint: `https://100.105.0.115:8006/` | Setting | Value |
- node: `flex` |---------|-------|
- clone source: template `9000` (`ubuntu-2404-k8s-template`) | Endpoint | `https://100.105.0.115:8006/` |
- auth: API token | Node | `flex` |
| Clone source | Template VMID `9000` (`ubuntu-2404-k8s-template`) |
| Storage | `Flash` |
### 2. Backblaze B2 Bucket (for Terraform State) ## Local Setup
1. Go to [Backblaze B2](https://secure.backblaze.com/b2_buckets.htm) Create local variables from the example:
2. Click **Create a Bucket**
3. Set bucket name: `k8s-terraform-state` (must be globally unique)
4. Choose **Private** access
5. Click **Create Bucket**
6. Create application key:
- Go to **App Keys** → **Add a New Application Key**
- Name: `terraform-state`
- Allow access to: `k8s-terraform-state` bucket only
- Type: **Read and Write**
- Copy **keyID** (access key) and **applicationKey** (secret key)
7. Note your bucket's S3 endpoint (e.g., `https://s3.eu-central-003.backblazeb2.com`)
### 3. SSH Key Pair
```bash
ssh-keygen -t ed25519 -C "k8s@proxmox" -f ~/.ssh/infra
```
### 4. Local Tools
- [Terraform](https://terraform.io/downloads) >= 1.0
- [Ansible](https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html) >= 2.9
- Python 3 with `jinja2` and `pyyaml`
## Setup
### 1. Clone Repository
```bash
git clone <your-gitea-repo>/HetznerTerra.git
cd HetznerTerra
```
### 2. Configure Variables
```bash ```bash
cp terraform.tfvars.example terraform.tfvars cp terraform.tfvars.example terraform.tfvars
``` ```
Edit `terraform.tfvars`: Important defaults in `terraform.tfvars.example`:
```hcl ```hcl
proxmox_endpoint = "https://100.105.0.115:8006/" proxmox_endpoint = "https://100.105.0.115:8006/"
proxmox_api_token_id = "terraform-prov@pve!k8s-cluster" proxmox_api_token_id = "terraform-prov@pve!k8s-cluster"
proxmox_api_token_secret = "your-proxmox-token-secret" proxmox_api_token_secret = "your-proxmox-api-token-secret"
ssh_public_key = "~/.ssh/infra.pub" ssh_public_key = "~/.ssh/infra.pub"
ssh_private_key = "~/.ssh/infra" ssh_private_key = "~/.ssh/infra"
s3_access_key = "your-backblaze-key-id" s3_access_key = "your-backblaze-key-id"
s3_secret_key = "your-backblaze-application-key" s3_secret_key = "your-backblaze-application-key"
s3_endpoint = "https://s3.eu-central-003.backblazeb2.com" s3_endpoint = "https://s3.eu-central-003.backblazeb2.com"
s3_bucket = "k8s-terraform-state" s3_bucket = "k8s-terraform-state"
tailscale_auth_key = "tskey-auth-..."
tailscale_tailnet = "yourtailnet.ts.net" tailscale_tailnet = "yourtailnet.ts.net"
kube_api_vip = "10.27.27.40"
kube_api_vip = "10.27.27.40"
``` ```
### 3. Initialize Terraform Initialize Terraform with backend credentials:
```bash ```bash
cd terraform terraform -chdir=terraform init \
-backend-config="endpoint=<s3-endpoint>" \
# Create backend config file (or use CLI args) -backend-config="bucket=<s3-bucket>" \
cat > backend.hcl << EOF -backend-config="region=auto" \
endpoint = "https://s3.eu-central-003.backblazeb2.com" -backend-config="access_key=<s3-access-key>" \
bucket = "k8s-terraform-state" -backend-config="secret_key=<s3-secret-key>" \
access_key = "your-backblaze-key-id" -backend-config="skip_requesting_account_id=true"
secret_key = "your-backblaze-application-key"
skip_requesting_account_id = true
EOF
terraform init -backend-config=backend.hcl
``` ```
### 4. Plan and Apply ## Common Commands
Terraform:
```bash ```bash
terraform plan -var-file=../terraform.tfvars terraform -chdir=terraform fmt -recursive
terraform apply -var-file=../terraform.tfvars terraform -chdir=terraform validate
terraform -chdir=terraform plan -var-file=../terraform.tfvars
terraform -chdir=terraform apply -var-file=../terraform.tfvars
``` ```
### 5. Generate Ansible Inventory Ansible setup:
```bash ```bash
cd ../ansible ansible-galaxy collection install -r ansible/requirements.yml
cd ansible
python3 generate_inventory.py python3 generate_inventory.py
ansible-playbook site.yml --syntax-check
``` ```
### 6. Bootstrap Cluster Manual Ansible bootstrap uses the same extra vars as the deploy workflow:
```bash ```bash
ansible-playbook site.yml cd ansible
ansible-playbook site.yml \
-e "tailscale_auth_key=$TAILSCALE_AUTH_KEY" \
-e "tailscale_tailnet=$TAILSCALE_TAILNET" \
-e "tailscale_oauth_client_id=$TAILSCALE_OAUTH_CLIENT_ID" \
-e "tailscale_oauth_client_secret=$TAILSCALE_OAUTH_CLIENT_SECRET" \
-e "doppler_hetznerterra_service_token=$DOPPLER_HETZNERTERRA_SERVICE_TOKEN" \
-e "tailscale_api_key=${TAILSCALE_API_KEY:-}" \
-e "grafana_admin_password=${GRAFANA_ADMIN_PASSWORD:-}" \
-e "cluster_name=k8s-cluster"
``` ```
### 7. Get Kubeconfig Flux/Kustomize verification:
```bash ```bash
kubectl kustomize infrastructure/addons/<addon>
kubectl kustomize infrastructure/addons
kubectl kustomize clusters/prod/flux-system
```
Refresh kubeconfig after rebuilds:
```bash
scripts/refresh-kubeconfig.sh 10.27.27.30
export KUBECONFIG=$(pwd)/outputs/kubeconfig export KUBECONFIG=$(pwd)/outputs/kubeconfig
kubectl get nodes kubectl get nodes
``` ```
Use `scripts/refresh-kubeconfig.sh <cp1-public-ip>` to refresh kubeconfig against the primary control-plane public IP after rebuilds. Run the tailnet smoke check from cp1:
```bash
ssh ubuntu@10.27.27.30 'bash -s' < scripts/smoke-check-tailnet-services.sh
```
## Gitea CI/CD ## Gitea CI/CD
This repository includes Gitea workflows for: The supported full rebuild path is the Gitea deploy workflow.
- **deploy**: End-to-end Terraform + Ansible + Flux bootstrap + restore + health checks | Workflow | Trigger | Purpose |
- **destroy**: Cluster teardown with backup-aware cleanup |----------|---------|---------|
- **dashboards**: Fast workflow that updates Grafana datasources/dashboards only | `.gitea/workflows/deploy.yml` | PR to `main`, push to `main`, manual dispatch | PRs run Terraform plan; pushes run Terraform apply, Ansible bootstrap, Flux bootstrap, addon gates, health checks, and tailnet smoke checks |
| `.gitea/workflows/destroy.yml` | Manual dispatch with `confirm: destroy` | Terraform destroy with retries; no Rancher backup gate |
| `.gitea/workflows/dashboards.yml` | Grafana content changes or manual dispatch | Fast Grafana datasource/dashboard update through `ansible/dashboards.yml` |
### Required Gitea Secrets Deploy and destroy share `concurrency.group: prod-cluster` so they do not run at the same time.
Set these in your Gitea repository settings (**Settings** → **Secrets** → **Actions**): Deploy sequence on push to `main`:
1. Terraform fmt/init/validate/plan/apply.
2. Cleanup/retry around known transient Proxmox clone and disk-update failures.
3. Generate Ansible inventory from Terraform outputs.
4. Prepare critical image archives with `skopeo` on the runner.
5. Run `ansible/site.yml` to bootstrap nodes, K3s, kube-vip, prerequisite secrets, and kubeconfig.
6. Apply Flux CRDs/controllers and the `clusters/prod/flux-system` graph.
7. Gate cert-manager, External Secrets, Tailscale, NFS, Rancher, and observability.
8. Run post-deploy health checks and Tailscale service smoke checks.
Required Gitea secrets:
| Secret | Description | | Secret | Description |
|--------|-------------| |--------|-------------|
| `PROXMOX_ENDPOINT` | Proxmox API endpoint (for example `https://100.105.0.115:8006/`) | | `PROXMOX_ENDPOINT` | Proxmox API endpoint, for example `https://100.105.0.115:8006/` |
| `PROXMOX_API_TOKEN_ID` | Proxmox API token ID | | `PROXMOX_API_TOKEN_ID` | Proxmox API token ID |
| `PROXMOX_API_TOKEN_SECRET` | Proxmox API token secret | | `PROXMOX_API_TOKEN_SECRET` | Proxmox API token secret |
| `S3_ACCESS_KEY` | Backblaze B2 keyID | | `S3_ACCESS_KEY` | S3/Backblaze access key for Terraform state |
| `S3_SECRET_KEY` | Backblaze B2 applicationKey | | `S3_SECRET_KEY` | S3/Backblaze secret key for Terraform state |
| `S3_ENDPOINT` | Backblaze S3 endpoint (e.g., `https://s3.eu-central-003.backblazeb2.com`) | | `S3_ENDPOINT` | S3 endpoint, for example `https://s3.eu-central-003.backblazeb2.com` |
| `S3_BUCKET` | S3 bucket name (e.g., `k8s-terraform-state`) | | `S3_BUCKET` | Terraform state bucket, for example `k8s-terraform-state` |
| `TAILSCALE_AUTH_KEY` | Tailscale auth key for node bootstrap | | `TAILSCALE_AUTH_KEY` | Tailscale auth key for node bootstrap |
| `TAILSCALE_TAILNET` | Tailnet domain (e.g., `yourtailnet.ts.net`) | | `TAILSCALE_TAILNET` | Tailnet domain, for example `silverside-gopher.ts.net` |
| `TAILSCALE_OAUTH_CLIENT_ID` | Tailscale OAuth client ID for Kubernetes Operator | | `TAILSCALE_OAUTH_CLIENT_ID` | Tailscale OAuth client ID for the Kubernetes operator |
| `TAILSCALE_OAUTH_CLIENT_SECRET` | Tailscale OAuth client secret for Kubernetes Operator | | `TAILSCALE_OAUTH_CLIENT_SECRET` | Tailscale OAuth client secret for the Kubernetes operator |
| `DOPPLER_HETZNERTERRA_SERVICE_TOKEN` | Doppler service token for `hetznerterra` runtime secrets | | `TAILSCALE_API_KEY` | Optional API key used to delete stale offline reserved devices before service proxies exist |
| `GRAFANA_ADMIN_PASSWORD` | Optional admin password for Grafana (auto-generated if unset) | | `DOPPLER_HETZNERTERRA_SERVICE_TOKEN` | Doppler service token for runtime cluster secrets |
| `GRAFANA_ADMIN_PASSWORD` | Optional Grafana admin password |
| `SSH_PUBLIC_KEY` | SSH public key content | | `SSH_PUBLIC_KEY` | SSH public key content |
| `SSH_PRIVATE_KEY` | SSH private key content | | `SSH_PRIVATE_KEY` | SSH private key content |
## GitOps (Flux) ## GitOps Graph
This repo uses Flux for continuous reconciliation after Terraform + Ansible bootstrap. Flux entrypoint:
### Stable private-only baseline ```text
clusters/prod/flux-system/
├── gotk-components.yaml
├── gitrepository-platform.yaml
├── kustomization-infrastructure.yaml
└── kustomization-apps.yaml # suspend: true
```
The current default target is the HA private baseline: Active infrastructure addons from `infrastructure/addons/kustomization.yaml`:
- `3` control plane nodes - `addon-nfs-storage`
- `5` worker nodes - `addon-external-secrets`
- private Proxmox network only - `addon-cert-manager`
- Tailscale for operator and service access - `addon-tailscale-operator`
- Flux-managed platform addons with `apps` suspended by default - `addon-tailscale-proxyclass`
- `traefik` HelmRelease manifests applied directly by the top-level infrastructure Kustomization
- `addon-observability`
- `addon-observability-content`
- `addon-rancher`
- `addon-rancher-config`
Detailed phase gates and success criteria live in `STABLE_BASELINE.md`. Chart/source strategy:
This is the default until rebuilds are consistently green. High availability, public ingress, and app-layer expansion come later. - Vendored charts are intentional: `cert-manager`, `traefik`, `kube-prometheus-stack`, `tailscale-operator`, and `rancher` live under `infrastructure/charts/`.
- External Secrets, Loki, and Promtail use Flux `OCIRepository` sources.
- NFS storage is raw Kubernetes manifests, not a Helm chart.
- Rancher backup/restore is not part of the current live graph.
### Runtime secrets Doppler bootstrap details:
Runtime cluster secrets are moving to Doppler + External Secrets Operator. - `ansible/roles/doppler-bootstrap` creates the `external-secrets` namespace and the Doppler token secret only.
- The deploy workflow creates `ClusterSecretStore/doppler-hetznerterra` after ESO CRDs and webhook endpoints exist.
- The checked-in `infrastructure/addons/external-secrets/clustersecretstore-doppler-hetznerterra.yaml` is not included by the addon kustomization.
- Doppler project: `hetznerterra` ## Access URLs
- Initial auth: service token via `DOPPLER_HETZNERTERRA_SERVICE_TOKEN`
- First synced secrets:
- `GRAFANA_ADMIN_PASSWORD`
Terraform/bootstrap secrets remain in Gitea Actions secrets and are not managed by Doppler. | Service | URL |
|---------|-----|
| Rancher | `https://rancher.silverside-gopher.ts.net/` |
| Grafana | `http://grafana.silverside-gopher.ts.net/` |
| Prometheus | `http://prometheus.silverside-gopher.ts.net:9090/` |
### Repository layout Fallback port-forward from a tailnet-connected machine:
- `clusters/prod/`: cluster entrypoint and Flux reconciliation objects
- `clusters/prod/flux-system/`: `GitRepository` source and top-level `Kustomization` graph
- `infrastructure/`: infrastructure addon reconciliation graph
- `infrastructure/addons/*`: per-addon manifests for Flux-managed cluster addons
- `apps/`: application workload layer (currently scaffolded)
### Reconciliation graph
- `infrastructure` (top-level)
- `addon-nfs-storage`
- `addon-tailscale-operator`
- `addon-observability`
- `addon-observability-content` depends on `addon-observability`
- `apps` depends on `infrastructure`
### Bootstrap notes
1. Install Flux controllers in `flux-system`.
2. Create the Flux deploy key/secret named `flux-system` in `flux-system` namespace.
3. Apply `clusters/prod/flux-system/` once to establish source + reconciliation graph.
4. Bootstrap-only Ansible creates prerequisite secrets; Flux manages addon lifecycle after bootstrap.
### Current addon status
- Core infrastructure addons are Flux-managed from `infrastructure/addons/`.
- Active Flux addons for the current baseline: `addon-nfs-storage`, `addon-cert-manager`, `addon-external-secrets`, `addon-tailscale-operator`, `addon-tailscale-proxyclass`, `addon-observability`, `addon-observability-content`, `addon-rancher`, `addon-rancher-config`, `addon-rancher-backup`, `addon-rancher-backup-config`.
- `apps` remains suspended until workload rollout is explicitly enabled.
- Ansible is limited to cluster bootstrap, prerequisite secret creation, pre-proxy Tailscale cleanup, and kubeconfig finalization.
- Weave GitOps / Flux UI is no longer deployed; use Rancher or the `flux` CLI for Flux operations.
### Rancher access
- Rancher is private-only and exposed through Tailscale at `https://rancher.silverside-gopher.ts.net/`.
- Rancher and the Kubernetes API stay private; kube-vip provides the API VIP on the LAN.
- Rancher stores state in embedded etcd; no external database is used.
### Stable baseline acceptance
A rebuild is considered successful only when all of the following pass without manual intervention:
- Terraform create succeeds for the default `3` control planes and `5` workers.
- Ansible bootstrap succeeds end-to-end.
- All nodes become `Ready`.
- Flux core reconciliation is healthy.
- External Secrets Operator is ready.
- Tailscale operator is ready.
- Tailnet smoke checks pass for Rancher, Grafana, and Prometheus.
- Terraform destroy succeeds cleanly or succeeds after workflow retries.
## Observability Stack
Flux deploys a lightweight observability stack in the `observability` namespace:
- `kube-prometheus-stack` (Prometheus + Grafana)
- `loki`
- `promtail`
Grafana content is managed as code via ConfigMaps in `infrastructure/addons/observability-content/`.
Grafana and Prometheus are exposed through dedicated Tailscale LoadBalancer services when the Tailscale Kubernetes Operator is healthy.
### Access Grafana and Prometheus
Preferred private access:
- Grafana: `http://grafana.silverside-gopher.ts.net/`
- Prometheus: `http://prometheus.silverside-gopher.ts.net:9090/`
Fallback (port-forward from a tailnet-connected machine):
Run from a tailnet-connected machine:
```bash ```bash
export KUBECONFIG=$(pwd)/outputs/kubeconfig export KUBECONFIG=$(pwd)/outputs/kubeconfig
kubectl -n observability port-forward svc/kube-prometheus-stack-grafana 3000:80 kubectl -n observability port-forward svc/kube-prometheus-stack-grafana 3000:80
kubectl -n observability port-forward svc/kube-prometheus-stack-prometheus 9090:9090 kubectl -n observability port-forward svc/kube-prometheus-stack-prometheus 9090:9090
``` ```
Then open: Grafana user is `admin`; password comes from the `GRAFANA_ADMIN_PASSWORD` Doppler secret or the workflow-provided fallback.
- Grafana: http://127.0.0.1:3000 ## Operations
- Prometheus: http://127.0.0.1:9090
Grafana user: `admin` Scale workers by updating `terraform.tfvars` counts, IP lists, and VMID lists together. If node names or VMIDs change, also update the hard-coded retry cleanup target map in `.gitea/workflows/deploy.yml`.
Grafana password: value of `GRAFANA_ADMIN_PASSWORD` secret (or the generated value shown by Ansible output)
### Verify Tailscale exposure Upgrade K3s by changing the role defaults in `ansible/roles/k3s-server/defaults/main.yml` and `ansible/roles/k3s-agent/defaults/main.yml`. Check Rancher chart compatibility before moving to a Kubernetes minor outside `<1.35.0-0`.
Destroy through the Gitea `Destroy` workflow with `confirm: destroy`, or locally with:
```bash ```bash
export KUBECONFIG=$(pwd)/outputs/kubeconfig terraform -chdir=terraform destroy -var-file=../terraform.tfvars
```
## Troubleshooting
Check K3s from cp1:
```bash
ssh ubuntu@10.27.27.30 'sudo k3s kubectl get nodes -o wide'
ssh ubuntu@10.27.27.30 'sudo journalctl -u k3s -n 120 --no-pager'
```
Check Flux and Rancher:
```bash
kubectl -n flux-system get gitrepositories,kustomizations,helmreleases,ocirepositories
kubectl -n flux-system describe helmrelease rancher
kubectl -n cattle-system get pods,deploy -o wide
```
Check Tailscale services:
```bash
kubectl -n tailscale-system get pods kubectl -n tailscale-system get pods
kubectl -n cattle-system get svc rancher-tailscale kubectl -n cattle-system get svc rancher-tailscale
kubectl -n observability get svc grafana-tailscale prometheus-tailscale kubectl -n observability get svc grafana-tailscale prometheus-tailscale
@@ -294,121 +271,14 @@ kubectl -n observability describe svc grafana-tailscale | grep TailscaleProxyRea
kubectl -n observability describe svc prometheus-tailscale | grep TailscaleProxyReady kubectl -n observability describe svc prometheus-tailscale | grep TailscaleProxyReady
``` ```
If `TailscaleProxyReady=False`, check: If local `kubectl` falls back to `localhost:8080`, refresh `outputs/kubeconfig` with `scripts/refresh-kubeconfig.sh 10.27.27.30`.
```bash
kubectl -n tailscale-system logs deployment/operator --tail=100
```
Common cause: OAuth client missing tag/scopes permissions.
### Fast dashboard iteration workflow
Use the `Deploy Grafana Content` workflow when changing dashboard/data source templates.
It avoids full cluster provisioning and only applies Grafana content resources:
- `ansible/roles/observability-content/templates/grafana-datasources.yaml.j2`
- `ansible/roles/observability-content/templates/grafana-dashboard-k8s-overview.yaml.j2`
- `ansible/dashboards.yml`
## File Structure
```
.
├── terraform/
│ ├── main.tf
│ ├── variables.tf
│ ├── servers.tf
│ ├── outputs.tf
│ └── backend.tf
├── ansible/
│ ├── inventory.tmpl
│ ├── generate_inventory.py
│ ├── site.yml
│ ├── roles/
│ │ ├── common/
│ │ ├── k3s-server/
│ │ ├── k3s-agent/
│ │ ├── addon-secrets-bootstrap/
│ │ ├── observability-content/
│ │ └── observability/
│ └── ansible.cfg
├── .gitea/
│ └── workflows/
│ ├── terraform.yml
│ ├── ansible.yml
│ └── dashboards.yml
├── outputs/
├── terraform.tfvars.example
└── README.md
```
## Firewall Rules
This repo no longer manages cloud firewalls. Access control is expected to be handled on your LAN infrastructure and through Tailscale.
Important cluster-local ports still in use:
| Port | Source | Purpose |
|------|--------|---------|
| 22 | Admin hosts / CI | SSH |
| 6443 | 10.27.27.0/24 + VIP | Kubernetes API |
| 9345 | 10.27.27.0/24 | k3s Supervisor |
| 2379 | 10.27.27.0/24 | etcd Client |
| 2380 | 10.27.27.0/24 | etcd Peer |
| 8472/udp | 10.27.27.0/24 | Flannel VXLAN |
| 10250 | 10.27.27.0/24 | Kubelet |
## Operations
### Scale Workers
Edit `terraform.tfvars`:
```hcl
worker_count = 5
```
Then:
```bash
terraform apply
ansible-playbook site.yml
```
### Upgrade k3s
```bash
ansible-playbook site.yml -t upgrade
```
### Destroy Cluster
```bash
terraform destroy
```
## Troubleshooting
### Check k3s Logs
```bash
ssh ubuntu@<control-plane-ip> sudo journalctl -u k3s -f
```
### Reset k3s
```bash
ansible-playbook site.yml -t reset
```
## Security Notes ## Security Notes
- Control plane has HA (3 nodes, can survive 1 failure) - Never commit `terraform.tfvars`, kubeconfigs, private keys, `outputs/`, or real secret values.
- Kubernetes API HA is provided by kube-vip on `10.27.27.40` - Terraform/bootstrap/CI secrets stay in Gitea Actions secrets.
- Rotate API tokens regularly - Runtime cluster secrets are sourced from Doppler through External Secrets.
- Use network policies in Kubernetes - This repo does not manage Proxmox/LAN firewalls or public ingress.
- Enable audit logging for production
## License ## License
ansible/inventory.tmpl
-1
View File
@@ -16,5 +16,4 @@ workers
ansible_user=ubuntu ansible_user=ubuntu
ansible_python_interpreter=/usr/bin/python3 ansible_python_interpreter=/usr/bin/python3
ansible_ssh_private_key_file={{ private_key_file }} ansible_ssh_private_key_file={{ private_key_file }}
k3s_version=latest
kube_api_endpoint={{ kube_api_lb_ip }} kube_api_endpoint={{ kube_api_lb_ip }}