TerraHome/nixos/kubeadm/README.md

# Kubeadm Cluster Layout (NixOS)

This folder defines role-based NixOS configs for a kubeadm cluster.

## Topology

- Control planes: `cp-1`, `cp-2`, `cp-3`
- Workers: `wk-1`, `wk-2`, `wk-3`

## What this provides

- Shared Kubernetes/node prerequisites in `modules/k8s-common.nix`
- Shared cluster defaults in `modules/k8s-cluster-settings.nix`
- Role-specific settings for control planes and workers
- Generated per-node host configs from `flake.nix` (no duplicated host files)
- Bootstrap helper commands:
  - `th-kubeadm-init`
  - `th-kubeadm-join-control-plane`
  - `th-kubeadm-join-worker`
  - `th-kubeadm-status`

## Hardware config files

The flake automatically imports `hosts/hardware/<host>.nix` if present.
Copy each node's generated hardware config into this folder:

```bash
sudo nixos-generate-config
sudo cp /etc/nixos/hardware-configuration.nix ./hosts/hardware/cp-1.nix
```

Repeat for each node (`cp-2`, `cp-3`, `wk-1`, `wk-2`, `wk-3`).

## Deploy approach

Start from one node at a time while experimenting:

```bash
sudo nixos-rebuild switch --flake .#cp-1
```

For remote target-host workflows, use your preferred deploy wrapper later
(`nixos-rebuild --target-host ...` or deploy-rs/colmena).

## Bootstrap runbook (kubeadm + kube-vip + Cilium)

1. Apply Nix config on all nodes (`cp-*`, then `wk-*`).
2. On `cp-1`, run:

```bash
sudo th-kubeadm-init
```

This infers the control-plane VIP as `<node-subnet>.250` on `eth0`, creates the
kube-vip static pod manifest, and runs `kubeadm init`.

3. Install Cilium from `cp-1`:

```bash
helm repo add cilium https://helm.cilium.io
helm repo update
helm upgrade --install cilium cilium/cilium \
  --namespace kube-system \
  --set kubeProxyReplacement=true
```

4. Generate join commands on `cp-1`:

```bash
sudo kubeadm token create --print-join-command
sudo kubeadm init phase upload-certs --upload-certs
```

5. Join `cp-2` and `cp-3`:

```bash
sudo th-kubeadm-join-control-plane '<kubeadm join ... --control-plane --certificate-key ...>'
```

6. Join workers:

```bash
sudo th-kubeadm-join-worker '<kubeadm join ...>'
```

7. Validate from a control plane:

```bash
kubectl get nodes -o wide
kubectl -n kube-system get pods -o wide
```

## Repeatable rebuild flow (recommended)

1. Copy and edit inventory:

```bash
cp ./scripts/inventory.example.env ./scripts/inventory.env
$EDITOR ./scripts/inventory.env
```

2. Rebuild all nodes and bootstrap cluster:

```bash
./scripts/rebuild-and-bootstrap.sh
```

3. If you only want to reset Kubernetes state on existing VMs:

```bash
./scripts/reset-cluster-nodes.sh
```

For a full nuke/recreate lifecycle:
- run Terraform destroy/apply for VMs first,
- then run `./scripts/rebuild-and-bootstrap.sh` again.

## Notes

- Scripts are intentionally manual-triggered (predictable for homelab bring-up).
- If `.250` on the node subnet is already in use, change `controlPlaneVipSuffix`
  in `modules/k8s-cluster-settings.nix` before bootstrap.