docs: add guide for deploying app repos to the cluster

Document the recommended two-repo model for application delivery, including
Flux attachment objects, Doppler/ExternalSecret wiring, Tailscale service
exposure, and the steps for enabling the suspended apps layer.

APP_REPO_DEPLOYMENT_GUIDE.md

@@ -0,0 +1,287 @@
+# App Repo Deployment Guide
+
+This guide explains the recommended way to deploy an application to this cluster.
+
+## Recommended Model
+
+Use two repos:
+
+- `HetznerTerra` (this repo): cluster, addons, shared infrastructure, Flux wiring
+- `your-app-repo`: application source, Dockerfile, CI, Kubernetes manifests or Helm chart
+
+Why:
+
+- cluster lifecycle stays separate from app code
+- app CI can build and tag images independently
+- this repo remains the source of truth for what the cluster is allowed to deploy
+
+## Current Cluster Assumptions
+
+- Flux is already installed and reconciles this repo from `main`
+- `clusters/prod/flux-system/kustomization-apps.yaml` points at `./apps`
+- `apps` is suspended by default
+- private access is through Tailscale
+- runtime secrets should come from Doppler via External Secrets
+
+## Deployment Options
+
+### Option A: Separate app repo
+
+Recommended for most real applications.
+
+Flow:
+
+1. App repo builds and pushes an image.
+2. This repo defines a `GitRepository` pointing at the app repo.
+3. This repo defines a `Kustomization` pointing at a path in the app repo.
+4. Flux pulls the app repo and applies the manifests.
+
+### Option B: In-repo app manifests
+
+Only use this when the application is tiny or tightly coupled to the platform.
+
+Flow:
+
+1. Put Kubernetes manifests directly under `apps/` in this repo.
+2. Unsuspend the top-level `apps` Kustomization.
+
+This is simpler, but mixes platform and app changes together.
+
+## App Repo Structure
+
+Suggested layout:
+
+```text
+your-app-repo/
+├── src/
+├── Dockerfile
+├── .gitea/workflows/
+└── deploy/
+    ├── base/
+    │   ├── namespace.yaml
+    │   ├── deployment.yaml
+    │   ├── service.yaml
+    │   ├── externalsecret.yaml
+    │   └── kustomization.yaml
+    └── prod/
+        ├── kustomization.yaml
+        └── patch-*.yaml
+```
+
+If you prefer Helm, replace `deploy/base` and `deploy/prod` with a chart path and point Flux at that instead.
+
+## What the App Repo Should Own
+
+- application source code
+- image build pipeline
+- image tag strategy
+- Deployment / Service / Ingress or Tailscale-facing Service manifests
+- app-specific `ExternalSecret` manifests
+- app-specific namespace
+
+## What This Repo Should Own
+
+- cluster-level permission to deploy the app
+- the `GitRepository` and top-level `Kustomization` that attach the app repo to the cluster
+- whether the `apps` layer is suspended or active
+
+## Recommended First App Integration
+
+In this repo, add Flux objects under `apps/` that point to the app repo.
+
+Example files to add:
+
+- `apps/gitrepository-my-app.yaml`
+- `apps/kustomization-my-app.yaml`
+- update `apps/kustomization.yaml`
+
+Example `apps/gitrepository-my-app.yaml`:
+
+```yaml
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: GitRepository
+metadata:
+  name: my-app
+  namespace: flux-system
+spec:
+  interval: 1m
+  ref:
+    branch: main
+  secretRef:
+    name: flux-system
+  url: ssh://git@<your-git-host>:<port>/<org>/<your-app-repo>.git
+```
+
+Example `apps/kustomization-my-app.yaml`:
+
+```yaml
+apiVersion: kustomize.toolkit.fluxcd.io/v1
+kind: Kustomization
+metadata:
+  name: my-app
+  namespace: flux-system
+spec:
+  interval: 10m
+  prune: true
+  sourceRef:
+    kind: GitRepository
+    name: my-app
+  path: ./deploy/prod
+  wait: true
+  timeout: 5m
+  dependsOn:
+    - name: infrastructure
+```
+
+Then update `apps/kustomization.yaml`:
+
+```yaml
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - gitrepository-my-app.yaml
+  - kustomization-my-app.yaml
+```
+
+## App Secrets
+
+Recommended path:
+
+1. Put runtime values in Doppler.
+2. In the app manifests, create an `ExternalSecret` that reads from `doppler-hetznerterra`.
+3. Reference the resulting Kubernetes Secret from the Deployment.
+
+Example app-side `ExternalSecret`:
+
+```yaml
+apiVersion: external-secrets.io/v1
+kind: ExternalSecret
+metadata:
+  name: my-app-env
+  namespace: my-app
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: doppler-hetznerterra
+    kind: ClusterSecretStore
+  target:
+    name: my-app-env
+    creationPolicy: Owner
+  data:
+    - secretKey: DATABASE_URL
+      remoteRef:
+        key: MY_APP_DATABASE_URL
+```
+
+## Image Delivery
+
+Recommended flow:
+
+1. App repo CI builds a container image.
+2. CI pushes it to a registry.
+3. The app repo updates the Kubernetes image tag in `deploy/prod`.
+4. Flux notices the Git change and deploys it.
+
+Keep the first version simple. Do not add image automation until the basic deploy path is proven.
+
+## Exposing the App
+
+Pick one:
+
+### Private app over Tailscale
+
+Best fit for this cluster right now.
+
+Create a Service like the existing Rancher/Grafana/Prometheus pattern:
+
+```yaml
+apiVersion: v1
+kind: Service
+metadata:
+  name: my-app-tailscale
+  namespace: my-app
+  annotations:
+    tailscale.com/hostname: my-app
+    tailscale.com/tags: "tag:prod"
+    tailscale.com/proxy-class: infra-stable
+spec:
+  type: LoadBalancer
+  loadBalancerClass: tailscale
+  selector:
+    app.kubernetes.io/name: my-app
+  ports:
+    - name: http
+      port: 80
+      protocol: TCP
+      targetPort: 3000
+```
+
+Use `http://my-app.<your-tailnet>` or your chosen hostname.
+
+### Cluster-internal only
+
+Create only a `ClusterIP` Service.
+
+### Public ingress
+
+Not recommended as the first app path in this repo. Get the private path working first.
+
+## Enabling the Apps Layer
+
+The cluster-wide `apps` Kustomization is suspended by default.
+
+When you are ready to let Flux deploy app attachments from `apps/`, unsuspend it:
+
+```bash
+kubectl -n flux-system patch kustomization apps --type=merge -p '{"spec":{"suspend":false}}'
+```
+
+Or commit a change to `clusters/prod/flux-system/kustomization-apps.yaml` changing:
+
+```yaml
+suspend: true
+```
+
+to:
+
+```yaml
+suspend: false
+```
+
+## First Deploy Checklist
+
+Before deploying the first app, make sure:
+
+1. app image builds successfully
+2. app repo contains valid `deploy/prod` manifests
+3. this repo contains the `GitRepository` + `Kustomization` attachment objects
+4. required Doppler secrets exist
+5. `apps` is unsuspended if you are using the top-level `apps` layer
+
+## Verification Commands
+
+From a machine with cluster access:
+
+```bash
+kubectl -n flux-system get gitrepositories,kustomizations
+kubectl get ns
+kubectl -n my-app get deploy,svc,pods,externalsecret,secret
+```
+
+If private over Tailscale:
+
+```bash
+kubectl -n my-app get svc my-app-tailscale -o wide
+```
+
+## Minimal Recommendation
+
+If you want the simplest, lowest-risk first deploy:
+
+1. create a separate app repo
+2. add `deploy/base` + `deploy/prod`
+3. add a `GitRepository` + `Kustomization` in this repo under `apps/`
+4. keep the app private with a Tailscale `LoadBalancer` Service
+5. use Doppler + `ExternalSecret` for runtime config
+
+That matches the current cluster design with the least surprise.