Add Lightpanda browser services and automation scripts

Add lightpanda-cdp and lightpanda-mcp services for local (Tilt) and
production (deploy/prod/) with Traefik routing, HPAs, and CI image
publishing. Create scripts for concurrent Bing search via MCP and
CDP screenshots. Update flake.nix with websocat dependency. Make
webshare-secret helper tolerant of missing Doppler credentials.
This commit is contained in:
2026-05-05 07:56:22 +01:00
parent fff06469a9
commit ed6415894f
20 changed files with 1009 additions and 21 deletions
+2
View File
@@ -25,6 +25,7 @@ jobs:
service:
- site-analyzer
- scraper
- lightpanda-mcp
steps:
- name: Checkout
@@ -64,6 +65,7 @@ jobs:
set -euo pipefail
sed -i -E "/name: ghcr\.io\/openstaticfish\/microservices\/site-analyzer/{n;s/newTag: .*/newTag: ${GITHUB_SHA}/;}" deploy/prod/kustomization.yaml
sed -i -E "/name: ghcr\.io\/openstaticfish\/microservices\/scraper/{n;s/newTag: .*/newTag: ${GITHUB_SHA}/;}" deploy/prod/kustomization.yaml
sed -i -E "/name: ghcr\.io\/openstaticfish\/microservices\/lightpanda-mcp/{n;s/newTag: .*/newTag: ${GITHUB_SHA}/;}" deploy/prod/kustomization.yaml
- name: Commit production image tag update
run: |
+39 -19
View File
@@ -2,28 +2,48 @@
## Repo Shape
- This repo is a minimal Nix + Kind + Tilt local Kubernetes setup, not a general Go monorepo yet.
- The only app is `services/hello-world`, a Go 1.22 HTTP service built by `services/hello-world/Dockerfile` and deployed by `k8s/hello-world.yaml`.
- `Tiltfile` is the wiring source of truth for dev orchestration; `flake.nix` is the source of truth for available CLI dependencies.
- Nix + Kind + Tilt local Kubernetes setup for Go microservices + Lightpanda browser automation.
- Three Go apps under `services/`:
- `scraper` — web scraping service (port 8080)
- `site-analyzer` — site analysis service (port 8090)
- `lightpanda-mcp` — wraps Lightpanda MCP behind `supergateway` for remote agents (port 8000)
- `lightpanda-cdp` runs the official `lightpanda/browser:nightly` CDP server (port 9222); not a Go app, deployed from a prebuilt image.
- `Tiltfile` wires all services; `flake.nix` declares CLI deps and dev shell.
- No test, lint, formatter, typecheck, or pre-commit config exists.
## Commands
- Use `nix run .#setup-kind` before Tilt if the cluster/context may not exist; it creates/selects the `kind-openstaticfish` context.
- Use `nix run .#tilt -- up` instead of calling a bare `tilt` from outside `nix develop`; the wrapper adds `kind`, `kubectl`, and `docker` to `PATH` so Tilt can load images into Kind.
- Use `nix run .#tilt -- down` to remove Tilt-managed resources.
- Use `nix develop` when running repeated manual commands; inside it, `setup-kind`, `tilt`, `kubectl`, `docker`, and `go` are available.
- Use `nix flake show` as the quick flake sanity check after editing `flake.nix`.
- `nix run .#setup-kind` — creates/selects the `kind-openstaticfish` Kind cluster.
- `nix run .#tilt -- up` — deploys all services with live reload.
- `nix run .#tilt -- down` — tears down Tilt-managed resources.
- `nix develop` — enters the dev shell with all deps (`tilt`, `kubectl`, `docker`, `go`, `jq`, `websocat`, etc.).
- `nix flake show` — sanity check after editing `flake.nix`.
## Verification Paths
- **Infra config changes**: `nix flake show`
- **Cluster + Tilt**: `nix run .#setup-kind && nix run .#tilt -- up`
- **Scraper**: `curl http://localhost:8080/health`
- **Site analyzer**: `curl http://localhost:8090/health`
- **Lightpanda CDP**: `curl http://localhost:9222/json/version`
- **Lightpanda MCP**: `curl http://localhost:8000/healthz`
## Lightpanda Browser Automation
- `lightpanda-cdp` exposes CDP on `localhost:9222`. No graphical rendering engine — CDP screenshots return blank. Use `browserless/chrome` or similar for visual screenshots.
- `lightpanda-mcp` exposes MCP Streamable HTTP on `localhost:8000/mcp`. Stateful sessions with 3-minute timeout. Tools: `goto`, `evaluate`, `markdown`, `links`, `click`, `fill`, etc.
- `scripts/test-lightpanda-search` — concurrent Bing search via MCP. Each worker gets its own browser session. Usage:
```bash
scripts/test-lightpanda-search --concurrent 5 --results 5
```
- `scripts/screenshot-lightpanda-page` — captures CDP screenshot (blank output — Lightpanda has no renderer). For visual screenshots use a real Chrome service.
## Gotchas
- Nix flakes only see git-tracked or staged files. After adding files referenced by the flake, Tiltfile, Dockerfile, or manifests, run `git add <file>` or `git add -A` before `nix run` verification.
- `Tiltfile` intentionally allows only `kind-openstaticfish` via `allow_k8s_contexts`; do not switch it to a broad context unless explicitly requested.
- Do not re-add `default_registry('localhost:5000')` unless a local registry is also created; this setup relies on Tilt loading images directly into Kind.
- The hello-world Dockerfile intentionally copies only `go.mod`; there is no `go.sum` because the app currently has no external Go dependencies.
- Docker must be running and reachable by the user before `setup-kind` or Tilt image builds can work.
## Verification
- For infra config changes, run `nix flake show`.
- For cluster/Tilt changes, the focused path is `nix run .#setup-kind` then `nix run .#tilt -- up`, followed by `curl http://localhost:8080`.
- There is no test, lint, formatter, CI, or pre-commit config in this repo yet; do not invent commands.
- **Nix flake visibility**: Nix only sees git-tracked or staged files. After adding files referenced by `flake.nix`, `Tiltfile`, `Dockerfile`, or manifests, run `git add <file>` or `git add -A` before `nix run` verification.
- **Tilt context**: `allow_k8s_contexts('kind-openstaticfish')` is locked to Kind; do not broaden unless explicitly requested.
- **No default registry**: Tilt loads images directly into Kind. Do not add `default_registry('localhost:5000')` unless you also create a local registry.
- **Docker must be running** before `setup-kind` or Tilt image builds.
- **Production manifests** live in `deploy/prod/`. On `main` pushes, CI builds GHCR images (`ghcr.io/openstaticfish/microservices/{site-analyzer,scraper,lightpanda-mcp}`) and commits updated `newTag` to `kustomization.yaml`.
- **Secrets**: managed by Doppler (`openstaticfish-microservices` / `dev` config). `scripts/apply-webshare-secret` reads from Doppler; if Doppler is not authenticated, it prints a warning and skips (non-fatal).
- **No tests, lint, or CI checks exist** in this repo yet — do not invent commands that don't exist.
+122 -1
View File
@@ -50,6 +50,13 @@ Analyze a site once Tilt is running:
scripts/analyze-site adaptive.co.uk
```
Check Lightpanda once Tilt is running:
```bash
curl http://localhost:9222/json/version
curl http://localhost:8000/healthz
```
Stop Tilt resources:
```bash
@@ -262,6 +269,108 @@ The response includes:
CDNs can hide the real origin. If a site is proxied through Cloudflare, the public DNS and IPs usually identify Cloudflare rather than the origin host. Origin detection is therefore best-effort and depends on leaked signals such as CSP entries, headers, HTML references, or provider-specific domains.
## Lightpanda Browser
The Lightpanda services provide a lightweight browser runtime for parallel automation workflows and agents.
### CDP Service
The `lightpanda-cdp` service runs the official `lightpanda/browser:nightly` image and exposes Lightpanda's Chrome DevTools Protocol server.
Local endpoint:
```text
http://localhost:9222
```
Health/version check:
```bash
curl http://localhost:9222/json/version
```
Clients can connect with Puppeteer or another CDP-compatible client. In local Tilt, use `localhost:9222`. In production, route through:
```text
http://apps.silverside-gopher.ts.net/lightpanda-cdp
```
If a client needs the WebSocket URL directly in production, keep the `/lightpanda-cdp` prefix on the WebSocket path so Traefik can route it to the service.
### MCP HTTP Service
The `lightpanda-mcp` service wraps `lightpanda mcp` with `supergateway` and exposes MCP Streamable HTTP for remote agents.
Local endpoint:
```text
http://localhost:8000/mcp
```
Health check:
```bash
curl http://localhost:8000/healthz
```
Example MCP initialize request:
```bash
curl -X POST http://localhost:8000/mcp \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2024-11-05","capabilities":{},"clientInfo":{"name":"curl-test","version":"1.0"}}}'
```
Production endpoint:
```text
http://apps.silverside-gopher.ts.net/lightpanda-mcp/mcp
```
### Concurrent Search via MCP
`scripts/test-lightpanda-search` demonstrates concurrent browser automation via the MCP endpoint. It searches Bing (Google and DuckDuckGo serve bot challenges in automated environments) and extracts organic result titles and URLs:
```bash
# Single search, 5 results
scripts/test-lightpanda-search
# 5 concurrent searches
scripts/test-lightpanda-search --concurrent 5
# Custom query
scripts/test-lightpanda-search --query "kubernetes helm" --results 3 --concurrent 5
```
Each worker initializes a stateful MCP session, navigates to Bing, evaluates JavaScript to extract `li.b_algo h2 a` elements, and decodes Bing redirect URLs to their real destinations.
### CDP Screenshot
`scripts/screenshot-lightpanda-page` captures a PNG via the CDP WebSocket endpoint. Lightpanda has no graphical rendering engine, so CDP screenshots return a blank canvas. For real visual screenshots, run a graphical headless Chrome service such as `browserless/chrome`:
```bash
# Bing search (Lightpanda CDP — no visual output)
scripts/screenshot-lightpanda-page
# Custom query
scripts/screenshot-lightpanda-page --query "kubernetes helm"
# Custom URL
scripts/screenshot-lightpanda-page --url https://example.com --output /tmp/example.png
```
### Scaling
Local Tilt starts one pod of each Lightpanda service. Scale manually in Kind when testing parallel workflows:
```bash
kubectl scale deployment/lightpanda-cdp-deployment --replicas=3
kubectl scale deployment/lightpanda-mcp-deployment --replicas=3
```
Production manifests start two replicas of each Lightpanda service and include HPAs that can scale each deployment up to ten pods based on CPU utilization.
## Secrets
Runtime secrets are managed with Doppler in the `openstaticfish-microservices` project using the `dev` config.
@@ -293,6 +402,8 @@ On pushes to `main`, GitHub Actions builds and pushes these GHCR images:
- `ghcr.io/openstaticfish/microservices/site-analyzer:<git-sha>`
- `ghcr.io/openstaticfish/microservices/scraper:main`
- `ghcr.io/openstaticfish/microservices/scraper:<git-sha>`
- `ghcr.io/openstaticfish/microservices/lightpanda-mcp:main`
- `ghcr.io/openstaticfish/microservices/lightpanda-mcp:<git-sha>`
After pushing images, CI updates `deploy/prod/kustomization.yaml` to the immutable `<git-sha>` tag and commits that change back to `main`. Flux then applies the production manifests from this repo.
@@ -308,19 +419,29 @@ Runtime secrets, including the scraper Webshare configuration, stay in Doppler p
├── deploy/
│ └── prod/
│ ├── kustomization.yaml
│ ├── lightpanda-cdp-deployment.yaml
│ ├── lightpanda-mcp-deployment.yaml
│ ├── scraper-deployment.yaml
│ └── site-analyzer-deployment.yaml
├── k8s/
│ ├── lightpanda-cdp.yaml
│ ├── lightpanda-mcp.yaml
│ ├── scraper.yaml
│ └── site-analyzer.yaml
├── scripts/
│ ├── analyze-site
│ └── setup-kind
│ ├── apply-webshare-secret
│ ├── load-test-site-analyzer
│ ├── screenshot-lightpanda-page
│ ├── setup-kind
│ └── test-lightpanda-search
└── services/
├── scraper/
│ ├── Dockerfile
│ ├── go.mod
│ └── main.go
├── lightpanda-mcp/
│ └── Dockerfile
└── site-analyzer/
├── Dockerfile
├── go.mod
+30 -1
View File
@@ -23,13 +23,16 @@ allow_k8s_contexts('kind-openstaticfish')
# Builds the Docker image and deploys the manifest with live reload
docker_build('scraper', './services/scraper')
docker_build('site-analyzer', './services/site-analyzer')
docker_build('lightpanda-mcp', './services/lightpanda-mcp')
k8s_yaml('./k8s/scraper.yaml')
k8s_yaml('./k8s/site-analyzer.yaml')
k8s_yaml('./k8s/lightpanda-cdp.yaml')
k8s_yaml('./k8s/lightpanda-mcp.yaml')
local_resource(
'webshare-secret',
cmd='doppler run -- bash ./scripts/apply-webshare-secret',
cmd='bash ./scripts/apply-webshare-secret',
labels=['utility'],
)
@@ -47,6 +50,18 @@ k8s_resource(
labels=['site-analyzer'],
)
k8s_resource(
workload='lightpanda-cdp-deployment',
port_forwards='9222:9222',
labels=['lightpanda'],
)
k8s_resource(
workload='lightpanda-mcp-deployment',
port_forwards='8000:8000',
labels=['lightpanda'],
)
# --- Local Resources ---
# Run a simple health check as a local resource
local_resource(
@@ -62,3 +77,17 @@ local_resource(
resource_deps=['site-analyzer-deployment'],
labels=['utility'],
)
local_resource(
'lightpanda-cdp-health-check',
cmd='curl -sf http://localhost:9222/json/version || echo "Lightpanda CDP not ready"',
resource_deps=['lightpanda-cdp-deployment'],
labels=['utility'],
)
local_resource(
'lightpanda-mcp-health-check',
cmd='curl -sf http://localhost:8000/healthz || echo "Lightpanda MCP not ready"',
resource_deps=['lightpanda-mcp-deployment'],
labels=['utility'],
)
@@ -21,3 +21,22 @@ spec:
services:
- name: scraper
port: 8080
- match: Host(`apps.silverside-gopher.ts.net`) && PathPrefix(`/lightpanda-cdp`)
kind: Rule
middlewares:
- name: microservices-strip-prefix
services:
- name: lightpanda-cdp
port: 9222
- match: Host(`apps.silverside-gopher.ts.net`) && PathPrefix(`/lightpanda-mcp`)
kind: Rule
middlewares:
- name: microservices-strip-prefix
services:
- name: lightpanda-mcp
port: 8000
sticky:
cookie:
name: lightpanda_mcp_pod
httpOnly: true
sameSite: lax
+8
View File
@@ -5,6 +5,12 @@ resources:
- webshare-api-externalsecret.yaml
- site-analyzer-deployment.yaml
- site-analyzer-service.yaml
- lightpanda-cdp-deployment.yaml
- lightpanda-cdp-service.yaml
- lightpanda-cdp-hpa.yaml
- lightpanda-mcp-deployment.yaml
- lightpanda-mcp-service.yaml
- lightpanda-mcp-hpa.yaml
- scraper-deployment.yaml
- scraper-service.yaml
- traefik-middleware-strip-prefix.yaml
@@ -14,3 +20,5 @@ images:
newTag: 076f5c70d35d162c19eced58997a556f37fe69ef
- name: ghcr.io/openstaticfish/microservices/scraper
newTag: 076f5c70d35d162c19eced58997a556f37fe69ef
- name: ghcr.io/openstaticfish/microservices/lightpanda-mcp
newTag: aade3e297de6f79f7ac5d3e80cc2b77ad9c3f9d9
@@ -0,0 +1,63 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: lightpanda-cdp
namespace: microservices
labels:
app: lightpanda-cdp
spec:
replicas: 2
selector:
matchLabels:
app: lightpanda-cdp
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template:
metadata:
labels:
app: lightpanda-cdp
spec:
terminationGracePeriodSeconds: 30
containers:
- name: lightpanda-cdp
image: lightpanda/browser:nightly
args:
- /bin/lightpanda
- serve
- --host
- 0.0.0.0
- --port
- "9222"
- --log-level
- warn
ports:
- containerPort: 9222
env:
- name: LIGHTPANDA_DISABLE_TELEMETRY
value: "true"
resources:
requests:
cpu: 250m
memory: 128Mi
limits:
cpu: "1"
memory: 512Mi
livenessProbe:
httpGet:
path: /json/version
port: 9222
initialDelaySeconds: 2
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
readinessProbe:
httpGet:
path: /json/version
port: 9222
initialDelaySeconds: 2
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2
+19
View File
@@ -0,0 +1,19 @@
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: lightpanda-cdp
namespace: microservices
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: lightpanda-cdp
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
+14
View File
@@ -0,0 +1,14 @@
apiVersion: v1
kind: Service
metadata:
name: lightpanda-cdp
namespace: microservices
spec:
type: ClusterIP
selector:
app: lightpanda-cdp
ports:
- name: cdp
protocol: TCP
port: 9222
targetPort: 9222
@@ -0,0 +1,56 @@
apiVersion: apps/v1
kind: Deployment
metadata:
name: lightpanda-mcp
namespace: microservices
labels:
app: lightpanda-mcp
spec:
replicas: 2
selector:
matchLabels:
app: lightpanda-mcp
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template:
metadata:
labels:
app: lightpanda-mcp
spec:
imagePullSecrets:
- name: ghcr-pull-secret
terminationGracePeriodSeconds: 30
containers:
- name: lightpanda-mcp
image: ghcr.io/openstaticfish/microservices/lightpanda-mcp:main
ports:
- containerPort: 8000
env:
- name: LIGHTPANDA_DISABLE_TELEMETRY
value: "true"
resources:
requests:
cpu: 250m
memory: 192Mi
limits:
cpu: "1"
memory: 768Mi
livenessProbe:
httpGet:
path: /healthz
port: 8000
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
readinessProbe:
httpGet:
path: /healthz
port: 8000
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2
+19
View File
@@ -0,0 +1,19 @@
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: lightpanda-mcp
namespace: microservices
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: lightpanda-mcp
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
+14
View File
@@ -0,0 +1,14 @@
apiVersion: v1
kind: Service
metadata:
name: lightpanda-mcp
namespace: microservices
spec:
type: ClusterIP
selector:
app: lightpanda-mcp
ports:
- name: http
protocol: TCP
port: 8000
targetPort: 8000
@@ -8,3 +8,5 @@ spec:
prefixes:
- /site-analyzer
- /scraper
- /lightpanda-cdp
- /lightpanda-mcp
+1
View File
@@ -45,6 +45,7 @@
# --- Utilities ---
curl
jq
websocat
yq-go
];
+80
View File
@@ -0,0 +1,80 @@
# --- Lightpanda CDP Kubernetes Manifests ---
# Managed by Tilt - run `tilt up` to deploy
# Tear down with: `tilt down`
apiVersion: apps/v1
kind: Deployment
metadata:
name: lightpanda-cdp-deployment
labels:
app: lightpanda-cdp
spec:
replicas: 1
selector:
matchLabels:
app: lightpanda-cdp
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template:
metadata:
labels:
app: lightpanda-cdp
spec:
terminationGracePeriodSeconds: 30
containers:
- name: lightpanda-cdp
image: lightpanda/browser:nightly
args:
- /bin/lightpanda
- serve
- --host
- 0.0.0.0
- --port
- "9222"
- --log-level
- warn
ports:
- containerPort: 9222
env:
- name: LIGHTPANDA_DISABLE_TELEMETRY
value: "true"
resources:
requests:
memory: "128Mi"
cpu: "250m"
limits:
memory: "512Mi"
cpu: "1000m"
livenessProbe:
httpGet:
path: /json/version
port: 9222
initialDelaySeconds: 2
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
readinessProbe:
httpGet:
path: /json/version
port: 9222
initialDelaySeconds: 2
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2
---
apiVersion: v1
kind: Service
metadata:
name: lightpanda-cdp-service
spec:
selector:
app: lightpanda-cdp
ports:
- name: cdp
protocol: TCP
port: 9222
targetPort: 9222
type: ClusterIP
+71
View File
@@ -0,0 +1,71 @@
# --- Lightpanda MCP Kubernetes Manifests ---
# Managed by Tilt - run `tilt up` to deploy
# Tear down with: `tilt down`
apiVersion: apps/v1
kind: Deployment
metadata:
name: lightpanda-mcp-deployment
labels:
app: lightpanda-mcp
spec:
replicas: 1
selector:
matchLabels:
app: lightpanda-mcp
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template:
metadata:
labels:
app: lightpanda-mcp
spec:
terminationGracePeriodSeconds: 30
containers:
- name: lightpanda-mcp
image: lightpanda-mcp
ports:
- containerPort: 8000
env:
- name: LIGHTPANDA_DISABLE_TELEMETRY
value: "true"
resources:
requests:
memory: "192Mi"
cpu: "250m"
limits:
memory: "768Mi"
cpu: "1000m"
livenessProbe:
httpGet:
path: /healthz
port: 8000
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
readinessProbe:
httpGet:
path: /healthz
port: 8000
initialDelaySeconds: 5
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2
---
apiVersion: v1
kind: Service
metadata:
name: lightpanda-mcp-service
spec:
selector:
app: lightpanda-mcp
ports:
- name: http
protocol: TCP
port: 8000
targetPort: 8000
type: ClusterIP
+4
View File
@@ -1,6 +1,10 @@
#!/usr/bin/env bash
set -euo pipefail
if [ -z "${DOPPLER_INJECTED:-}" ] && command -v doppler >/dev/null 2>&1 && doppler whoami >/dev/null 2>&1; then
exec doppler run -- bash "$0" "$@"
fi
if [ -n "${WEBSHARE_API_KEY:-}" ] && [ -n "${WEBSHARE_PROXY_USERNAME:-}" ] && [ -n "${WEBSHARE_PROXY_PASSWORD:-}" ]; then
kubectl create secret generic webshare-api \
--from-literal=api-key="$WEBSHARE_API_KEY" \
+136
View File
@@ -0,0 +1,136 @@
#!/usr/bin/env bash
set -euo pipefail
CDP_URL="${CDP_URL:-http://localhost:9222}"
QUERY="chattybox ai"
OUTPUT="lightpanda-google-search.png"
WAIT_SECONDS=4
usage() {
cat <<'USAGE'
Usage:
scripts/screenshot-lightpanda-page [options]
Take a screenshot with the Lightpanda CDP service.
Options:
-q, --query TEXT Google search query (default: "chattybox ai")
-u, --url URL URL to capture instead of building a Google search URL
-o, --output PATH Screenshot output path (default: lightpanda-google-search.png)
-w, --wait SECONDS Seconds to wait after navigation (default: 4)
-h, --help Show this help message
Examples:
scripts/screenshot-lightpanda-page
scripts/screenshot-lightpanda-page --output /tmp/google-consent.png
scripts/screenshot-lightpanda-page --url https://example.com --output /tmp/example.png
USAGE
}
URL=""
while [ "$#" -gt 0 ]; do
case "$1" in
-h|--help)
usage
exit 0
;;
-q|--query)
QUERY="$2"
shift 2
;;
-u|--url)
URL="$2"
shift 2
;;
-o|--output)
OUTPUT="$2"
shift 2
;;
-w|--wait)
WAIT_SECONDS="$2"
shift 2
;;
--*)
echo "Unknown option: $1" >&2
usage >&2
exit 1
;;
*)
break
;;
esac
done
if [ -z "$URL" ]; then
encoded_query=$(jq -rn --arg q "$QUERY" '$q|@uri')
URL="https://www.bing.com/search?q=$encoded_query"
fi
if ! command -v websocat >/dev/null 2>&1; then
echo "ERROR: websocat is required. Run through nix develop after updating the flake." >&2
exit 1
fi
version_json=$(curl -sf "$CDP_URL/json/version") || {
echo "ERROR: Lightpanda CDP not reachable at $CDP_URL" >&2
echo "Make sure Tilt is running: nix run .#tilt -- up" >&2
exit 1
}
ws_url=$(printf '%s\n' "$version_json" | jq -r '.webSocketDebuggerUrl')
ws_url="${ws_url/0.0.0.0/localhost}"
echo "Capturing: $URL"
echo "Output: $OUTPUT"
coproc CDP { websocat "$ws_url"; }
trap 'kill "$CDP_PID" >/dev/null 2>&1 || true' EXIT
send_cdp() {
local id="$1"
local method="$2"
local params="{}"
if [ "$#" -ge 3 ]; then
params="$3"
fi
jq -cn --argjson id "$id" --arg method "$method" --argjson params "$params" \
'{id: $id, method: $method, params: $params}' >&"${CDP[1]}"
}
read_cdp_id() {
local wanted_id="$1"
local line id
while IFS= read -r line <&"${CDP[0]}"; do
id=$(printf '%s\n' "$line" | jq -r '.id // empty' 2>/dev/null || true)
if [ "$id" = "$wanted_id" ]; then
printf '%s\n' "$line"
return 0
fi
done
return 1
}
send_cdp 1 "Page.enable"
read_cdp_id 1 >/dev/null
send_cdp 2 "Emulation.setDeviceMetricsOverride" '{"width":1280,"height":900,"deviceScaleFactor":1,"mobile":false}'
read_cdp_id 2 >/dev/null
navigate_params=$(jq -cn --arg url "$URL" '{url: $url}')
send_cdp 3 "Page.navigate" "$navigate_params"
read_cdp_id 3 >/dev/null
sleep "$WAIT_SECONDS"
send_cdp 4 "Page.captureScreenshot" '{"format":"png","fromSurface":true}'
screenshot_response=$(read_cdp_id 4)
screenshot_data=$(printf '%s\n' "$screenshot_response" | jq -r '.result.data // empty')
if [ -z "$screenshot_data" ]; then
echo "ERROR: Screenshot response did not include image data" >&2
printf '%s\n' "$screenshot_response" >&2
exit 1
fi
printf '%s' "$screenshot_data" | base64 -d > "$OUTPUT"
echo "Saved screenshot to $OUTPUT"
+289
View File
@@ -0,0 +1,289 @@
#!/usr/bin/env bash
set -euo pipefail
MCP_URL="${MCP_URL:-http://localhost:8000/mcp}"
CONCURRENT=1
QUERY="chattybox ai"
MAX_RESULTS=5
usage() {
cat <<'USAGE'
Usage:
scripts/test-lightpanda-search [options]
Test Lightpanda MCP by searching Google and extracting top results.
Options:
-c, --concurrent N Number of concurrent searches (default: 1, max: 10)
-q, --query TEXT Search query (default: "chattybox ai")
-r, --results N Number of results to return per worker (default: 5)
-h, --help Show this help message
Examples:
scripts/test-lightpanda-search
scripts/test-lightpanda-search --concurrent 5
scripts/test-lightpanda-search --query "kubernetes helm" --results 3 --concurrent 5
USAGE
}
while [ "$#" -gt 0 ]; do
case "$1" in
-h|--help)
usage
exit 0
;;
-c|--concurrent)
if [ "$#" -lt 2 ]; then
echo "Missing value for $1" >&2
usage >&2
exit 1
fi
CONCURRENT="$2"
shift 2
;;
-q|--query)
if [ "$#" -lt 2 ]; then
echo "Missing value for $1" >&2
usage >&2
exit 1
fi
QUERY="$2"
shift 2
;;
-r|--results)
if [ "$#" -lt 2 ]; then
echo "Missing value for $1" >&2
usage >&2
exit 1
fi
MAX_RESULTS="$2"
shift 2
;;
--*)
echo "Unknown option: $1" >&2
usage >&2
exit 1
;;
*)
break
;;
esac
done
if [ "$CONCURRENT" -lt 1 ]; then CONCURRENT=1; fi
if [ "$CONCURRENT" -gt 10 ]; then CONCURRENT=10; fi
ENCODED_QUERY=$(jq -rn --arg q "$QUERY" '$q|@uri')
SEARCH_URL="https://www.bing.com/search?q=$ENCODED_QUERY"
health_url="${MCP_URL%/mcp}"
if [ -z "$health_url" ]; then health_url="http://localhost:8000"; fi
if ! curl -sf "$health_url/healthz" >/dev/null 2>&1; then
echo "ERROR: Lightpanda MCP not reachable at $health_url" >&2
echo "Make sure Tilt is running: nix run .#tilt -- up" >&2
exit 1
fi
# Extract JSON from SSE response body
extract_json_from_sse() {
sed -n 's/^data: //p'
}
# Extract session ID from response headers file
extract_session_id() {
local headers_file="$1"
grep -i '^Mcp-Session-Id:' "$headers_file" 2>/dev/null | awk '{print $2}' | tr -d '\r' || true
}
search_worker() {
local worker_id="$1"
local tmpdir
tmpdir=$(mktemp -d)
local headers_file="$tmpdir/headers.txt"
echo "[$worker_id] Searching for: $QUERY"
# Initialize session
local init_payload
init_payload=$(jq -n \
--argjson id "$worker_id" \
'{
jsonrpc: "2.0",
id: $id,
method: "initialize",
params: {
protocolVersion: "2024-11-05",
capabilities: {},
clientInfo: {
name: "search-test-\($id)",
version: "1.0"
}
}
}')
local init_response
init_response=$(curl -sf -X POST "$MCP_URL" \
-D "$headers_file" \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-d "$init_payload" 2>"$tmpdir/curl.err") || {
echo "[$worker_id] ERROR: Initialize failed" >&2
cat "$tmpdir/curl.err" >&2
rm -rf "$tmpdir"
return 1
}
local session_id
session_id=$(extract_session_id "$headers_file")
if [ -z "$session_id" ]; then
echo "[$worker_id] ERROR: No session ID received" >&2
rm -rf "$tmpdir"
return 1
fi
# Navigate to search page
local goto_payload
goto_payload=$(jq -n \
--argjson id "$((worker_id + 100))" \
--arg url "$SEARCH_URL" \
'{
jsonrpc: "2.0",
id: $id,
method: "tools/call",
params: {
name: "goto",
arguments: {
url: $url
}
}
}')
curl -sf -X POST "$MCP_URL" \
-H "Mcp-Session-Id: $session_id" \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-d "$goto_payload" >/dev/null 2>"$tmpdir/goto.err" || {
echo "[$worker_id] WARNING: Goto may have issues" >&2
cat "$tmpdir/goto.err" >&2
}
sleep 2
# Extract results
local js_file="$tmpdir/extract.js"
cat > "$js_file" <<'JSEOF'
(() => {
function decodeBingUrl(href) {
try {
const url = new URL(href);
const encoded = url.searchParams.get('u');
if (!encoded || !encoded.startsWith('a1')) return href;
const base64 = encoded.slice(2).replace(/-/g, '+').replace(/_/g, '/');
const padded = base64.padEnd(Math.ceil(base64.length / 4) * 4, '=');
return atob(padded);
} catch (_) {
return href;
}
}
const results = [];
const links = Array.from(document.querySelectorAll('li.b_algo h2 a[href]'));
for (const a of links) {
if (results.length >= 5) break;
const title = a.textContent.replace(/\s+/g, ' ').trim();
const href = decodeBingUrl(a.href);
if (title.length < 3) continue;
if (!href.startsWith('http')) continue;
results.push({title, url: href});
}
return JSON.stringify({
url: location.href,
title: document.title,
resultCount: results.length,
results,
});
})()
JSEOF
local js
js=$(tr '\n' ' ' < "$js_file")
local eval_payload
eval_payload=$(jq -n \
--argjson id "$((worker_id + 200))" \
--arg script "$js" \
'{
jsonrpc: "2.0",
id: $id,
method: "tools/call",
params: {
name: "evaluate",
arguments: {
script: $script
}
}
}')
local eval_response
eval_response=$(curl -sf -X POST "$MCP_URL" \
-H "Mcp-Session-Id: $session_id" \
-H "Content-Type: application/json" \
-H "Accept: application/json, text/event-stream" \
-d "$eval_payload" 2>"$tmpdir/eval.err") || {
echo "[$worker_id] ERROR: Evaluate failed" >&2
cat "$tmpdir/eval.err" >&2
rm -rf "$tmpdir"
return 1
}
local result_text
result_text=$(printf '%s\n' "$eval_response" | extract_json_from_sse | jq -r '.result.content[0].text // empty' 2>/dev/null)
echo "[$worker_id] Results:"
if [ -n "$result_text" ]; then
local result_count
result_count=$(printf '%s\n' "$result_text" | jq -r '.resultCount // 0' 2>/dev/null || printf '0')
if [ "$result_count" -gt 0 ]; then
printf '%s\n' "$result_text" | jq -r --argjson max "$MAX_RESULTS" '.results[:$max] | .[] | " \(.title)\n \(.url)\n"' 2>/dev/null || printf ' Raw: %s\n' "$result_text"
else
echo " (No search results found)"
printf '%s\n' "$result_text" | jq -r '" Page: \(.title) at \(.url)"' 2>/dev/null || printf ' Raw: %s\n' "$result_text"
rm -rf "$tmpdir"
return 1
fi
else
echo " (No results found)"
local parsed
parsed=$(printf '%s\n' "$eval_response" | extract_json_from_sse | jq . 2>/dev/null)
if [ -n "$parsed" ]; then
printf ' Response: %s\n' "$parsed"
else
printf ' Raw response:\n%s\n' "$eval_response"
fi
fi
rm -rf "$tmpdir"
echo "[$worker_id] Done"
}
echo "=== Lightpanda Search Test ==="
echo "Query: $QUERY"
echo "URL: $SEARCH_URL"
echo "Workers: $CONCURRENT"
echo "Results per worker: $MAX_RESULTS"
echo ""
pids=()
for i in $(seq 1 "$CONCURRENT"); do
search_worker "$i" &
pids+=("$!")
sleep 0.5
done
for pid in "${pids[@]}"; do
wait "$pid" || true
done
echo ""
echo "=== All workers completed ==="
+21
View File
@@ -0,0 +1,21 @@
# --- Lightpanda MCP Service Dockerfile ---
# Exposes Lightpanda's stdio MCP server over Streamable HTTP for remote agents.
FROM lightpanda/browser:nightly AS lightpanda
FROM oven/bun:1.3.0-debian
RUN apt-get update -yq && \
apt-get install -yq --no-install-recommends ca-certificates tini && \
rm -rf /var/lib/apt/lists/*
RUN bun add --global [email protected]
COPY --from=lightpanda /bin/lightpanda /usr/local/bin/lightpanda
ENV LIGHTPANDA_DISABLE_TELEMETRY=true
EXPOSE 8000
ENTRYPOINT ["/usr/bin/tini", "--"]
CMD ["supergateway", "--stdio", "lightpanda mcp", "--outputTransport", "streamableHttp", "--stateful", "--sessionTimeout", "180000", "--streamableHttpPath", "/mcp", "--healthEndpoint", "/healthz", "--port", "8000", "--logLevel", "info"]