Local end-to-end runbook

Step-by-step instructions to bring the full Neon Law Navigator stack up in a local KIND cluster and walk through the OIDC + admin flow in Chrome. Every command in this document is verified against the manifests in the repo. The runtime steps (docker, kind, kubectl) run on your machine, so they're marked with 🔧 you run; everything else has been mechanically validated.

0. Prerequisites

docker --version    # any modern Docker / colima / OrbStack works
kind --version      # >= 0.32 — `kind load` needs it for the v1.36.1 node's containerd config v4
kubectl version --client
helm version        # OCI Helm chart installs the Restate Operator
restate --version   # Restate CLI — workflows-service registration

On macOS, install the cluster tooling with Homebrew (Docker comes from Docker Desktop / OrbStack / colima):

brew install kind kubectl helm          # cluster tooling
brew install restatedev/tap/restate     # Restate CLI

You also need to be in the docker group — verify with docker info (it should succeed without sudo). If your machine wants sudo docker, either add yourself to the group via sudo usermod -aG docker $USER (then afterwards log out and log back in to refresh) or run every cargo run -p cli -- … invocation with sudo -E.

1. Bring up the cluster (🔧 you run)

cd ~/Code/navigator
cargo run --release -p cli -- dev kind up

What this does (look in cli/src/devx/mod.rs → kind_up_steps):

kind create cluster --name navigator --config k8s/kind-config.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/.../deploy.yaml
kubectl --namespace ingress-nginx wait --for=condition=ready pod ...
helm upgrade --install restate-operator oci://ghcr.io/restatedev/restate-operator-helm ...

Expected output ends with something like:

Set kubectl context to "kind-navigator"
pod/ingress-nginx-controller-... condition met

Takes ~60 seconds on a warm Docker daemon (longer the first time since it pulls the KIND node image).

Quick sanity check

kubectl cluster-info --context kind-navigator
kubectl get nodes
# Expect: navigator-control-plane Ready, navigator-worker Ready

2. Pull the published images + deploy everything (🔧 you run)

cargo run --release -p cli -- dev deploy

What this does:

Resolve the YY.M.D ghcr tag to pull — NAVIGATOR_IMAGE_TAG if set, else the latest published tag.
docker pull ghcr.io/<owner>/navigator-web:<tag> then retag it to navigator-web:dev (the name the manifests reference), and the same for navigator-workflows-service. CI (deploy.yml) builds and publishes these images; the local loop no longer builds them — pin a known-good release with NAVIGATOR_IMAGE_TAG=YY.M.D for a reproducible demo.
kind load docker-image navigator-web:dev --name navigator — pushes the pulled image into the cluster.
kubectl apply -k k8s/overlays/kind — the full stack including navigator-web.
kubectl --namespace navigator rollout status deployment/navigator-web --timeout=300s.

Expected final line:

deployment "navigator-web" successfully rolled out

What's now running

kubectl --namespace navigator get pods

Expect ~7 rows, all Running or Completed:

NAME                              READY   STATUS      RESTARTS   AGE
keycloak-xxxxxxxxxx-xxxxx         1/1     Running     0          2m
garage-0                          1/1     Running     0          2m
navigator-web-xxxxxxxxxx-xxxxx    2/2     Running     0          1m
opa-xxxxxxxxxx-xxxxx              1/1     Running     0          2m
postgres-xxxxxxxxxx-xxxxx         1/1     Running     0          2m
restate-0                         1/1     Running     0          2m

navigator-web shows 2/2 because the OPA sidecar runs in the same pod. The Rust dev up bootstrap configures Garage's single-node layout, lane buckets, and lane-scoped keys before dependent workloads become ready.

If any pod is stuck Pending or CrashLoopBackOff:

kubectl --namespace navigator describe pod <name>     # events at the bottom
kubectl logs -n navigator <name> --all-containers --tail=100

3. Grant staff the `staff` role in the DB

This is the deliberate split: Keycloak knows the staff user exists but the authz tier lives in our persons table. Every person carries exactly one role — client, staff, or admin — in the persons.role column (see docs/access-model.md). The Keycloak realm import creates the staff user in the IdP; the matching persons row is ours to seed.

Seed before you sign in. Sign-in does not create a persons row: an IdP-authenticated email with no pre-seeded row is refused with a 403 (no pre-seeded persons row for the supplied email). The sole exception is NAVIGATOR_BOOTSTRAP_ADMIN_EMAIL, which is created just-in-time as admin and role-healed back to admin on each sign-in. So do this before step 4 opens the browser:

cargo run -p cli -- dev grant-staff

That seeds staff@neonlaw.com with role = 'staff' and is idempotent — re-run it any time to heal the row back. It takes --database-url to target a worktree database instead of the in-cluster one.

Prefer the CLI over hand-rolled SQL here. persons.id, inserted_at, and updated_at are NOT NULL with no DB default, so a partial INSERT fails on the missing id; grant-staff supplies them and upserts on the email conflict.

Verify:

kubectl --namespace navigator exec deployment/postgres -- \
    psql -U navigator -d navigator -c "SELECT id, email, oidc_subject, role FROM persons;"

You should see staff with role = staff. After it logs in, oidc_subject populates with its Keycloak UUID.

4. Open Chrome (🔧 you do)

Five URLs to visit, in order. Each one exercises a different piece of the stack.

4.1 Neon Law Navigator home page

http://localhost:8080

Verifies: nginx-ingress → navigator-web Service → pod → axum handler chain. Should render the home page immediately (no auth required).

4.2 Start the OIDC flow

http://localhost:8080/auth/login?return_to=/portal

What happens behind the scenes:

navigator-web generates a PKCE verifier + a CSRF state.
Sets the navigator_pre_auth cookie (HMAC-signed, HttpOnly).
302-redirects to Keycloak's /realms/navigator/protocol/openid-connect/auth?...&code_challenge=....

Chrome will follow the 302 and land on the Keycloak login page.

Chrome lands on Keycloak at http://localhost:8080/keycloak/realms/navigator/protocol/openid-connect/auth?... — the nginx ingress routes /keycloak/* to the in-cluster Keycloak Service. The pod separately reaches Keycloak via cluster DNS at http://keycloak:8080/keycloak/... for the backchannel /token exchange; Keycloak's hostname-v2 config (KC_HOSTNAME + KC_HOSTNAME_BACKCHANNEL_DYNAMIC=true) keeps the two channels straight.

Credentials (from k8s/staging/keycloak.yaml realm import):

Field	Value
Username	`staff`
Password	`staff`

Click "Sign In". Keycloak issues a one-time code, redirects back to http://localhost:8080/auth/callback?code=...&state=....

4.4 Callback completes, /portal renders

If you pre-seeded the staff role in step 3:

Callback decodes id_token (sub=<keycloak-uuid>, email=staff@neonlaw.com). upsert_person_from_claims matches the seeded row by email, promotes it (stamps oidc_subject), reads role = staff.
Session cookie set: { sub, email, person_id, role: "staff", exp, csrf_token }. 302 → /portal — the role-aware landing. OPA allows any authenticated person here, so the dashboard renders; role == "staff" only becomes load-bearing on the /staff/* routes below.

Now try the admin routes — each /staff/* path hits the staff gate, while /portal and /portal/projects need only an authenticated session:

http://localhost:8080/portal http://localhost:8080/staff/people http://localhost:8080/staff/entities http://localhost:8080/staff/jurisdictions http://localhost:8080/staff/entity-types http://localhost:8080/staff/templates http://localhost:8080/staff/questions http://localhost:8080/portal/projects

All should return 200 and render their table.

4.5 Revoke the role, see the gate fire

kubectl --namespace navigator exec deployment/postgres -- \
    psql -U navigator -d navigator -c \
    "UPDATE persons SET role = 'client' WHERE email = 'staff@neonlaw.com';"

Then in Chrome:

Hit http://localhost:8080/auth/logout to clear the session.
Re-do http://localhost:8080/auth/login?return_to=/portal.
Log in as staff again.
/portal still loads (she is authenticated), but /staff now returns 403.

This proves the gate is database-sourced — Keycloak hasn't changed, the token is identical, but access is gone.

5. Other consoles

URL	Login
http://localhost:30080/keycloak/	Keycloak admin (`admin` / `admin`)
http://localhost:30900	Garage S3 endpoint

The Keycloak admin console lets you confirm the navigator realm, navigator-web client, and staff user are all live. The Garage endpoint speaks S3 and requires the credentials written to .devx/env; use the storage conformance test rather than an unauthenticated browser request to inspect it.

6. Tail logs while you click around

# navigator-web (web container only)
kubectl logs -n navigator -f deployment/navigator-web -c web

# the OPA sidecar in the same pod
kubectl logs -n navigator -f deployment/navigator-web -c opa

# Keycloak (verbose; grep for 'event' or 'authorize')
kubectl logs -n navigator -f deployment/keycloak

7. Tear down

cargo run --release -p cli -- dev kind down

Removes the entire KIND cluster. Re-run cargo run --release -p cli -- dev deploy; it calls dev kind up first.

7b. Fast loop — `web` on the host, deps in KIND

KIND's full and dependency-only wrappers both consume k8s/staging, the provider-neutral disposable staging base. The base owns Postgres, Keycloak, OPA, Restate, the worker, the git-serving volume contract, the mounted brand bundle, and telemetry. KIND adds Garage and the credential-free local forge; k8s/overlays/cloud-staging instead selects hosted GCS through Workload Identity. Environment destruction and reset remain explicit lifecycle actions, not application boot behavior.

Guarded staging lifecycle

Use NAVIGATOR_ENVIRONMENT=staging cargo run -p cli -- dev staging up|reset|status|down to manage the disposable KIND boundary. Reset and down require the explicit KIND context plus Navigator-managed staging labels and an immutable environment ID. Reset deletes the complete namespace, including Postgres, Garage PVCs, the Restate journal, and local git-serving state.

When you're actively editing the web crate, running navigator dev deploy on every change is too slow. Reach instead for navigator dev up: KIND hosts every dependency, but cargo run -p web runs in your shell so a Ctrl-C + cargo run restart costs a single Rust rebuild rather than a docker build + kind load + rollout.

Bring it up

cargo run --release -p cli -- dev up

What this does (look in cli/src/devx/mod.rs):

kind create cluster (skipped if one already exists with the same name).
Installs nginx-ingress, then kubectl apply for every directory under k8s/ except k8s/web/.
Waits for Garage, provisions its layout/buckets/keys, then waits for the remaining dependencies.

Starts background kubectl port-forward processes:

Service	In-cluster	Host
Postgres	`:5432`	`localhost:15432` (5432 is often taken by a host Postgres)
Restate ingress	`:8080`	`localhost:9080` (8080 is taken by KIND's nginx)
Restate admin	`:9070`	`localhost:9070`
OPA	`:8181`	`localhost:8181`
Keycloak	NodePort `:30080`	`localhost:30080` (kind-config mapping)
Garage	`:3900`	`localhost:30900` (CLI-managed port-forward)

Writes PIDs to .devx/pids and the env file to .devx/env.

Run the web server locally

set -a; source .devx/env; set +a
cargo run -p web

The set -a block exports every KEY=VALUE line in .devx/env into your shell. cargo run -p web then binds :3001 with DATABASE_URL pointing at the in-cluster Postgres via the forwarded port, OAuth pointing at Keycloak on :30080, OPA on :8181, Garage on :30900.

The Keycloak realm in k8s/staging/keycloak.yaml whitelists http://localhost:3001/auth/callback, the :8080 ingress redirect URI, and a http://localhost:*/auth/callback wildcard, so the OIDC flow works in either deploy mode — and on any per-worktree web port (section 7c) — without realm edits.

Open in Chrome

URL	What it verifies
http://localhost:3001	Local `cargo run -p web` → home page (no auth)
http://localhost:3001/auth/login?return_to=/portal	OIDC flow against in-cluster Keycloak
http://localhost:30080/keycloak/	Keycloak admin (`admin` / `admin`)
http://localhost:30900	Garage S3 endpoint

The OIDC login uses staff / staff (same realm as section 4.3 above). After login, /portal renders for any authenticated person; the staff gate applies to /staff/*, reached via the port-forward instead of an in-pod sidecar (same policy either way).

Hot-restart the web

Edit code, then in the same shell:

# Ctrl-C the running web, then:
cargo run -p web

No kubectl, no docker, no kind interaction needed — only the web binary recompiles. The cluster keeps its state across restarts.

Keep the deps up across sessions (the persistent fixture)

The dependency tier — the KIND cluster with Postgres, Keycloak, OPA, Garage, and Restate — is a reusable dev fixture, not a per-task resource. Stand it up once and leave it running between sessions; at handoff clean up only the host-side web process, your task's target/ artifacts, and rebuilt dev images. Creating the cluster is the slow step, so reuse beats rebuild. Reach for full teardown (below) only when you intend to rebuild from scratch.

Re-arm after a sleep or reboot. The host-side kubectl port-forward processes (PIDs in .devx/pids) die when the laptop sleeps or the shell closes, even though the cluster keeps running — so web suddenly can't reach Postgres or OPA. Just re-run dev up: it detects the existing cluster (KIND cluster 'navigator' already exists, reusing), re-applies the manifests (a no-op for unchanged pods, so Postgres data is preserved), kills any stale port-forwards, and respawns them. No cluster recreate, so it returns in seconds and rewrites .devx/env.
Bounce one wedged dependency without touching the others:
```
kubectl --namespace navigator rollout restart deployment/keycloak   # or opa, postgres
```
Keycloak re-imports its realm from the ConfigMap and OPA re-reads its policy bundle, so those come back clean. But Garage and Restate use PVCs and survive pod restarts. After a postgres bounce, restart web so its boot path re-runs migrations and the canonical seed. Re-run dev up first if the port-forwards also need re-arming.
Reclaim disk without losing the fixture. Prune images, build cache, and any leaked postgres:17-alpine testcontainers (docker image prune -a, docker builder prune). Never prune Docker volumes or run down/kind delete for routine cleanup — those destroy the cluster and force a full rebuild. See the Resource cleanup section of agent-workflows.md.

Tear down

Full teardown — only when you want a clean rebuild or to reclaim the cluster's disk, not as routine session cleanup:

cargo run --release -p cli -- dev down   # kills port-forwards, then `kind delete cluster`

7c. One checkout and host port per git worktree

When several agents or developers work in parallel, each worktree needs its own host web process without colliding on the port. navigator dev worktree-env provides that on top of the section-7b fast loop: the KIND dependency tier and its navigator Postgres database stay the one shared persistent fixture, so this topology does not isolate data. Each worktree gets its own host web port and .devx descriptor. The first successful setup persists the port, and later runs reuse it even when a detached harness checkout gains a topic branch. Setup and teardown hold one repository-wide lock, and setup treats every other live worktree descriptor as a reservation, so a stopped web process cannot lose its port during parallel work.

Use one command to prepare the topic checkout and stand up its environment:

cargo run -p cli -- dev worktree-env up --branch <topic-branch>

If NAVIGATOR_WORKTREE_PATH or an integration-compatible worktree path is set, the CLI fetches origin/main and creates the topic branch in that existing checkout. An explicit --path has the same meaning. Otherwise it creates .worktrees/<topic> beside the primary checkout. Continue in the task checkout path printed by the command. In both cases, that one worktree owns one host port and one .devx descriptor; the database remains shared.

# In the worktree (cwd = the worktree root):
cargo run -p cli -- dev worktree-env up      # migrate the shared database, reserve a port, write .devx/env
set -a; source .devx/env; set +a
cargo run -p web                         # listens on this worktree's own port (e.g. :3042)

cargo run -p cli -- dev worktree-env status  # slug, mode, database, port, reachability
cargo run -p cli -- dev worktree-env down    # remove this worktree's .devx state; leaves shared deps and data

up brings the shared deps up automatically if they aren't already (otherwise it reuses them — only the first worktree pays the cluster-creation cost). It is idempotent for the same worktree path and keeps the same port across re-runs, so the OAuth redirect URI stays stable; the realm whitelists http://localhost:*/auth/callback, so OIDC login works on any worktree port. Host web and the in-cluster workflows-service both use navigator, so Restate-backed flows stay observable while migrations and rows remain shared across worktrees. down is idempotent and leaves the shared port-forwards and cluster running because other worktrees depend on them. Use navigator dev down for full teardown.

Authenticated browser checks in a worktree

Use this path for portal/staff screenshots, browser-driven review, and any local check that needs a real Keycloak session while web runs from the current checkout:

cargo run -p cli -- dev worktree-env up --path "$PWD"
cargo run -p cli -- dev grant-staff

set -a; source .devx/env; set +a
SENDGRID_EVENTS_SECRET=kind-stub \
SENDGRID_EVENTS_PUBLIC_KEY=kind-stub \
DOCUSIGN_HMAC_KEY=kind-stub \
  cargo run -p web

Then open the worktree URL from .devx/env, sign in through Keycloak as staff / staff, and capture the live page. The session must come from the OIDC flow; do not hand-write app cookies. dev grant-staff reads the same DATABASE_URL that web uses after .devx/env is loaded, so the login row and staff gate land in the shared navigator database. The same database agreement lets this host web drive Restate-backed flows through the in-cluster worker.

Reproduce the deploy.yml browser gate — `dev browser-e2e`

navigator dev browser-e2e runs the exact gate .github/workflows/deploy.yml's integration job runs — the browser (browser_e2e) and accessibility (accessibility_e2e) suites against the live stack — so regressions surface before a YY.M.D tag is cut, not after. It assembles the harness the CI job wires by hand: it resolves and caches the pinned Chrome for Testing build — the single CHROME_FOR_TESTING_VERSION constant in cli/src/devx/chrome.rs, kept in lockstep with deploy.yml by a guard test — starts that build's chromedriver on a free port, verifies Postgres and web are reachable, grants Staff in the database web reads, then runs both suites with NAV_REQUIRE_HARNESS=1 (so a "harness unreachable" self-skip fails instead of passing green).

Bring the fixture up and start web first (the previous section), then, in a shell with .devx/env sourced:

cargo run -p cli -- dev browser-e2e

It reads the worktree's NAV_BASE_URL/PORT and DATABASE_URL from the sourced .devx/env; override either with --base-url / --database-url. The pinned Chrome downloads once to ~/.cache/navigator/chrome-for-testing/ (set NAVIGATOR_CHROME_CACHE_DIR to relocate) and is reused across worktrees.

On a worktree, every suite passes except the one Restate-backed test — the retainer questionnaire walk (staff_walks_…_end_to_end) — which fails with journal: RecordNotFound for the database-agreement reason above. The command prints a WARNING: Restate-backed retainer-walk test will fail on this database pre-flight so the failure is expected, not a mystery. For full parity with the CI gate (all suites green), run it against the shared-database topology of section 7b (navigator dev up, web and the worker both on navigator):

cargo run -p cli -- dev browser-e2e \
    --base-url http://localhost:3001 \
    --database-url postgres://navigator:navigator@localhost:15432/navigator

Agent-harness wiring. A worktree-based integration exports NAVIGATOR_WORKTREE_PATH and runs the idempotent setup command when it creates the checkout:

NAVIGATOR_WORKTREE_PATH=/path/to/worktree cargo run -p cli -- dev worktree-env up

If the integration does not provide a teardown hook, the agent runs the matching cleanup command while the worktree still exists:

cargo run -p cli -- dev worktree-env down

web still needs the Doppler-only secrets (SENDGRID_EVENTS_SECRET, SENDGRID_EVENTS_PUBLIC_KEY, DOCUSIGN_HMAC_KEY) to boot — run it under doppler run … or with a gitignored stub .env, exactly as the section-7b loop does.

Full-stack demo. To show the product running entirely in KIND from the published ghcr images (no host web), use cargo run -p cli -- dev worktree-env up --demo (optionally --tag YY.M.D to pin a release). This delegates to navigator dev deploy; it is a single in-cluster stack, reached through the ingress at http://localhost:8080.

7d. Running the test suite

cargo test needs exactly one Postgres for the whole run — never one per test binary. Two ways to get it:

# Zero setup: the first run starts ONE reuse-labeled container; every
# later run, in any crate, reuses it. Reclaim it any time with:
#   docker rm -f $(docker ps -aq --filter label=org.navigator.test-postgres=shared)
cargo test --workspace

# Or point tests at an already-running Postgres (no Docker in the test
# path) — e.g. the KIND Postgres from `navigator dev up`:
export TEST_DATABASE_URL=postgres://navigator:navigator@localhost:15432/navigator
cargo test --workspace

Each test still creates its own test_<id> schema, so tests run in parallel and never pollute the dev data even when they share a server. The full rationale and the env contract are in test-database.md.

8. What this verifies end-to-end

Walking through steps 1–5 demonstrates, live:

Kubernetes deploy of the full stack on a single laptop.
OIDC Authorization Code + PKCE against a real Keycloak.
Person upsert keyed on the OIDC sub, with email-match promotion for seeded rows.
DB-sourced authz — flipping persons.role in Postgres changes the gate decision on the next login.
OPA policy decision via the in-pod sidecar (zero-RTT localhost call).
Ingress + Service + NodePort routing through KIND's port mappings (k8s/kind-config.yaml).

The same three guarantees are verified statically by web/tests/oidc_e2e.rs — six integration tests against wiremock'd IdP + OPA. Run them with cargo test -p web --test oidc_e2e.