Cloud operations

This page replaces the old private cloud runbooks with one common operating model. Public docs are the shared surface every LLM and human maintainer should read first.

Neon Law Navigator is GCP-wired and provider-agnostic. The production path uses GKE Autopilot, Cloud SQL for Postgres, GCS, Secret Manager, Cloud Logging, Cloud Trace, BigQuery billing export, and Restate Cloud. The application code keeps the cloud boundary behind traits, protocols, and env vars: cloud::StorageService, SeaORM/Postgres, OIDC, OPA, Restate, SendGrid, Kubernetes, and web::agent_router::AgentRouter.

Former private-runbook coverage

The collapse rule is simple: durable policy, invariants, architecture, and operator recipes live in docs/.

Deployment profiles and operator boundary

Exact NAVIGATOR_ENVIRONMENT=staging selects the disposable Kubernetes profile. Empty or unset selects production; mixed case, surrounding whitespace, and every other nonempty value fail parsing. Production keeps hosted-persistence requirements: real GCS and no emulator endpoint. Normal staging uses real non-production SendGrid and DocuSign demo credentials stamped NAVIGATOR_CREDENTIAL_ENVIRONMENT=staging; only NAVIGATOR_CI_HARNESS=1 may use fakes.

The deployment operator who supplies Kubernetes, cloud accounts, secrets, domains, and environment values is not a Navigator application admin. persons.role = 'admin' is an application authorization tier and grants no cloud or cluster authority.

Local development

navigator ops staging requires exact NAVIGATOR_ENVIRONMENT=staging, an explicit non-production context, Navigator-managed labels, and an immutable environment ID before reset or deletion.

The standard local loop is KIND through the navigator CLI:

cargo run --release -p cli -- dev up   # once; reuses an existing cluster on re-run
set -a; source .devx/env; set +a
cargo run -p web                                  # Ctrl-C and re-run to iterate
cargo run --release -p cli -- dev down                # full teardown — only for a clean rebuild, not routine cleanup

dev up brings up Postgres, Keycloak, Garage, OPA, Restate, workflows-service, and Grafana LGTM in KIND, then writes .devx/env for the host-side web process. The cluster is a persistent dev fixture: leave it up between sessions and re-run dev up to restore port-forwards after a sleep or reboot (it reuses the existing cluster). See RUNBOOK.md.

Scratch artifacts go under /tmp, never the repo. Screenshots normally go under /tmp/navigator-screenshots/.

The KIND dependency tier is the exception to "local stacks are task resources": it is a reusable dev fixture, so leave the cluster up between sessions. Everything else an agent spins up — rebuilt dev images, browser drivers, the host-side web process — is a per-task resource to stop at handoff. So before handing off a created or updated PR, stop web and task-created browser drivers, remove task-created standalone containers/images, and prune task-created Docker build cache — but do not down/kind delete the dependency cluster as routine cleanup, and do not prune Docker volumes unless the user approves the data loss. Full teardown is for a deliberate clean rebuild only.

GCP setup

navigator ops gcp setup provisions GCP by calling REST APIs directly from cli/src/devx/gcp/ with reqwest. There is no gcloud shell-out for the setup pipeline and no broad Google SDK wrapper. That is deliberate: raw REST gives the CLI a single dry-run intercept point and keeps endpoint behavior testable with wiremock.

When touching cli/src/devx/gcp/, keep four things correct:

Every step follows the same conventions:

When an endpoint drifts, update the module's wiremock test to match Google's current docs first, then update the implementation and run the dry-run command from oss-install.md.

Production deploy

Code reaches production through PRs and dated images:

  1. Merge through the normal PR flow in gitops.md.
  2. The release-tag workflow cuts a YY.M.D tag.
  3. The deploy workflow builds and publishes the service images to ghcr.io: navigator-web, navigator-workflows-service, and navigator-git.
  4. An operator rolls GKE onto the published tag.

navigator ops ship --tag YY.M.D is the self-contained, run-from-anywhere reconcile. It confirms every image is published at the tag, renders the CLI-embedded GKE manifest tree from the deployer's NAVIGATOR_* values and the tag itself, checks the boot invariants against that rendered tree while everything is still local, and only then kubectl diff -ks and applies it — unconditionally, so a manifest change merged to main reaches the cluster instead of silently rotting behind an image-only push. The render mechanics — the placeholder→env table, the by-name bail on a missing var, and --dry-run — are owned by gke-prod.md. navigator-web, workflows-service, and the optional navigator-git Deployment all land on the same YY.M.D tag (version skew is an avoidable production risk; when navigator-git is present, ship first verifies its app container is named git), and ship re-registers the worker with Restate.

The tag is a substitution token in the manifests, resolved at render time — the apply itself lands the real image. It is deliberately not a placeholder that a follow-up kubectl set image corrects: navigator-git (strategy: Recreate) and workflows-service (maxSurge: 0) both delete the running pod before the replacement is ready, so an apply carrying an unpullable tag takes those tiers down for the whole gap. One write, one ReplicaSet.

The preflight ordering is load-bearing, not incidental. web enforces its boot invariants at startup and crash-loops when a required key is missing, so ship diffs the requirements against the production Secret plus each web-binary Deployment's env before the reconcile touches the cluster. An unsatisfied requirement therefore aborts a ship that has changed nothing. It reads the env from the rendered manifests rather than the live Deployments for two reasons: the manifests are the state about to be applied (the live env is what the apply overwrites), and on a first-ever ship no Deployment exists to read.

ship never patches the Secret for you — it prints the exact kubectl patch naming every missing key, and stops. Fill in the values from Doppler prd and re-run. Keys the manifests supply as Deployment env (NAVIGATOR_OPA_URL, say) belong in the manifest tree, not the Secret; the preflight counts either as satisfying a requirement.

If the image tag is unchanged and only a Secret changed, --restart-only rollout-restarts the service deployments so pods re-read envFrom.

Run production cluster commands under the production secret context. Never paste real secret values into chat, docs, commits, or PR bodies.

Production database

Production is Cloud SQL for Postgres. Ad-hoc access goes through cloud-sql-proxy with IAM service-account impersonation, not password shortcuts.

Read-only SELECTs are allowed when the user asks for inspection. Before any INSERT, UPDATE, DELETE, or DDL:

The canonical seed is idempotent: it inserts missing rows and does not update existing production rows. A live data fix needs a guarded update, a migration, or an app seam.

Spend reporting

Report GCP spend from the BigQuery Cloud Billing export, not console guesses or rate-card math. Always show:

Discover the project from env and the billing table from BigQuery. Do not hard-code billing account generated table names into docs or code.

Observability

Every service binary emits through telemetry::init("navigator-<name>"). With no OTEL_EXPORTER_OTLP_ENDPOINT, logs stay human-readable on stdout. With the endpoint set, logs become JSON and traces/metrics export through OTLP.

The load-bearing rule is:

Identifiers and counts, never content.

Safe telemetry fields include ids, service names, outcomes, durations, status codes, and counts. Unsafe fields include client names, email addresses, answer bodies, document bodies, privileged facts, and full request or tool arguments. This rule applies in local Grafana LGTM, Cloud Logging, Cloud Trace, BigQuery, and any future sink.

Use navigator ops doctor, Cloud Logging/BigQuery, the Restate console, and the six-hourly Heartbeat email to debug missing periodic jobs or durable workflow failures. The architecture details live in observability.md and durable-workflows.md.

Website publication

Top-level files in docs/ are already published at /docs/:slug by web::docs. The site bakes the docs into the binary with include_str!, renders markdown under the Foundation brand, and rewrites top-level doc links to site routes. That gives every maintainer and LLM the same documentation surface.

Good next steps for the website: