GKE production deployment

The Neon Law Navigator production deployment runs on GKE Autopilot with every supporting service managed by Google or Restate. The daily operator workload is reviewing dependency PRs and glancing at dashboards; no node patching, no DB failover drills, no Helm chart babysitting.

Architecture at a glance

internet
   │
   ▼
┌───────────────────────────────────┐
│  Global External App LB (Gateway) │ ← Cloud Armor (DDoS + WAF)
│  ← Certificate Manager (TLS)      │ ← Identity-Aware Proxy (admin)
└───────────────────────────────────┘
   │
   ▼
┌───────────────────────────────────┐
│  GKE Autopilot                    │ ← Workload Identity for GCP
│  ┌─────────────────────────────┐  │
│  │ navigator-web (+OPA sidecar)│──┼──→  Cloud SQL for Postgres (PSC)
│  │ workflows-service           │──┼──→  Restate Cloud
│  │ navigator-git writer        │──┼──→  Project repo PVC
│  └─────────────────────────────┘  │──→  GCS (object storage)
└───────────────────────────────────┘   ↑
   ▲                                    │
   │   ┌────────────────────────────────┘
   │   │ Secrets via Secret Manager CSI driver
   │   ▼
   │ Secret Manager
   │
   └── Config Sync pulls from github.com/neonlaw/Navigator
       (path: examples/deploy/k8s/gke)

Deployment decisions are summarized here and in cloud-operations.md.

What lives where

Bootstrap

A fresh cluster requires gcloud commands that can't run from CI — they need a human under gcloud auth login. The navigator CLI prints the exact sequence:

cargo run --release -p cli -- gke-bootstrap

Output is the full set of gcloud services enable …, gcloud container clusters create-auto …, RootSync manifest, and post-bootstrap verification commands. Run them top-to-bottom; the RootSync at the end points at examples/deploy/k8s/gke on main.

After the cluster is up, fill in the placeholders in examples/deploy/k8s/gke/ (search for YOUR_PROJECT_ID, <navigator-domain>, <restate-tenant>, <cloud-sql-host>). Commit and push — Config Sync reconciles within ~15 seconds.

Data and access boundary

The public edge reaches three service Deployments:

• navigator-web serves the portal, AIDA/API routes, webhooks, and client-facing matter views.
• workflows-service hosts the Restate durable worker.
• navigator-git is optional but enabled in the reference GKE overlay; it is the single mounted writer for Project Git repos and Git LFS.

The storage buckets sit behind those services. YOUR_PROJECT_ID-assets is the only public bucket. YOUR_PROJECT_ID-documents is private and holds content-addressed client documents and Git LFS objects through cloud::StorageService; clients never receive bucket IAM, object keys, or raw GCS URLs. A client sees a file only when web resolves their session to a persons row and finds a matching person_project_roles row for the Project. Staff/admin users can use the Git clone/PAT workflow for editor-style matter work; clients use the portal Documents/Engagements/Invoices surface. YOUR_PROJECT_ID-logs is the Nearline audit-log archive.

The navigator-web backend attaches a GKE BackendConfig that injects X-Navigator-Client-Region:{client_region} before requests reach the pod. The web app uses that bounded, edge-derived geography bucket for visitor analytics. Analytics do not store raw IP addresses and do not trust the left-most X-Forwarded-For value as a geography source.

Daily deploy flow

CI/CD is exactly three workflows (see gitops.md): a lean PR flow (ci.yml), a nightly cron flow that cuts a calendar release tag (release-tag.yml), and a tag flow that integration-tests and publishes the images (deploy.yml).

PR merged to main
  └─→ .github/workflows/ci.yml runs fmt + clippy + cargo test --workspace
      (no images built — the PR flow is lean by design)

Daily 01:11 UTC
  └─→ .github/workflows/release-tag.yml cuts tag YY.M.D (e.g. 26.6.18)
      and pushes it with secrets.RELEASE_PAT
            └─→ the tag push triggers .github/workflows/deploy.yml
                  ├─ KIND integration suite (e2e + interop + browser)
                  ├─ build + push service images to ghcr.io tagged YY.M.D + latest
                  └─ post a "ready to deploy" hand-off to the engineering Slack channel
                        Images are on the shelf, tagged by date.

Manual same-day publish
  └─→ gh workflow run deploy.yml
        └─→ .github/workflows/deploy.yml derives YY.M.D.H in UTC
              (for example, 26.6.25.14 for June 25, 2026 at 14:00 UTC)

The images are published, not rolled out — promoting a dated image to the GKE cluster (the Config Sync reconcile, or an operator-driven ship) is a separate, deliberate step, not part of the nightly tag flow.

navigator ops ship --tag YY.M.D pins navigator-web, workflows-service, and the optional navigator-git Deployment to the published tag. Installs that satisfy the git boot invariant by mounting NAVIGATOR_GIT_REPO_ROOT may omit navigator-git; ship detects that shape and prints a skip notice. When the git-serving Deployment exists, ship updates its ghcr.io/<owner>/navigator-git:YY.M.D image and waits for the Recreate rollout just like the other service deployments, so the single-writer tier does not drift from the web binary it shares. A present navigator-git Deployment must contain the app container named git; ship preflights that shape and stops before changing any image when an older manifest still names the container differently.

The published packages are public on ghcr.io, so the GKE nodes pull them anonymously — there is no imagePullSecret and no registry credential to rotate. Old dated tags are pruned after 14 days by the maintenance workflow (cleanup.yml); a fork that defers a roll past two weeks should pin and roll a tag while it is still on the shelf.

Friday-Sunday is deliberately skipped — see the comment in deploy.yml.

Manual rollout: gh workflow run deploy.yml triggers the same sequence on demand and publishes under the UTC-hour-suffixed tag.

Manifest delivery

examples/deploy/k8s/gke/ is white-labeled by design — it is the rebrandable reference (the navigator ops rebrand seam), so it ships placeholders like YOUR_PROJECT_ID, www.your-domain.example, and YOUR_OAUTH_CLIENT_ID_*. Running a plain kubectl apply -k of it against a real cluster would rewrite the ManagedCertificate domains, the OAuth client ids, and every domain-derived env back to those placeholders. The substitutions the base needs are pure values — the GCP project (buckets, Cloud SQL instance, GSA), the primary domain (ManagedCertificate + Ingress hosts, OAUTH_REDIRECT_URI, NAV_BASE_URL, GOOGLE_OAUTH_REQUIRED_HD), and the two public OAuth client IDs — not structure.

navigator ops ship supplies those values itself. The whole manifest tree (examples/deploy/k8s/gke + the shared k8s/base it references) is embedded in the CLI with include_dir!; every ship renders it — the placeholders substituted from NAVIGATOR_* env (Doppler prd) — into a throwaway temp dir, then kubectl diff -k (surface drift) and kubectl apply -k (reconcile). The apply is unconditional: any structural change on main (a renamed container, a new sidecar, a volume, an env-list edit) reaches the cluster on the next ship rather than silently rotting. There is no persistent overlay folder and no image-only fall-through — the CLI generates from env what a deployer used to hand-keep. The render is from the embedded tree, so ship runs from any directory, even a Homebrew install with no workspace checkout.

The substitution values live only in Doppler prd (and Secret Manager for the secret half) — never in this repo:

A missing or blank required var bails by name before anything is written — a half-substituted manifest never reaches the cluster. Firm-specific values that are secrets or operator toggles (SendGrid, DocuSign, the inbound-email host, DKIM enforcement) stay in the navigator-web-secrets K8s Secret and arrive via envFrom. The base's inline-env $patch: replace does not touch that Secret reference, so a full apply preserves them. The git-image-tag the old overlay carried is now the ship --tag itself: the image roll pins navigator-git to the same YY.M.D as web.

A healthy cluster's kubectl diff -k against the freshly rendered tree is a near no-op — that is the acceptance bar for source == cluster. --dry-run renders and diffs but never applies, so "see what will change" needs no folder:

# Preview the reconcile without applying (renders, diffs, drops the temp dir):
doppler run --project navigator --config prd -- navigator ops ship --tag <YY.M.D> --dry-run

The reconcile runs against prod (propose-only — run it yourself under Doppler prd; never let an agent ship to prod, and beware the shared kubeconfig defaulting to the prod GKE context):

doppler run --project navigator --config prd -- navigator ops ship --tag <YY.M.D>

What lives outside this repo

These are operator-managed resources you maintain via gcloud / console, not via this repo's manifests:

1. Cloud SQL instance — provisioned once via gcloud sql instances create. The connection URL goes into Secret Manager as navigator-database-url.
2. GCS buckets — YOUR_PROJECT_ID-assets, YOUR_PROJECT_ID-documents, and YOUR_PROJECT_ID-logs. The Workload Identity service account for navigator-web needs roles/storage.objectAdmin on the private documents bucket; only the assets bucket receives a public binding.
3. Identity Platform tenant — configured via the console. OAuth client secret goes into Secret Manager as navigator-oauth-client-secret.
4. Restate Cloud tenant — register at https://cloud.restate.dev. The tenant URL and bearer token go into Secret Manager as navigator-restate-broker-url (consumed as RESTATE_BROKER_URL) and navigator-restate-auth-token (consumed as RESTATE_AUTH_TOKEN).
5. DNS A record for <navigator-domain> → the static IP reserved as navigator-gateway-ip.
6. Cloud Logging → GCS sink — a log router sink that archives web container logs to gs://YOUR_PROJECT_ID-logs for long-horizon audit. Provisioned via gcloud (see "Long-term log archive" below), not Config Sync.

Everything else flows through Config Sync.

Long-term log archive

GKE forwards every container's stdout to Cloud Logging automatically, but the _Default bucket retains only 30 days — fine for live triage, not enough for multi-year audit. A log router sink copies Navigator service logs into the NEARLINE bucket gs://YOUR_PROJECT_ID-logs (the same bucket navigator ops gcp setup already creates), where the 365-day → Coldline lifecycle keeps them cheap and durable for years.

The original design routed these via a Config Connector LoggingLogSink CR. That path is shelved: Config Connector does not reliably reconcile on this cluster's GKE version, so the sink is provisioned directly with gcloud and lives outside the repo's manifests on purpose — there is nothing under examples/deploy/k8s/gke/ to keep it in sync with, and putting it there would falsely imply Config Sync owns it.

Substitute YOUR_PROJECT_ID (the deployer's NAVIGATOR_GCP_PROJECT_ID) before running:

# Create the sink: route Navigator service logs in the navigator namespace to GCS.
gcloud logging sinks create navigator-web-to-gcs \
  storage.googleapis.com/YOUR_PROJECT_ID-logs \
  --log-filter='resource.type="k8s_container"
                resource.labels.namespace_name="navigator"
                AND (resource.labels.container_name="web"
                     OR resource.labels.container_name="worker"
                     OR resource.labels.container_name="git")' \
  --project=YOUR_PROJECT_ID

# Grant the sink's auto-created writer identity permission to write the bucket.
WRITER=$(gcloud logging sinks describe navigator-web-to-gcs \
  --project=YOUR_PROJECT_ID --format='value(writerIdentity)')
gsutil iam ch "${WRITER}:roles/storage.objectCreator" gs://YOUR_PROJECT_ID-logs

Verify the sink is writing (objects appear under logs/... prefixes within ~1 hour of the next matching log line):

gcloud logging sinks describe navigator-web-to-gcs --project=YOUR_PROJECT_ID
gsutil ls "gs://YOUR_PROJECT_ID-logs/k8s_container/**" | head

This is operator-managed state, like the Cloud SQL instance and Identity Platform tenant above — it is not rebuilt by kubectl apply -k. If you later get Config Connector reconciling, the LoggingLogSink CR can replace these commands; until then this section is the source of truth for the sink's existence.

Verifying a deploy

# Pod rollout (`navigator-git` only when the git-serving tier is enabled)
kubectl --namespace navigator rollout status deployment/navigator-web
kubectl --namespace navigator rollout status deployment/workflows-service
kubectl --namespace navigator rollout status deployment/navigator-git

# Image actually in use
kubectl --namespace navigator get deployment/navigator-web \
    -o jsonpath='{.spec.template.spec.containers[?(@.name=="web")].image}'
kubectl --namespace navigator get deployment/navigator-git \
    -o jsonpath='{.spec.template.spec.containers[?(@.name=="git")].image}'

# Repo-PVC backup schedule (git-serving tier) — the CronJob and its snapshots
kubectl --namespace navigator get cronjob/navigator-git-repos-backup
kubectl --namespace navigator get volumesnapshot \
    -l app=navigator-git-repos-backup --sort-by=.metadata.creationTimestamp

Trust boundary

Pull-based deploy = no external system holds cluster credentials. Specifically:

• GHCR push token is scoped to package writes only. GitHub Actions repo write token can commit to main but never reaches kube-apiserver. Config Sync inside the cluster uses an in-cluster ServiceAccount to pull the public repo — no external token at all. Workload Identity binds each Kubernetes ServiceAccount to a GCP service account, so pods talk to GCP without JSON keys.

Result: nothing outside Google's perimeter has a credential that can touch the cluster control plane or the data plane.

Restore from backup

Backup for GKE snapshots run daily at 05:00 UTC; retention is 30 days. Restore is a single CLI:

gcloud container backup-restore backups list \
    --location=us-west4 --backup-plan=navigator-daily

gcloud container backup-restore restores create my-restore \
    --location=us-west4 \
    --restore-plan=<plan> \
    --backup=<backup-id>

Restoring the Project git repos. The navigator-git-repos PVC (the single-writer git store) is captured either by this namespace-wide Backup for GKE plan or by the standalone VolumeSnapshot CronJob in git/repo-backup.yaml — see Manifest delivery and git-project-repos.md for which one an install runs. Restoring from a standalone snapshot replaces the RWO claim in place: quiesce the single writer so the disk detaches, recreate the claim with the snapshot as its dataSource, then bring the writer back. Run every step under the prod context (mind the shared-kubeconfig default; pass --context explicitly):

set -eu

# 1. Pick the snapshot to restore from, then verify that exact object:
kubectl --context <prod-ctx> -n navigator get volumesnapshot -l app=navigator-git-repos-backup \
    --sort-by=.metadata.creationTimestamp
SNAP=navigator-git-repos-YYYYMMDD-HHMMSS
test -n "$SNAP"
kubectl --context <prod-ctx> -n navigator get volumesnapshot "$SNAP"

# 2. Quiesce the single writer so the RWO disk detaches:
kubectl --context <prod-ctx> -n navigator scale deployment/navigator-git --replicas=0
kubectl --context <prod-ctx> -n navigator wait --for=delete pod -l app=navigator-git --timeout=120s

# 3. Replace the claim from the snapshot. This deletes the current PD — that
#    IS the restore; take a fresh snapshot first if the live disk is unsure.
kubectl --context <prod-ctx> -n navigator delete pvc navigator-git-repos
kubectl --context <prod-ctx> -n navigator apply -f - <<YAML
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: navigator-git-repos
  namespace: navigator
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: premium-rwo
  resources:
    requests:
      storage: 50Gi
  dataSource:
    name: $SNAP            # the VolumeSnapshot chosen in step 1
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
YAML

# 4. Bring the writer back onto the restored disk:
kubectl --context <prod-ctx> -n navigator scale deployment/navigator-git --replicas=1
kubectl --context <prod-ctx> -n navigator rollout status deployment/navigator-git

Practice both restore paths quarterly against a scratch cluster. Untested backups are theatre.