Operating Neon Law Navigator

set -a; eval "$(grep ' = "' deployments/neon-law-stg/config.toml | sed 's/ = /=/')"; set +a
navigator ops gcp setup --dry-run

navigator ops gcp hub setup \
  --project-id ghcr --region us-west4 --dry-run
navigator ops gcp hub setup \
  --project-id ghcr --region us-west4

set -a; eval "$(grep ' = "' deployments/neon-law-stg/config.toml | sed 's/ = /=/')"; set +a
navigator ops gcp setup

Setup provisions no database and generates no credential, so there is nothing printed to record. The store's own credentials go into deployments/<name>/secrets.enc.yaml (sops set — the plaintext never touches disk), then navigator ops secrets apply --deployment <name>; rotation happens at the provider first, per docs/deployment-secrets.md.

Resolve every reserved address and record the public value as NAVIGATOR_GATEWAY_IP in the matching config.toml — an IP is a public coordinate, so it is edited and committed like any other:

for name in $(ls deployments); do
  set -a; eval "$(grep ' = "' "deployments/${name}/config.toml" | sed 's/ = /=/')"; set +a
  echo "${name}:"
  gcloud compute addresses describe "$NAVIGATOR_GATEWAY_IP_NAME" \
    --global --project "$NAVIGATOR_GCP_PROJECT_ID" --format 'value(address)'
done

Populate the matching local Kubernetes context before shipping a row:

set -a; eval "$(grep ' = "' deployments/neon-law-stg/config.toml | sed 's/ = /=/')"; set +a
gcloud container clusters get-credentials \
  "$NAVIGATOR_GKE_CLUSTER_NAME" --region "$NAVIGATOR_GCP_LOCATION" \
  --project "$NAVIGATOR_GCP_PROJECT_ID"

The resulting context name must equal that config's NAVIGATOR_GKE_CONTEXT. Repeat once for each deployment you are operating.

The canonical matrix is the compact reference for every row.

A real run first enables twenty-three Google Cloud APIs in bounded Service Usage batchEnable calls: compute, servicenetworking, storage, iam, iamcredentials, sts, cloudresourcemanager, artifactregistry, container, gkehub, gkebackup, anthosconfigmanagement, logging, monitoring, cloudtrace, secretmanager, certificatemanager, identitytoolkit, speech, drive, admin, cloudkms, and aiplatform.

With storage in place, the provisioner creates the one private Artifact Registry repository that every navigator container image lives in, and the identities that push to and pull from it:

• A Docker-format repository (default navigator) at your-region-docker.pkg.dev/your-project-id/navigator.
• A keep-the-last-10-versions cleanup policy — a KEEP rule retaining the last 10 versions of each image plus a DELETE rule for everything else. Retention is a count rather than an age on purpose: an age-based rule is only safe while releases outrun it, and releases are tag-driven, so a quiet fortnight under the old 7-day rule would have let the registry delete the versions production was running. A count cannot expire. Keep policies take precedence over delete policies, which is what makes the pair mean "keep ten, delete the rest" — the delete half matches every version, so it is never applied alone.
• A CI push identity (navigator-ci-pusher service account) with a repo-scoped roles/artifactregistry.writer binding, plus a GitHub Workload Identity federation pool and provider so CI authenticates keyless — no downloaded service-account key — pinned to this one repository and to the refs allowed to publish.
• A repo-scoped roles/artifactregistry.reader binding for the GKE Autopilot node identity, so the cluster can pull.

Two values decide whether that federation works at all, and both are easy to get wrong in a way nothing reports:

issuerUri           https://token.actions.githubusercontent.com
                    (on your own GHE tenant: https://token.actions.<your-tenant>.ghe.com)
attributeCondition  assertion.repository == '<owner>/<repo>'
                        && (assertion.ref == 'refs/heads/main' || assertion.ref.startsWith('refs/tags/'))

Navigator itself runs on github.com, so the public issuer is the one this workshop's own deployment uses. The tenant issuer above is for readers who do not.

The cluster is the one part driven through gcloud rather than REST. In order, the provisioner reserves a static IP, creates the GKE Autopilot cluster, and registers it as a Fleet member:

cargo run -p cli -- ops gcp setup --project-id your-project-id --region us-west4

OAUTH_ISSUER_URL=...        # the provider's issuer; discovery hangs off /.well-known/openid-configuration
OAUTH_CLIENT_ID=...
OAUTH_CLIENT_SECRET=...
OAUTH_REDIRECT_URI=https://www.your-domain.example/auth/callback

Navigator speaks that flow against the provider (/auth/login → /auth/callback) and discovers every endpoint from <issuer>/.well-known/openid-configuration, so no provider URL is hard-coded. Worked examples for Rauthy, Google, Auth0, and Okta live in .env.example. The provider asserts only who you are (a stable sub and an email); your persons row owns what you can do (the single role), so granting or revoking access is one SQL statement — see docs/oidc.md for the full model.

For this workshop, that row should make you the owner. Navigator has five stored authorization roles, plus the anonymous public visitor:

• owner — the system owner, highest in authority, inheriting every Admin and Lawyer capability. Only Owner may govern another Owner identity.
• admin — a licensed lawyer with installation-wide administration authority. Admin cannot manage an Owner.
• lawyer — a licensed lawyer working assigned matters and supervising any Clerk capability the application grants.
• clerk — a supervised non-lawyer worker. Clerk's /clerk surface is a read-only list of firm-assigned Projects whose disclosed is_lawyer_dri row is a lawyer. Clerk has no legal-advice, approval, Git, MCP, or /app/lawyer authority by inheritance; upload and preparation work still need their own narrow, supervised routes.
• client — represented people using the portal for their own matters.

Owner is the deployer's role because this class touches billing, secrets, OIDC, release state, and every Project. The role still lives in the database, not the IdP token; your OIDC provider proves identity, then Navigator reads the persons.role value to decide the tier.

One environment variable answers which person is the protected bootstrap Owner:

NAVIGATOR_BOOTSTRAP_OWNER_EMAIL=owner@example.com

Set NAVIGATOR_BOOTSTRAP_OWNER_EMAIL in the deployment config.toml; ops ship renders it into the web pod. Do not copy the deployed value into Git. An unset, empty, or whitespace-only value disables bootstrap-Owner creation and refuses the ship. On a fresh installation, the first successful OIDC login with the configured email JIT-creates its persons row as owner; later sign-ins restore that role if the database has drifted. Its entire Person record is immutable in Navigator so an administrator cannot rename, demote, or delete the installation's recovery identity by accident.

After signing in as Owner or Admin, open /app/admin/people to manage the directory and change another Person's system-wide role among owner, admin, lawyer, clerk, and client. Owner appears first because it owns the deployed system. Only an Owner can assign or modify Owner. Admin can manage Admin and every lower tier. The bootstrap-Owner row remains read-only, and the command boundary rejects a hand-written update or delete as well. The Lawyer workbench does not grant role-management power. These values are persons.role; Project assignments such as attorney, paralegal, client, and co-counsel are Participation records and do not create another authorization tier.

We recommend Google. Verifying that a person is who they claim is real work — risk signals, step-up challenges, and hardware-key (passkey / security-key) verification — and Google invests far more in it than we could. A Google-backed sign-in is stronger than any password we would host ourselves. That is exactly the path NeonLaw's own prod takes: Sign in with Google, wired by the four env vars above with OAUTH_ISSUER_URL=https://accounts.google.com — the same standard redirect flow, no password anywhere. (Google-hosted email/password is a separate, opt-in door: set NAVIGATOR_IDENTITY_PLATFORM_API_KEY and /auth/login also renders a password form whose POST /auth/password checks the credential against GCP Identity Platform — Google still owns the password and lockout, never us. Reset and email confirmation need the admin door too: set NAVIGATOR_GCP_PROJECT_ID, run web where the GCE metadata server can mint a service-account bearer token, and grant that service account roles/identitytoolkit.admin. Leave the key unset and sign-in stays a pure Google redirect. Config lives in .env.example, not the redirect swap above.)

No-Google path. "Sign in with Google" cannot be the only front door of a public legal-services portal — the person a clinic serves may have no Google account. Any standards-compliant OIDC provider — discovery, Authorization Code + PKCE, RS256-signed ID tokens with RSA keys in JWKS — that hosts its own email/password login works with the env-swap above and zero Neon Law Navigator code changes. Rauthy — the same open-source IdP the local KIND loop already runs — serves email/password, self-registration, reset, and verification from its own pages, in your cluster with no per-user fee. Auth0 / Okta are hosted SaaS equivalents: same four env vars, same redirect flow.

All three organizations and private Apps were created on GitHub Free with those exact slugs. Each App is installed only on the organization in its row, selects all current and future repositories, and grants repository Administration and Contents read/write. Webhook delivery is disabled. Each deployment's config.toml contains that row's App ID and installation ID, and its secrets.enc.yaml the distinct private key. The complete value-by-value worksheet lives in docs/provider-environment-parity.md.

Updating a provider credential

Rotation is two-sided and ordered: at the provider first, in the repository second. Re-encrypting alone revokes nothing — anyone holding repository history and the KMS key can still read every prior ciphertext, so the old value stays valid until the provider stops honouring it.

Two of these credentials cannot be re-read once created, which makes "rotate" mean replace, not look up:

• Google OAuth client secret. Google removed retrieval entirely; the console shows only a masked suffix. Replace it with Add secret, capture the value from the one-time dialog, then delete the old secret.
• GitHub App private key. GitHub hands over the .pem once, at generation. Replace it by generating a new key, then deleting the superseded one.

Capture the value at the moment it is shown. A secret minted without capturing it is not recoverable — it is only deletable, and it leaves a second live credential behind until you remove it.

Write the new value straight into the deployment's encrypted file. sops reads its key from the .sops.yaml creation rule for that path, encrypts per value on save, and never writes plaintext to disk:

sops deployments/neon-law-stg/secrets.enc.yaml

A PEM needs a YAML block scalar so its newlines survive:

NAVIGATOR_GITHUB_APP_PRIVATE_KEY: |
  -----BEGIN RSA PRIVATE KEY-----
  ...
  -----END RSA PRIVATE KEY-----

Then push it to that deployment's own Secret Manager. The CSI driver projects the new version into the pods:

navigator ops secrets apply --deployment <row> --deployments-dir . --dry-run
navigator ops secrets apply --deployment <row> --deployments-dir .

The dry run prints the target project and object names without decrypting anything, so it needs no KMS permission — run it first to confirm you are aimed at the deployment you meant.

Public coordinates are not key material and do not belong in this file. A GitHub App ID, an installation ID, and an OAuth client ID are greppable, diffable, reviewable values: they live in config.toml. Only the App private key and the OAuth client secret cross into secrets.enc.yaml.

Google OAuth: one client per deployment, not one shared secret

Create one OAuth client per deployment in its row's GCP project: a Web application client for browser sign-in, with exactly the callback in the table.

The staging client lives in the neon-law-stg GCP project's Google Auth Platform consent configuration. The three production clients live in their matching projects. Store the browser ID in NAVIGATOR_OAUTH_CLIENT_ID_BROWSER and its secret in OAUTH_CLIENT_SECRET. Google matches a browser redirect exactly, so do not put several sites' callbacks on one client. An Internal audience requires the project to belong to the matching Workspace organization; an External audience needs test users or the applicable verification and domain-ownership work.

The staging browser client exists with the exact name and callback in this section. Its consent configuration is External/Testing, and the authenticated operator is its initial test user. Its deployment config carries that browser ID and secret, and ops ship renders a browser-only allowlist.

These clients are configured in Google Auth Platform → Clients. They are general OAuth clients, not IAP or Workforce Identity Federation clients. Google does not permit creating or modifying them programmatically, so neither ordinary gcloud services enable/service-account commands nor gcloud iam oauth-clients replace this console step.

Create a browser OAuth client and save it safely

Do this once for each row that does not already have its browser client. The Google Auth Platform page shown below is safe to include in an operating record because it contains the client name only—never capture the following creation dialog, which reveals the secret once.

1. In the Google Cloud console, switch to the GCP project in the deployment matrix for the target deployment. This is neon-law for neon-law-prod, neon-law-prod for neon-law-prod, and neon-law-stg for neon-law-stg. Do not create a client's credentials in a different project and copy them across.

2. Open Google Auth Platform → Clients, then select Create client. If Google first opens the Auth Platform setup screen, complete the organization-approved branding, support contact, audience, and contact-email setup before continuing. That one-time configuration is project-scoped; it is not a replacement for the client below.

3. Choose OAuth client ID, set application type to Web application, and name it navigator-<deployment>-browser. For example, Neon Law production is navigator-neon-law-prod-browser.

4. Leave Authorized JavaScript origins empty. Under Authorized redirect URIs, add exactly the single callback for that config from the preceding table. Do not add a wildcard, a second deployment's callback, or a guessed local URL.

5. Select Create. Google displays a dialog with two values: Client ID and Client secret. Copy both before closing it. The secret cannot be recovered from this dialog later; create a replacement client if it is lost.

6. In that deployment's deployments/<name>/ tree, save the values under these exact names:

   Google creation-dialog value	Where it lives
   Client ID	NAVIGATOR_OAUTH_CLIENT_ID_BROWSER in config.toml
   Client secret	OAUTH_CLIENT_SECRET in secrets.enc.yaml (sops set)

   Save them only in their matching deployment. The Client ID is public metadata but is still deployment-specific; the client secret is a credential and must never go in a terminal transcript, Slack, this repository in plaintext, or a screenshot.

7. Reopen Google Auth Platform → Clients and verify the client name and its single redirect URI. Then run the normal secret synchronization and ship dry run. The deployment operator preflight is deliberately non-interactive: it refuses a row whose NAVIGATOR_OAUTH_CLIENT_ID_BROWSER or OAUTH_CLIENT_SECRET is absent and prints that row's project, client name, and exact callback. It verifies presence, not that a non-empty value is a usable Google credential.

Do not recycle a browser client, callback, or secret across rows.

Delete the obsolete navigator-neon-law-stg-browser, navigator-neon-law-stg-gemini, navigator-neon-staging-browser, and navigator-neon-staging-gemini registrations from the neon-law-stg project. Their retired configs are not proof that the Google registrations are gone, and no surviving deployment may reuse one of those client IDs, callbacks, or secrets.

GitHub: four organizations and four private Apps

For each organization, a GitHub owner must:

1. confirm the existing GitHub Free organization, contact email, controlling business, and current sole owner; add no bootstrap invitation, then handle two-factor enforcement and any approved recovery owner in a separate access review;
2. confirm the private, organization-owned GitHub App in the table is installed only in that organization;
3. grant repository Contents and Issues read/write;
4. put that row's organization, App ID, and private key in NAVIGATOR_GITHUB_ORG, NAVIGATOR_GITHUB_APP_ID, and NAVIGATOR_GITHUB_APP_PRIVATE_KEY.

The Apps own no Project repositories: Navigator provisions none. What is left is the neon-law/navigator webhook and the DevX Restate services, which are the neon-law-stg singleton.

Provider signups that require a human account owner

• DocuSign: create Developer/demo attachments for staging. Production needs production eSignature accounts and Go-Live-approved integrations. The deployment's secrets.enc.yaml receives the account/base/OAuth IDs, JWT app/user/private key, signer, Connect HMAC, and path secret. Prove a completed demo or deliberate production envelope and its verified Connect delivery.
• Twilio SendGrid: create account or subuser boundaries with four separately revocable keys and webhook configurations, and authenticate each sender domain. The deployment's secrets.enc.yaml receives the mail API key, From address, inbound and event secrets, and signed-event public key. Prove outbound delivery, inbound parse, and a signed event callback.
• Google Workspace Drive: create three Shared Drives and service accounts. A Super Admin grants Drive domain-wide delegation to each service-account OAuth client. The deployment's tree receives the selected Drive ID and delegated user (config.toml) and service-account JSON (secrets.enc.yaml). Prove a synthetic file can be created, read, and archived only in the matching Drive.
• Restate Cloud: arrange an account and plan supporting at least three environments; the free tier is insufficient. The deployment's secrets.enc.yaml receives that row's broker, ingress, admin URL, and API key. Prove the worker registers and completes a durable workflow in that environment.
• Production contracts: an API-capable DocuSign plan and a SendGrid plan supporting the required webhook count may be paid subscriptions. A production deployment must not contain a demo DocuSign host, demo account, or staging sender.

For SendGrid, create a restricted mail-send API key rather than a billing-capable key and enable signed webhook verification. For Drive, authorize only https://www.googleapis.com/auth/drive. For Restate, never point two rows at the same environment in the target state: its journal is state, not a stateless endpoint.

Keep provider attachments deployment-local

Do not copy a production provider bundle from staging or another brand. Each deployment's deployments/<name>/ tree owns its GitHub App, Restate journal, DocuSign attachment, SendGrid credentials and webhooks, OAuth clients, Drive root, database, session key, and application-signing keys. Run the names-only parity gate (it runs inside the workspace test suite on every pull request) and the provider smoke tests per deployment, then let a --dry-run ship enforce the actual boot-key contract.

ops ship --dry-run detects a missing or empty boot key before it applies a workload, but it cannot prove that a non-empty provider credential is valid. Treat a placeholder such as SG.test only as a bounded diagnostic that proves the next preflight branch; replace it with the deployment's restricted SendGrid key and run the outbound, inbound, and signed-event smoke tests before a production ship. A placeholder can let a pod boot while the first real email fails, which is useful evidence during setup but never production readiness.

When a dry-run prints a kubectl patch secret remedy, read it as the exact missing-key diagnosis—not as the durable repair. Add the value only to the named deployment's secrets.enc.yaml (or config.toml for a coordinate), then run navigator ops secrets apply --deployment <name> and rerun the same ops ship --dry-run. A hand-patched Kubernetes Secret is drift the CSI projection will replace. Work one reported requirement at a time: the guard stops before the workload apply, so the next dry-run is the authoritative check for the next missing boot dependency.

Before the first ship, write each deployment's Secret Manager objects from its tree:

navigator ops secrets apply --deployment <row> --deployments-dir . --dry-run
navigator ops secrets apply --deployment <row> --deployments-dir .

Repeat per deployments/ directory. The dry run reads key names only — no KMS call, no decrypted value — and fails closed listing any object the SecretProviderClass projects that the tree does not supply. It never writes DNSimple, gcloud, or operator-session values into Secret Manager; the tree cannot even express them.

Presence is only the first audit, and it runs in CI: the parity gate in cli/src/devx/deployments.rs checks every deployment's key names against store::deployment::WEB_REQUIREMENTS on every PR. Then run navigator ops ship --deployment <name> --dry-run and perform the provider smoke tests above. Never paste a private key, token, service-account JSON, or decrypted value into Slack or workshop notes — see docs/deployment-secrets.md.

• staging
• Neon production
• Neon Law production

Set exactly one website to one published version:

Choose its name from the deployments/ tree (host is that config's NAVIGATOR_PUBLIC_HOST), set tag, then run:

name=neon-law-stg tag=YY.M.D
host=$(sed -n 's/^NAVIGATOR_PUBLIC_HOST = "\(.*\)"$/\1/p' "deployments/${name}/config.toml")
gcloud auth login --force --update-adc

set -a; eval "$(grep ' = "' "deployments/${name}/config.toml" | sed 's/ = /=/')"; set +a
gcloud container clusters get-credentials "$NAVIGATOR_GKE_CLUSTER_NAME" \
  --region "$NAVIGATOR_GCP_LOCATION" \
  --project "$NAVIGATOR_GCP_PROJECT_ID"

navigator ops secrets apply --deployment "$name"
navigator ops observability --deployment "$name"
navigator ops ship --deployment "$name" --tag "$tag" --dry-run
navigator ops ship --deployment "$name" --tag "$tag"

curl --fail --show-error --silent "https://${host}/readyz" >/dev/null
curl --fail --show-error --silent "https://${host}/version" |
  jq --exit-status --arg tag "$tag" '.release == $tag'

Change only name and tag; the host follows from the config. The current tree maps:

• neon-law-stg → staging.neonlaw.com
• neon-law-prod → www.neonlaw.com
• neon-law-prod → www.neonlaw.com

All three hosts resolve to their own deployment's gateway IP, so all three answer /readyz and /version directly.

export DNS_ACCT=174981
export DNSIMPLE_TOKEN="$DNS_SIMPLE"

dnsimple records list neonlaw.com --account "$DNS_ACCT" --json |
  jq --exit-status '
    .data as $records |
      ([$records[] | select(.name == "" and .type == "URL")] == [{
        id: 80303423,
        zone_id: "neonlaw.com",
        type: "URL",
        name: "",
        content: "https://www.neonlaw.com",
        ttl: 300,
        regions: ["global"],
        created_at: "2026-07-23T22:08:37Z",
        updated_at: "2026-07-23T22:08:37Z"
      }]) and
      ([$records[] |
        select(
          .name == "staging" or
          .name == "workflows-staging" or
          .name == "neon" or
          .name == "workflows-neon-law-prod" or
          .name == "www" or
          .name == "workflows"
        )
      ] == [{
        id: 80303569,
        zone_id: "neonlaw.com",
        type: "URL",
        name: "www",
        content: "https://www.neonlaw.com",
        ttl: 300,
        regions: ["global"],
        created_at: "2026-07-23T22:14:28Z",
        updated_at: "2026-07-23T22:14:28Z"
      }])
  '

dnsimple records delete neonlaw.com 80303569 --account "$DNS_ACCT" --yes

dnsimple records create neonlaw.com --account "$DNS_ACCT" --type A --name staging \
  --content 34.160.169.219 --ttl 300
dnsimple records create neonlaw.com --account "$DNS_ACCT" --type A --name workflows-staging \
  --content 34.160.169.219 --ttl 300
dnsimple records create neonlaw.com --account "$DNS_ACCT" --type A --name neon \
  --content 34.149.196.255 --ttl 300
dnsimple records create neonlaw.com --account "$DNS_ACCT" --type A --name workflows-neon-law-prod \
  --content 34.149.196.255 --ttl 300
dnsimple records create neonlaw.com --account "$DNS_ACCT" --type A --name www \
  --content 8.233.220.29 --ttl 300
dnsimple records create neonlaw.com --account "$DNS_ACCT" --type A --name workflows \
  --content 8.233.220.29 --ttl 300

Run the mail record groups once for the neonlaw.com zone—not once per deployment—using --google-workspace, --sendgrid, DKIM/link-branding targets, SPF includes, and DMARC settings. The command is additive and never deletes unrelated records. The full record ownership and Google Workspace forwarding recipe is in docs/dns.md.

That was a genuine defect rather than a configuration mistake, and the fix was to render the object list per deployment so the class references exactly what the deployment writes. It is described in docs/deployment-secrets.md. If your own deployment declines an integration the manifest names, this is the machinery that lets it.

Secret Manager was empty. With the object list correct, one command filled it:

navigator ops secrets apply --deployment <row> --deployments-dir .

Twenty-five objects written, twelve reported as skipped with their reasons. Read that skipped line — it is also the list of what this deployment's mount will not ask for.

A cold cluster has no namespace, and no Secret to preflight against. This is the one to plan around. ops ship confirms the deployment's Secret satisfies the running binary's boot invariants before it reconciles anything, which is the right order for every ship after the first — an unsatisfied requirement aborts a ship that has touched nothing. On a first ship it is a standoff: the namespace is created by the apply in step 5, and the Secret is projected by the CSI driver only while a pod mounts the volume, so neither exists when step 4 goes looking. The failure reads:

Error: kubectl get secret neon-law-prod-web-secrets failed:
Error from server (NotFound): namespaces "neon-law-prod" not found

We broke the standoff by hand: created the namespace, then seeded a plain neon-law-prod-web-secrets Secret from the Secret Manager objects ops secrets apply had just written, so the preflight had something real to read. The projected Secret takes the same name, so once the pods are up and the driver owns it, the seeded one is retired exactly as "Retire the plain Secret" describes.

Do not read that paragraph as a runbook. Doing it by hand is off the invariant this whole workshop rests on — the navigator CLI orchestrates every machine-bound flow, and reaching around it with ad hoc commands is precisely what leaves the next operator without a path. The bootstrap belongs in the CLI, and the shape it should take is a preflight that falls back to the key set the rendered SecretProviderClass will project when no live Secret exists yet: those objects are already proven to resolve to an ENABLED version before the reconcile, so it checks the same property from the authoritative source instead of from a cluster that has nothing in it. Until that lands, a first install into a cold cluster needs a human, and you should expect it.

A cold Autopilot cluster has no nodes, and that stops the reconcile. With the Secret readable, ops ship rendered and diffed, then failed on an admission webhook:

Error from server (InternalError): failed calling webhook
"validate.rules.gmp-operator.gke-gmp-system.monitoring.googleapis.com":
no endpoints available for service "gmp-operator"

The chain is circular and worth recognising on sight. Autopilot provisions nodes for workloads; a cluster nobody has shipped to has none. With no nodes, the managed Prometheus operator cannot schedule — ours had been Pending for 44 hours — so its admission webhook has no endpoints, and the Rules object in the exports overlay cannot be validated. The ship is blocked by the absence of the very workload it is trying to create.

ops observability is what breaks it, and this is exactly why the first-install order puts that command before ops ship rather than after. It applies the collector Deployment first — a workload, which makes Autopilot provision a node, which lets the operator schedule — and only then does it wait for the webhook and apply the monitoring manifests. Run it and watch a node appear:

navigator ops observability --deployment <row> --deployments-dir .

Skipping it, as we did, turns a documented ordering into forty minutes of diagnosis. Run the order as written.

Then the ship works. Every manifest applied, navigator-web and workflows-service both rolled out, and the five trigger CronJobs pinned to the tag. ops ship still exited non-zero, at the last step: it re-registers the worker with Restate and refuses to call a ship complete when that fails, because a stale handler list means webhook submissions fail silently later. That is the check behaving correctly — RESTATE_ADMIN_URL and RESTATE_ADMIN_TOKEN live in the deployment's encrypted tree for the operator, and CI supplies them from secrets.

The DNS cutover, and the gap it costs. The Ingress carries the annotation kubernetes.io/ingress.global-static-ip-name: neon-law-prod-gateway-ip, so it claims the address reserved during provisioning — wait for it to appear before touching DNS, because until GKE finishes the load balancer there is nothing behind that IP:

NAVIGATOR_GATEWAY_IP=$(gcloud compute addresses describe neon-law-prod-gateway-ip \
  --global --format='value(address)')
navigator ops dns setup --domain neonlaw.com --gateway-ip "$NAVIGATOR_GATEWAY_IP" --dry-run

Two calls: www patched off the old address, workflows created. The command never deletes. Then the wait described in Pointing a hostname at a deployment begins, and it is a real outage — the hostname serves neither the old site nor the new one while Google validates. Ours ran about twelve minutes for www. FAILED_NOT_VISIBLE on the way through is normal: it records validation attempts made while DNS still pointed elsewhere, and clears itself.

Retire the old certificate after the cutover has settled, not during it. www.neonlaw.com was previously served by a Certificate Manager certificate authorized through an _acme-challenge.www CNAME. That record is inert once the hostname points at GKE — managed certificates validate through the load balancer, not the ACME DNS challenge — but it is also what keeps the old certificate renewable, and the old certificate is what makes a DNS rollback instant. Retire the certificate, its authorization, and the record together, once you have decided not to roll back.

The lesson we would give another deployer is the one the release order already encodes: the deployment that carries real client matters goes last, and it goes last precisely because the first install is where you learn what the recipe assumes. Ours assumed a cluster that had already been shipped to once — and every gap above is one the two earlier deployments could never have found, because by the time they shipped, someone had already shipped to them.

• docs/oss-install.md — the full end-to-end install (env, Secret, overlay, image, verify). docs/deployment-secrets.md — production secret rendering. docs/third-party-integrations.md — the per-environment vendor-account convention. docs/docusign-esignature.md — e-signature setup and the one-app, two-environment model.