This is how we set up Neon Law Foundation

Everything above is the recipe. This is the log of us following it for the Foundation's own deployment, neon-law-prod in the neon-law-prod project, serving www.neonlaw.com. It is written down because the first install into a cold cluster went differently from the happy path, and the difference is worth knowing before you hit it.

The Foundation's deployment is the third row in our matrix and the one that holds real pro bono matters, so it went last — after the same release had proven itself on neon-law-stg and neon-law-prod. navigator ops gcp setup had already built the project's half of the world: the neon-law-prod GKE Autopilot cluster, the navigator-secrets KMS keyring, five storage buckets, and the reserved global address neon-law-prod-gateway-ip.

Then the first ops ship failed, and kept failing. Three things were in the way, in the order we hit them.

The object list was a superset. ops secrets apply --deployment <row> --deployments-dir . --dry-run failed closed naming nine DocuSign objects. The Foundation's deployment executes no documents: it supplies no DOCUSIGN_BASE_URL, declines the integration, and runs StubSignatureProvider. But the shared SecretProviderClass referenced all nine anyway, plus the three engineering-webhook objects scoped to the automation home that this project must never hold. A CSI mount fails the whole volume on one object it cannot read, so the only way past was a placeholder credential — and a placeholder boots the real provider, because DocuSignSignatureProvider::from_env returns Some for any non-empty value. A green deploy that fails on its first signature request, on the deployment holding real matters.

That was a genuine defect rather than a configuration mistake, and the fix was to render the object list per deployment so the class references exactly what the deployment writes. It is described in docs/deployment-secrets.md. If your own deployment declines an integration the manifest names, this is the machinery that lets it.

Secret Manager was empty. With the object list correct, one command filled it:

navigator ops secrets apply --deployment <row> --deployments-dir .

Twenty-five objects written, twelve reported as skipped with their reasons. Read that skipped line — it is also the list of what this deployment's mount will not ask for.

A cold cluster has no namespace, and no Secret to preflight against. This is the one to plan around. ops ship confirms the deployment's Secret satisfies the running binary's boot invariants before it reconciles anything, which is the right order for every ship after the first — an unsatisfied requirement aborts a ship that has touched nothing. On a first ship it is a standoff: the namespace is created by the apply in step 5, and the Secret is projected by the CSI driver only while a pod mounts the volume, so neither exists when step 4 goes looking. The failure reads:

Error: kubectl get secret neon-law-prod-web-secrets failed:
Error from server (NotFound): namespaces "neon-law-prod" not found

We broke the standoff by hand: created the namespace, then seeded a plain neon-law-prod-web-secrets Secret from the Secret Manager objects ops secrets apply had just written, so the preflight had something real to read. The projected Secret takes the same name, so once the pods are up and the driver owns it, the seeded one is retired exactly as "Retire the plain Secret" describes.

Do not read that paragraph as a runbook. Doing it by hand is off the invariant this whole workshop rests on — the navigator CLI orchestrates every machine-bound flow, and reaching around it with ad hoc commands is precisely what leaves the next operator without a path. The bootstrap belongs in the CLI, and the shape it should take is a preflight that falls back to the key set the rendered SecretProviderClass will project when no live Secret exists yet: those objects are already proven to resolve to an ENABLED version before the reconcile, so it checks the same property from the authoritative source instead of from a cluster that has nothing in it. Until that lands, a first install into a cold cluster needs a human, and you should expect it.

A cold Autopilot cluster has no nodes, and that stops the reconcile. With the Secret readable, ops ship rendered and diffed, then failed on an admission webhook:

Error from server (InternalError): failed calling webhook
"validate.rules.gmp-operator.gke-gmp-system.monitoring.googleapis.com":
no endpoints available for service "gmp-operator"

The chain is circular and worth recognising on sight. Autopilot provisions nodes for workloads; a cluster nobody has shipped to has none. With no nodes, the managed Prometheus operator cannot schedule — ours had been Pending for 44 hours — so its admission webhook has no endpoints, and the Rules object in the exports overlay cannot be validated. The ship is blocked by the absence of the very workload it is trying to create.

ops observability is what breaks it, and this is exactly why the first-install order puts that command before ops ship rather than after. It applies the collector Deployment first — a workload, which makes Autopilot provision a node, which lets the operator schedule — and only then does it wait for the webhook and apply the monitoring manifests. Run it and watch a node appear:

navigator ops observability --deployment <row> --deployments-dir .

Skipping it, as we did, turns a documented ordering into forty minutes of diagnosis. Run the order as written.

Then the ship works. Every manifest applied, navigator-web and workflows-service both rolled out, and the five trigger CronJobs pinned to the tag. ops ship still exited non-zero, at the last step: it re-registers the worker with Restate and refuses to call a ship complete when that fails, because a stale handler list means webhook submissions fail silently later. That is the check behaving correctly — RESTATE_ADMIN_URL and RESTATE_ADMIN_TOKEN live in the deployment's encrypted tree for the operator, and CI supplies them from secrets.

The DNS cutover, and the gap it costs. The Ingress carries the annotation kubernetes.io/ingress.global-static-ip-name: neon-law-prod-gateway-ip, so it claims the address reserved during provisioning — wait for it to appear before touching DNS, because until GKE finishes the load balancer there is nothing behind that IP:

NAVIGATOR_GATEWAY_IP=$(gcloud compute addresses describe neon-law-prod-gateway-ip \
  --global --format='value(address)')
navigator ops dns setup --domain neonlaw.com --gateway-ip "$NAVIGATOR_GATEWAY_IP" --dry-run

Two calls: www patched off the old address, workflows created. The command never deletes. Then the wait described in Pointing a hostname at a deployment begins, and it is a real outage — the hostname serves neither the old site nor the new one while Google validates. Ours ran about twelve minutes for www. FAILED_NOT_VISIBLE on the way through is normal: it records validation attempts made while DNS still pointed elsewhere, and clears itself.

Retire the old certificate after the cutover has settled, not during it. www.neonlaw.com was previously served by a Certificate Manager certificate authorized through an _acme-challenge.www CNAME. That record is inert once the hostname points at GKE — managed certificates validate through the load balancer, not the ACME DNS challenge — but it is also what keeps the old certificate renewable, and the old certificate is what makes a DNS rollback instant. Retire the certificate, its authorization, and the record together, once you have decided not to roll back.

The lesson we would give another deployer is the one the release order already encodes: the deployment that carries real client matters goes last, and it goes last precisely because the first install is where you learn what the recipe assumes. Ours assumed a cluster that had already been shipped to once — and every gap above is one the two earlier deployments could never have found, because by the time they shipped, someone had already shipped to them.