Observability

How Neon Law Navigator emits telemetry, where it lands for analysis, and how an operator debugs a durable-execution failure fast. Born from an incident: a trigger Job sat in ImagePullBackOff for days while a CronJob's concurrencyPolicy: Forbid silently skipped every run, and nothing emitted a queryable signal — the only telemetry was the nightly email, which was the thing that broke. This page exists so that never repeats.

The one rule for anyone adding a span, metric, or log field — identifiers and counts, never content. A notation_id, a service name, an outcome, a duration, a status code: yes. A client name, an answer body, an email address, a document body: never. Telemetry crosses the firm's trust boundary; client content does not. This is a standing engineering- and legal-council order, not a style preference.

One seam: telemetry::init

Every binary calls telemetry::init once in main and holds the returned guard until exit. There is no per-binary subscriber wiring anymore — web, the workflows-service worker, and all six *-trigger jobs share the one crate. Two modes, chosen by whether OTEL_EXPORTER_OTLP_ENDPOINT is set:

Unset (dev / CI / OSS fork)Set (prod)
stdouthuman-readable fmtstructured JSON (Cloud Logging parses every field)
traces—OTLP → collector
metrics—OTLP → collector
costzero — no networkone batch span + periodic metric push

Standard OTEL_* env vars (OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_SERVICE_NAME) drive everything; there is no product-specific telemetry config. The guard's drop flushes batched spans/metrics — important for the short-lived trigger jobs, which would otherwise exit before the periodic exporter fires.

What is instrumented

Every workflow trigger funnels through workflows::start_workflow, instrumented once there so every trigger inherits it:

The worker and web emit their own spans through the same subscriber, so new handlers inherit tracing for free.

Web also records first-party public website visits as aggregate analytics. The durable table and OTel counter (navigator.web.visit.count) use bounded dimensions only: UTC day, Axum matched route pattern, trusted edge country/region code, route-derived locale, coarse status class, and a source bucket derived from approved UTM/ref query parameters or referrer host classification. Do not add raw IP addresses, user agents, session or person identifiers, raw URL paths, arbitrary query parameters, full query strings, or full Referer URLs to visitor analytics. Unknown or sensitive query parameters are ignored, invalid allowed values collapse to invalid, missing referrers collapse to direct, same-site referrers to internal, and unrecognized external hosts to other. The operational view is admin-only at /admin/analytics.

Where it lands: BigQuery + the GCP consoles

Two doors, one of them is BigQuery on day one:

This complements the existing Iceberg archive (iceberg-archive guide): the nightly Archives workflow snapshots Postgres tables to Parquet on GCS for BigQuery external-table analysis. Telemetry (logs, traces, metrics) is the operational half; the Iceberg archive is the data half. Both query from BigQuery.

Seeing telemetry locally: Grafana LGTM

Emitting telemetry is only half the loop; the other half is seeing it before it reaches prod. In KIND, a single navigator dev up stands up Grafana LGTM (the one-process grafana/otel-lgtm image: Grafana, Loki for logs, Tempo for traces, Prometheus for Metrics, and a bundled OTel Collector that receives OTLP and fans it into the three stores). One pod, datasources auto-provisioned, no dashboards to import.

Bring-up is automatic. dev up applies the lgtm Deployment + Service, waits for its rollout, port-forwards Grafana 3000 → host :3000 and OTLP gRPC 4317 → host :4317, and writes OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 into .devx/env. Override the host ports with NAVIGATOR_KIND_GRAFANA_PORT / NAVIGATOR_KIND_OTLP_PORT if 3000/4317 clash. Storage is emptyDir, so tearing down (or restarting the pod) wipes all telemetry — the right default for KIND.

Generate telemetry. Host web started with .devx/env sourced picks up OTEL_EXPORTER_OTLP_ENDPOINT, which is exactly what flips telemetry::init from human-readable stdout to JSON logs + OTLP export; then drive a request. In-cluster web and workflows-service already carry OTEL_EXPORTER_OTLP_ENDPOINT pointing at http://lgtm.navigator.svc.cluster.local:4317 (baked into their manifests), so a cross-service trace stays intact. To run host web with plain stdout and no export (the pre-LGTM behavior), set OTEL_EXPORTER_OTLP_ENDPOINT= (empty) in .env — it loads before .devx/env and wins, and telemetry::init treats an empty endpoint as "do not export."

Look at it in Grafana. Open http://localhost:3000 (anonymous Admin, login disabled), then Explore (compass icon) and pick a datasource:

The local-LGTM wiring lives in three places:

Debugging "the workflow didn't run"

Work down the chain (full version in the durable-workflows guide); each rung now has telemetry:

  1. Did the trigger fire — and is a job wedged? Run navigator ops doctor. It reads the cluster and names, in plain language, any trigger Job stuck in ImagePullBackOff / CrashLoopBackOff or Active too long (which, under Forbid, skips every subsequent run) and any unready workload. It prints the exact kubectl command that fixes each finding. First stop for a missing nightly/periodic job.
  2. Did the ingress accept it? Query BigQuery for navigator.workflow.trigger.fired by service and outcome, or read the trigger-outcome log events: rejected with status=401 is a stale RESTATE_AUTH_TOKEN; status=404 is the registration gotcha; transport_error is an unreachable/hung ingress (now capped by a 30s client timeout + activeDeadlineSeconds).
  3. Did the worker run it? The Restate Cloud console → Invocations shows the journal; a failing step retries and surfaces there. The Heartbeat and Archives emails both deep-link it (RESTATE_CLOUD_CONSOLE_URL).
  4. Is durable execution alive at all? The six-hourly Heartbeat email is the liveness signal; its absence is the alert.

Tracing across the Restate boundary

A workflow kicked off from web continues the caller's trace into the durable handler, so a single trace spans "button click → ingress POST → snapshot/dispatch steps." workflows::trigger injects the current span's W3C traceparent into the outbound ingress POST (telemetry::current_trace_context_headers); each handler extracts it from ctx.headers() and parents its span on the result (telemetry::set_span_parent, used by Archives::run and every Notation handler). Only opaque trace context crosses — never a client field (LEGAL #2). When OTLP is unconfigured the inject/extract pair is a no-op, so dev / KIND / OSS forks stay zero-cost.

The Rust contract — inject produces a well-formed traceparent, extract recovers the same trace id — is covered by telemetry's round-trip test and workflows' trace_propagation integration test. The one thing only a live cluster confirms is that Restate forwards the ingress traceparent onto the handler invocation headers; verify once in KIND/prod by checking a web-initiated workflow and its steps share a trace id in Cloud Trace. If a future Restate version stops forwarding it, the fallback is to carry a trace_id in the request body and link (rather than parent) the handler span — no other code changes.

The hardening that came with this

See also