Health and readiness

Both rolter binaries serve two operational endpoints that answer two different questions. They are unauthenticated, cheap, and safe to probe on every pod.

The words mean the same thing on rolter-gateway and rolter-control.

Why the split exists

Collapsing both probes onto one endpoint has no correct answer, only a choice of which failure to accept:

• If the single endpoint answers as soon as Axum binds, Kubernetes marks the pod Ready before it can serve. A rolling update then routes dashboard and /internal/snapshot traffic — which every gateway in the fleet polls — at a control pod whose migrations have not run and whose pool has no connection.
• If the single endpoint instead checks Postgres, a database blip fails the liveness probe and Kubernetes kills every control pod at once. A recoverable dependency outage becomes a restart storm, and the restarts make the outage worse by reconnecting a stampede at the recovering database.

Splitting the endpoints lets a dependency failure drain traffic without touching the process.

/healthz — liveness

Returns 200 OK with the body ok whenever the HTTP server is accepting connections. It checks no dependency: not Postgres, not Redis, not ClickHouse, not an upstream provider. That is the point — nothing outside the process may cause a restart.

/readyz — readiness

Gateway

Returns 200 OK normally, and 503 Service Unavailable with the body draining once the instance has been drained from the dashboard’s cluster screen — the state arrives on the /internal/snapshot response the gateway already polls. A drained gateway therefore leaves the Service and finishes its in-flight requests, instead of being killed with them in progress.

Control plane

Returns a JSON body describing each check:

{
  "status": "ready",
  "checks": {"database": "ok", "migrations": "ok", "kek": "ok"}
}

200 when status is ready, 503 otherwise. The checks are:

• database — the Postgres pool hands out a connection within 2 seconds. The timeout is deliberately shorter than a typical probe timeoutSeconds so an exhausted pool answers a clean 503 rather than hanging until kubelet reports a timeout, which looks like a dead process.
• migrations — every migration embedded in this binary is applied to the database. A pod running ahead of its schema would serve wrong answers on the CRUD API and /internal/snapshot, so it is not ready.
• kek — ROLTER_KEK parses when it is set. A KEK set to an empty value seals nothing and decrypts nothing, so the pod cannot serve credentials. Reported as not configured (and not a failure) when the variable is unset.

Redis and ClickHouse are deliberately not checked. The control plane serves configuration without either — losing Redis costs the config-bump fan-out notification, losing ClickHouse costs the analytics screens — so their absence is degraded, not unready. Removing every control pod from the Service because analytics is down would be a strictly worse outage.

Running without --database-url (bootstrap config only), there is nothing to wait on and the answer is always ready.

The dashboard’s gateway pill

The status pill in every screen header (ui/src/components/ScreenHeader.tsx) polls the gateway’s /readyz every 30 seconds through the control plane’s /gw/* proxy (#1973).

/readyz is the one gateway signal every dashboard role can read: the proxy takes no admin token and the endpoint takes no key. /api/v1/cluster/nodes knows more but is superadmin-only, and /api/v1/health/* is provider health from ClickHouse, not the gateway’s. The probe sends no Authorization header.

readinessFrom in ui/src/lib/gateway.ts reads the answer strictly:

A 200 that is not ok is an SPA fallback in front of the control plane, and a 502 that is not the proxy’s JSON comes from an ingress that could not reach the control plane. Both read as unknown, so the pill never sends an operator to the wrong process.

One failed check does not blank a known answer. gatewayHealthFrom in ui/src/lib/gateway-health.ts keeps it for up to one and a half poll intervals, without the pulse and with the answer’s time in the tooltip, then drops to unknown. The poll pauses while the tab is hidden, and the header’s refresh button re-asks along with everything else without spinning for the background poll.

What the pill cannot see:

• other gateways behind the same ROLTER_GATEWAY_URL, and whether a gateway runs the current config version. Both need a fleet summary every signed-in caller can read (#2013).
• an unreachable gateway host. The proxy’s HTTP client has no connect timeout, so a host that drops packets hangs the request instead of answering 502, and the pill reports unknown after its own 10 second timeout rather than down (#2014).

Probe wiring

charts/rolter wires this for you:

readinessProbe:
  httpGet: {path: /readyz, port: http}
livenessProbe:
  httpGet: {path: /healthz, port: http}
startupProbe:               # control plane only
  httpGet: {path: /healthz, port: http}

The startupProbe on the control plane holds liveness off while first-boot migrations run against a cold database, so a slow migration is not mistaken for a wedged process. Tune the periods under control.probes in values.yaml.

What to alert on

• /readyz down for longer than a failover — the control plane cannot reach its database, or is running against a schema it has not migrated. Page.
• /healthz flapping — the process itself is crashing or wedging. Page.
• /readyz down on a single gateway — usually just a drained instance, which is expected during a rolling update. Do not page on this alone.