rolter documentation

Start here. rolter is a high-performance OpenAI/Anthropic-compatible AI gateway and load balancer (Rust data plane + control plane, shadcn/ui dashboard).

Architecture

• Overview — system shape, crates, data/control plane split
• Load balancing — strategies and the cache-aware design
• Caching — response cache and KV-cache affinity
• Provider prompt caching — portable Anthropic prompt-cache controls
• Multi-region deployment — active-active/passive state and failover topology
• Config & hot reload — reload-free updates
• Data model — tenancy, keys, pricing, budgets
• RBAC & auth — roles, virtual keys, membership sources
• Invitations — one-time onboarding links for deployments with no IdP
• Single sign-on (OIDC) — optional OIDC login and group→role mapping
• MCP OAuth — MCP servers, consent grants, sealed token sessions
• SCIM provisioning — IdP-driven user lifecycle and provisioning tokens
• Security — secret handling, threat model
• Observability — metrics, tracing, logs
• Performance — hot-path principles and targets

Decisions

• ADRs / decision log

API

• OpenAI & Anthropic surface

Development

• Setup
• Testing
• Contributing
• Commit conventions
• Packaging (uv / cargo / docker)

Deployment

• Zero-config quickstart — start with no keys, add providers at runtime
• Configuration reference
• Custom CA bundles — trust private upstream PKI without disabling TLS verification
• Docker

Planning

• Roadmap
• TODO

Architecture overview

rolter is split into a data plane (the hot proxy path) and a control plane (management, RBAC, config writes, UI host), sharing a set of Rust library crates. This separation lets the proxy stay lean and fast while the control plane handles CRUD, auth, and persistence.

Components

flowchart LR
  subgraph clients[Clients]
    C([OpenAI / Anthropic SDKs])
    A([Dashboard user])
  end
  C -->|/v1/*| GW["rolter-gateway<br/>data plane (Axum)"]
  A --> CTL["rolter-control<br/>control plane + UI host"]
  GW -->|forward + stream| UP["Upstreams<br/>OpenAI · Anthropic · vLLM pool"]
  CTL -->|read/write| PG[("PostgreSQL<br/>source of truth")]
  CTL -->|publish change events| RDS[("Redis / Valkey")]
  RDS -->|notify| GW
  GW -->|atomic snapshot swap| GW
  GW -->|async batched writes| CH[("ClickHouse<br/>logs + cost")]

Crates

• rolter-core — config model, domain types, error type, telemetry init. No I/O.
• rolter-balancer — LoadBalancer trait + strategies; pure and unit-tested.
• rolter-proxy — Forwarder over pooled reqwest clients; header injection, model rewrite, streaming, per-provider egress proxy.
• rolter-store — repository traits + in-memory impl; Postgres/Redis/ClickHouse backends land behind features.
• rolter-auth — virtual-key verification, roles, access checks.
• rolter-gateway (bin) — the data plane.
• rolter-control (bin) — the control plane + static UI host.

Data plane (rolter-gateway)

Built on Axum/Hyper/Tower. Holds an ArcSwap<Snapshot> routing table; reads are lock-free. Per request it:

1. parses the JSON body enough to read model and stream
2. authenticates the virtual key (when keys are configured) and checks model access
3. resolves the route, asks the route’s balancer to pick a target, then observes it (cache-aware learning)
4. forwards to the upstream provider with the right auth header, rewriting the model id
5. streams the response body straight back with minimal copying

See data plane details in performance.md.

Control plane (rolter-control)

Hosts the management REST API consumed by the dashboard and serves the built SPA as static assets (ServeDir). It owns all writes to Postgres, enforces RBAC, and publishes a config-change event to Redis so gateways hot-swap their snapshot without a restart (see config-and-hot-reload.md).

Datastores

• PostgreSQL — source of truth for tenancy, RBAC, providers, routes, virtual keys, pricing, budgets.
• Redis / Valkey — response cache, rate-limit counters, cooldown state, and pub/sub for config propagation.
• ClickHouse — high-volume request and cost logs, queried by the dashboard for usage analytics.

Request lifecycle (sequence)

sequenceDiagram
  participant Client
  participant Gateway
  participant Balancer
  participant Upstream
  participant ClickHouse
  Client->>Gateway: POST /v1/chat/completions
  Gateway->>Gateway: auth virtual key + model access
  Gateway->>Balancer: pick(route, ctx)
  Balancer-->>Gateway: target index
  Gateway->>Upstream: forward (Bearer/x-api-key, rewritten model)
  Upstream-->>Gateway: response stream
  Gateway-->>Client: streamed body
  Gateway--)ClickHouse: async usage + cost log

Extensibility

• New balancing strategy: implement rolter_balancer::LoadBalancer, wire into build().
• New provider protocol: extend ProviderKind and rolter-proxy translation.
• New storage backend: implement rolter-store traits behind a cargo feature.
• New modality (audio/image/video): add endpoints + provider adapters; the balancer/auth/logging layers are modality-agnostic.

Load balancing

Each route maps a public model name to one or more upstream targets and a strategy. Strategies implement rolter_balancer::LoadBalancer:

#![allow(unused)]
fn main() {
pub trait LoadBalancer: Send + Sync {
    fn name(&self) -> &'static str;
    fn pick(&self, ctx: &RouteContext, loads: &[u64]) -> Option<usize>;
    fn pick_eligible(
        &self,
        ctx: &RouteContext,
        loads: &[u64],
        eligible: &dyn Fn(usize) -> bool,
    ) -> Option<usize> {
        self.pick(ctx, loads)
    }
    fn observe(&self, target: usize, ctx: &RouteContext) {}
}
}

pick returns an index into the route’s targets; observe lets learning strategies (cache-aware) record what a target served. The gateway calls pick_eligible, passing which targets the attempt can use (not tried, not parked, healthy, breaker not open, allowed to the key). The default ignores that and calls pick, since the gateway skips an ineligible pick itself; cache_aware overrides it so a target that takes no traffic cannot shape its load comparison; power_of_two and the pipeline stacks do not yet (#1942). RouteContext carries an optional session_key (from x-session-id) and the request prompt used for affinity scoring. The gateway bounds prompt to the conversation’s first 32 KiB and passes the whole prompt’s size and identity alongside as prompt_len and prompt_digest (caching.md).

Strategies (v1)

• round_robin — sequential rotation; predictable, zero state.
• random — uniform random; good for simple homogeneous pools.
• power_of_two — pick the less loaded of two random targets; needs a load snapshot.
• consistent_hash — hash-ring keyed by session_key (falls back to a digest of the whole conversation: its text plus the identity of its images, tool calls and tool results, or to the raw body when the gateway could not read one); pins a session/user to a target for KV reuse, survives target changes with minimal reshuffle (100 vnodes per target).
• cache_aware — approximate prefix affinity on the conversation text, with a load guard: a warm replica keeps its affinity until it runs more than 2 requests and 1.5× past the least-loaded usable one, then the request spreads (#1851). Unhealthy, parked and breaker-open targets are left out of the comparison through LoadBalancer::pick_eligible; see caching.md.
• weighted — smooth weighted round-robin honouring each target’s weight.
• pipeline — composable filter → weighted-score → argmax selection: eligibility filtering drops ineligible targets, then a stack of Scorers (session affinity + static weight + in-flight load + prefix-cache affinity) is combined as a weighted sum and the argmax wins (ties broken randomly). Session affinity pins repeat requests from the same x-session-id to their last-served target (TTL-bounded) for warm-cache reuse. The extension point every future cost/latency/KV-cache scorer plugs into.
• precise_cache_aware — consumes each target’s vLLM ZMQ KV-event stream and scores the exact leading fraction of caller-supplied token blocks resident on that target. Missing token ids and stale, malformed, disconnected, or sequence-gapped streams stay neutral; least-load routing remains the fallback.
• lmcache_aware — polls each target’s configured LMCache controller signal and prefers available caches with free capacity (1 - occupancy). Empty, saturated, failed, and stale controllers stay neutral and fall back to least load.
• adaptive — a weighted blend of observed latency, catalog cost and in-flight load, governed by the deployment-wide [adaptive_routing] policy. See below.
• predicted_latency — rank targets by what this request is modelled to cost on each of them, from the queue it would join and its own prompt size, rather than by a per-target average. See below.
• lora_aware — LoRA-adapter affinity for a fleet serving many adapters over shared base weights: prefer a target that already holds the requested adapter resident, with prefix affinity and in-flight load behind it. See below.

Which strategies read a target’s weight

build_with_stats hands the weight slice to three strategies: weighted, pipeline (its static-weight scorer) and adaptive (its fallback stack). Every other strategy is built from the target count alone, so a weight on one of their targets is stored and never consulted. The dashboard mirrors that list as usesWeights() in ui/src/lib/strategies.ts: the Model Catalog, the model sheet and the Routing Rules cards state a traffic share only for those three, and say the weights are ignored everywhere else (#1979, #2133). A new strategy that reads weights has to be added there too, or the dashboard will tell operators its weights do nothing.

Balancer lifetime, and provider groups

A balancer belongs to the pool, not to the request. Most strategies carry selection state — round_robin’s counter, weighted’s running weights, cache_aware’s trie, adaptive’s samples — and that state is exactly what makes them balance. A balancer built per request is not a balancer: a fresh RoundRobin always returns index 0, and a fresh WeightedRoundRobin always returns the heaviest target, so every request lands on the same place.

Configured routes get this for free: Snapshot::build constructs one balancer per route and hands out references to it for the snapshot’s whole life, and a config reload builds a new snapshot (and so a new balancer, deliberately — a new snapshot may have different targets, and a carried-over index would name a different provider).

A provider group address (group-slug/model, ADR-0017 addendum) is the case that is not free. A group has no configured route: Snapshot::resolve_pinned builds a synthetic one per request, because the model each member forwards depends on what the caller asked for. Only the target list is per request, though — the strategy and the member weights are fixed for the life of the snapshot, so the group’s balancer is built once in Snapshot::build, stored in Snapshot::group_balancers keyed by slug, and shared by every synthetic entry built for that group. Before #1655 it was built inside resolve_pinned, and a whole fleet’s traffic went to whichever member was listed first, whatever the group’s strategy and weights said.

Costs for a group’s balancer come from each member’s declared upstream_model, since the requested model is not known at build time and a group-slug/model address is never itself a priced model. A group whose members are all passthrough is therefore cost-neutral, which is the right answer for the homogeneous fleet a group exists to describe.

A provider slug address pins a single provider, so its pool has one target and no selection state to keep; that balancer is still built per request.

Adaptive routing

adaptive is the only strategy whose behavior is owned by a global policy rather than the route:

[adaptive_routing]
enabled = false           # kill switch; the whole feature is off by default
latency_weight = 1.0
cost_weight = 0.5
load_weight = 0.25
exploration_ratio = 0.05  # clamped to [0, 0.5]
min_samples = 50

Three conditions must all hold before the blend routes a single request: the kill switch is on, at least one weight is non-zero, and the route has both served min_samples requests and gathered latency samples for at least two targets. Until then — and immediately again if the policy is switched off — every pick goes to the same pipeline stack the route would have used otherwise, so moving a route to adaptive shifts no traffic on its own. The fallback stack keeps learning while the blend is engaged, so disengaging lands on a warm session/prefix cache.

Warm-up

The latency condition cannot be satisfied by waiting (#1645). The fallback stack is free to send every pick to one target — a configured weight, session affinity or prefix affinity all do this legitimately — and the peer it skips then never earns the latency sample the evidence check is counting. Before the fix a route on adaptive over such a pair sat at engaged: false with fallback: 50 of 50 picks and samples: 0 on the unsampled target indefinitely, which made adaptive permanently equal to the strategy it replaced.

So while the evidence is thin, a disengaged route steers picks to the targets that have no latency sample yet, at most eight per target. Those picks are counted as exploration in the decision split, since they buy evidence rather than the best target. The budget is what bounds the cost: a target that never answers absorbs at most eight picks in total and is then left to the fallback stack, rather than swallowing a permanent share of the route. The budget is eight rather than one because a latency sample lands only when a request finishes, so a smaller one would be spent on picks still in flight and the warm-up would give up on a healthy target under concurrency. The kill switch still outranks all of this: a policy with enabled = false probes nothing.

The policy is also owned by the control plane: GET/PUT /api/v1/adaptive-routing-policy (superadmin only) persists it, audits the change as adaptive_routing_policy.update, and returns the affected_routes the change reaches. It travels to the data plane in the normal snapshot, so a change applies without a restart. The API refuses an all-zero blend outright — stopping adaptive routing is what enabled = false is for.

Once engaged, an exploration_ratio share of picks is made uniformly at random so a target the blend has learned to avoid keeps producing fresh latency samples instead of going dark. Operator input is clamped on the way in: negative weights become zero and exploration never exceeds half the traffic.

Telemetry

The scores the blend ranks on live in the balancer, which lives in the gateway process, so the control plane cannot read them directly. Each gateway therefore pushes a sample every 15s to POST /internal/adaptive-telemetry on the control plane — the same internal token and the same x-rolter-node-id identity as the snapshot poll, so one node is one row here and in the cluster inventory (#543). Nothing is written on the request path: the sample is taken by a background task, and a pick costs two relaxed atomics for per-target attribution.

Per route and target the sample carries the blended score, its latency/cost/load components, the raw signals behind them (smoothed latency in ms, catalog price, in-flight count), how many picks that target has served and how long ago the last one was — plus the decision split, and the sanitized policy that node actually runs, which can lag the stored policy until the node converges.

The control plane keeps the newest sample per (node, model) in adaptive_routing_telemetry and serves GET /api/v1/adaptive-routing-telemetry (superadmin only), grouped by route with one entry per reporting node. Samples older than 60s are excluded as no longer current, and rows are pruned after an hour so a scaled-down node leaves the scoreboard.

This is deliberately current state, not history: one row per node and route, overwritten on every report, so the table is the size of the fleet rather than of the traffic. A time series belongs in the request log, and would be a separate endpoint rather than a change to this one. Prometheus deployments get the same per-target scores without the dashboard from rolter_adaptive_routing_target_score{model,target}.

LoRA-aware routing

For a fleet serving many LoRA adapters off shared base weights, routing a request to a node that already holds the adapter is the same class of win as prefix-cache affinity (#853, borrowed from llm-d). lora_aware composes with the existing scorers rather than replacing them:

Adapter residency outranks prefix affinity because the costs are not comparable: missing a warm prefix recomputes some tokens, while missing a resident adapter can force the engine to load adapter weights before it decodes anything at all. Load stays in the stack so a fleet where every target holds the adapter still balances instead of pinning.

Residency is learned from traffic, not declared. rolter cannot see which adapters an engine currently holds, and a static declaration would go stale the moment the engine evicted one. Each target keeps a bounded LRU set of the adapters it has recently served, sized to mirror vLLM’s --max-loras, so it tracks what the engine plausibly still holds rather than everything it has ever served. An unbounded “has ever served” set would be worse than no scoring at all, because it steers confidently to a target that went cold long ago.

Adapter identity is the requested model, which is how vLLM addresses adapters over shared base weights. The gateway only sets it when the request addresses something other than the route’s own model — that is, a passthrough provider-group route (ADR-0017). On a single-model route the two are equal, no adapter is set, and the scorer is inert. This matters: adapter affinity deliberately pins rather than spreads, so treating a route’s one model as an adapter would pin the entire route to whichever target happened to serve first.

When no candidate holds the adapter — a cold adapter, or a request with none — every candidate scores 0.0. That is neutral rather than a penalty: an equal contribution cannot move the argmax, so the rest of the pipeline decides and this scorer stays silent instead of guessing.

Predicted-latency scheduling

fastest ranks on a per-target latency EWMA. That is the right signal when every request is the same size, and the wrong one otherwise: a target whose average is high because it happened to serve the long prompts looks slow even when it is the emptiest box in the fleet, and a target with a deep queue looks fast right up until the queue is what you join.

predicted_latency models the cost instead of averaging it (#853, borrowed from llm-d). Per target:

latency_ms ≈ w0 + w1 · queue_depth + w2 · prompt_ktokens

w0 is fixed overhead, w1 is what one queued request ahead of you costs, and w2 is prefill cost per thousand prompt tokens. The three coefficients are learned online from completed requests with normalized least mean squares — one multiply-add per feature per sample, no matrix, no allocation, no periodic refit.

Load stays in the stack for two reasons: it carries the route while the models are cold, and once they are warm it is the tiebreaker between targets the model rates equally — the common case on a homogeneous fleet, where the honest answer is “either, take the emptier”.

Three features, not more. A gateway sees queue depth and prompt size. It does not see batch composition, KV-cache pressure, or where the engine is in its scheduling loop, and a model with parameters it cannot observe fits noise. The interesting error is between “the queue is deep” and “the prompt is long”, and a linear model separates those. That is the honest ceiling for this vantage point.

A cold target predicts nothing. Below 8 completed requests a target returns no prediction, and the scorer reads that as unknown, not as slow. A route that switches to predicted_latency therefore behaves exactly like the least-load pipeline until the models have evidence, so the switch itself moves no traffic.

Only successful requests teach. The same gate the latency EWMA uses: a fast failure cannot train a target into looking cheap. Coefficients are clamped, so a client that held a stream open for a week produces one bad sample rather than a permanently poisoned model.

Queue depth is read before the increment, so the model learns the queue a request joined, not the one it created.

Models live in the load tracker, not the routing snapshot, for the same reason the latency EWMA does: a config reload must not throw away what they learned, or every reload would send the route back to cold behaviour. A route that gains a target keeps its existing models and the new target reads as unpredicted until the next process start — the conservative direction, since a target with no evidence should not be ranked.

Choosing a strategy

Both external strategies perform network I/O only in background tasks. The request hot path reads bounded in-process state and atomics.

Selecting a strategy from the dashboard

The route and provider-group editors offer every strategy in this table except adaptive, which is governed by the deployment-wide [adaptive_routing] policy and has its own screen — a per-route dropdown would misrepresent how it is controlled. precise_cache_aware and lmcache_aware are offered with a hint that they need a telemetry source on the target providers, since without one they fall back to least-load silently rather than failing.

The hint also says where that source is set. [providers.kv_events] and [providers.lmcache] exist only on a provider defined in rolter.toml: the postgres store loads every database provider with both set to None, and neither the CRUD API nor the provider sheet has a field for them. A route or group whose targets are all dashboard providers therefore always runs on the least-load fallback under these two strategies, and the hint says so instead of sending the operator to a provider setting that does not exist (#2137). Storing both on database providers is tracked in #2236; when it lands, that part of the hint goes. When the deployment has a documentation host, the hint links the user-facing cache-aware routing page through DocsLink; an air-gapped deployment with none gets the hint without a link.

The two strategies stay selectable: a route or group can target providers from rolter.toml that do carry a source, and an operator who wired one up must be able to choose them.

A picker always renders the value the route or group already holds, even one it would not otherwise offer — including a strategy set from rolter.toml or the API, or one added to the backend allowlist ahead of the dashboard. A native <select> whose value matches no option displays the first option instead, so before #897 a group balanced by adaptive read as round_robin. Whatever the menu chooses to offer, editing must never rewrite a strategy the operator did not touch.

Caching

rolter deals with two distinct kinds of caching.

1. KV-cache affinity (load balancing)

The big win for self-hosted fleets: vLLM/SGLang reuse the attention KV cache for shared prompt prefixes (system prompts, few-shot examples, conversation history). But that only helps if the next matching request lands on the same replica. Naive round-robin scatters related requests and destroys cache locality.

The cache_aware strategy keeps, per target, a byte trie of prompts it has served. For an incoming prompt it computes the fraction of leading bytes already present on each target and:

• if the best match ≥ threshold (default 0.5), pins the request to that target (cache hit) — unless that target is overloaded (below)
• otherwise spreads to the least-loaded target, or the least-warmed one when no load is known

The “prompt” is the conversation, not the request body (#1851). prompt_affinity::affinity reads each turn’s role and text in the order the model sees them — messages for chat, system plus messages for Anthropic, instructions plus input for Responses, prompt for completions — and the raw body is only the fallback for a shape it cannot read. Before #1851 the trie was fed the raw JSON, and the envelope every request to a model opens with ({"model":"…","messages":[{"role":…) cleared the threshold by itself, so unrelated requests all pinned to whichever replica served first.

Only the first 32 KiB of that text is copied into RouteContext::prompt, cut on a character boundary, since prefix affinity is decided by the leading bytes: the trie walk stays bounded and a long prompt never costs a second full copy of itself. The whole prompt is still measured and hashed as it is read, and travels as RouteContext::prompt_len and RouteContext::prompt_digest (xxh3). Both cover what the text leaves out. Tool calls and results (Anthropic tool_use / tool_result, OpenAI tool_calls, Responses function_call / function_call_output) are prefilled like any other token and are often most of an agent’s prompt, so they count towards the length and the digest. Images, audio and files count towards the digest only, since their encoded size says little about their token cost. Neither is copied: a non-string value is serialized straight into the hasher. prompt_affinity::route_context builds the context for every JSON request, so the proxy and the tests hand the balancer the same thing. Each prompt reader takes the part it needs:

• cache_aware and the pipeline’s prefix scorer read the bounded text.
• consistent_hash without a session keys on the digest, so requests that share a long system prompt and differ after it, or differ only in an attached image or tool result, still spread over the ring. Hashing the bounded text would send all of them to one target.
• The latency predictor’s token estimate is prompt_len / 4, so a long prompt keeps growing in size past the cut instead of stopping at 8192 tokens, and a tool-heavy agent turn is sized by its tool results too.

The load guard. Affinity alone is winner-take-all: the replica that served a shared system prompt first holds the longest match for every request that shares it. After SGLang’s cache-aware router, CacheAware::pick keeps the affinity only while the warm target’s in-flight count stays within both balance margins of the least-loaded target — more than BALANCE_ABS_THRESHOLD (2) requests ahead and more than BALANCE_REL_THRESHOLD (1.5×) its load is an imbalance, and the request goes to the least-loaded target instead. The absolute margin lets a warm replica take a short queue for its cache; the relative one keeps a busy pool from treating a gap of a request or two as an imbalance. With no load known there is nothing to balance against, and affinity stands.

Every comparison runs over the targets the gateway can use for the attempt. pick_untried passes its eligibility (not tried yet, not parked on a cooldown, healthy, breaker not open, allowed to the caller’s key) through LoadBalancer::pick_eligible, and CacheAware restricts the affinity match, the least-loaded minimum and the fallbacks to those targets. A replica that takes no traffic drains to an empty queue, and counting its zero made every warm replica with three requests in flight look overloaded, which sent the spill to the dead replica and from there to target 0 by index. When nothing is eligible the pick weighs every target, as a plain pick does, and the gateway fails open as before. The variant path passes the same rules to each variant’s balancer (variant_target_eligible). The eligibility is asked once per target and kept in a word for routes of up to 64 targets, so a pick allocates nothing. Each answer still takes the cooldown and breaker locks, and power_of_two and the pipeline stacks still pick blind to it; both are #1942.

This is approximate (no coupling to the engine). The per-target trie is capped at a node ceiling (default 1M nodes) with LRU eviction: inserting past the cap drops the least-recently-inserted prompt, pruning only the nodes that become unreferenced (shared prefixes survive). Each trie exposes an eviction counter for observability.

flowchart TD
  R[incoming prompt] --> S{best prefix match >= threshold?}
  S -- yes --> B{best target past both balance margins?}
  B -- no --> P[pin to best target<br/>cache hit]
  B -- yes --> L
  S -- no --> L[least-loaded / least-warmed target]
  P --> O[observe: insert prompt into target trie]
  L --> O

Precise vLLM mode

Set a provider’s [providers.kv_events] block and choose strategy = "precise_cache_aware". rolter supports the vLLM V1 msgpack KVEventBatch protocol documented for vLLM 0.22: a three-frame ZMQ publication (topic, big-endian sequence, payload) containing tagged BlockStored, BlockRemoved, and AllBlocksCleared events. BlockStored.token_ids and block_size derive stable local prefix identities; external block hashes allow exact removals. State is capped by max_blocks per provider.

Exact scoring needs the token ids produced by the same tokenizer as vLLM. Send them in x-rolter-vllm-token-ids as comma-separated unsigned integers. Without the header, with stale/malformed events, or after a sequence gap, the scorer is neutral and least-load routing takes over. A sequence gap clears the local index and precise scoring remains disabled until an AllBlocksCleared event establishes a clean boundary.

[[providers]]
name = "vllm-a"
kind = "openai_compatible"
api_base = "http://vllm-a:8000"

[providers.kv_events]
endpoint = "tcp://vllm-a:5557"
topic = "kv-events"
max_blocks = 1000000
stale_secs = 30

vLLM must enable KV events with the ZMQ publisher and matching topic. Metrics expose consumed/malformed events, stream failures, decision count, and per-provider freshness.

LMCache-aware mode

Set [providers.lmcache] and use strategy = "lmcache_aware". The supported controller signal is an HTTP 200 JSON object:

{"occupancy": 0.42, "cache_available": true}

occupancy is clamped to [0,1]; available targets score 1 - occupancy, unavailable targets score zero. Polling happens in the background. Failed, malformed, or stale signals are neutral, so existing least-load routing continues.

[providers.lmcache]
endpoint = "http://lmcache-a:9000/v1/occupancy"
refresh_secs = 2
stale_secs = 10

2. Response cache

Optional caching of full responses to cut cost/latency for repeated requests:

• exact: hash of the normalized request → cached response (Redis), short TTL, opt-in per route/key.
• semantic: after an exact miss, embed the normalized conversation text through a configured provider and compare cosine similarity against a bounded recent-entry window in Redis. The route controls the threshold and candidate cap. Embedding, Redis, and decode failures fail open to normal routing. Only compatible requests are ever compared — see Semantic compatibility.

Streaming responses are cached on completion and replayed as a synthetic stream. Cache status is surfaced via response headers (e.g. x-rolter-cache: hit|miss).

[cache]
enabled = true

[routes.cache]
enabled = true

[routes.cache.semantic]
provider = "openai"
model = "text-embedding-3-small"
threshold = 0.92
max_candidates = 256

Post-response policy on cached responses

Enabling the cache must never change which policy a completion passes through (#1477). A buffered, successful body is finalised by exactly one function, DeliveryPolicy::apply in crates/rolter-gateway/src/handlers.rs. It runs the built-in output guardrails, then the post_response plugin chain, then the PII sanitizer’s response leg, and returns either the body or the refusal to send instead. Every delivery path calls it:

The entry is written before the policy runs, so it is always the upstream’s own bytes. A transform is re-applied per delivery. A rule or plugin installed by a hot reload reaches entries cached before it. A PII restore uses the ticket of the request being answered and is never persisted. A refusal is returned as Delivery::Refuse with the reason for the log row. On the live and cacheable-miss paths it goes through withheld_response, so the upstream call is still billed. A refused hit writes a zero-cost withheld row (see Billed but withheld). Streamed responses skip all three stages exactly as they do uncached (the request path has already refused the stream or had it opted out), so a cached SSE replay gets the same treatment as a live stream.

crates/rolter-gateway/src/cache_policy_tests.rs holds the HTTP regression suite. It covers cache off, the first miss, and exact and semantic hits, with blocking, transforming and unreachable (fail-closed and fail-open) plugins, a hot reload that installs a plugin over existing entries, and per-request PII restoration with a check that nothing restored reaches the store.

Semantic compatibility

A semantic hit replays a reply produced for a different request, so similar wording is not enough on its own: two requests can read alike and still expect incompatible replies (#1476). crates/rolter-gateway/src/semantic.rs splits every request into two parts before any similarity is computed:

• the text that is embedded: the user and assistant turns (with an OpenAI name), or the prompt of a legacy completion;
• a partition: a sha-256 over the canonical, key-sorted form of every other field of the post-injection body, including stream, stream_options, tools, tool_choice, response_format, sampling and length parameters, the model, and the dialect. OpenAI system/developer messages (with their position) and the Anthropic top-level system are partition fields too, so instructions are matched exactly rather than by similarity. "Answer in German" and "Answer in Japanese" embed almost identically, so this is the only safe way to compare them.

The partition is part of the Redis index key, so the candidate scan never sees an entry from another partition. It is an allowlist of what may vary, not a list of what must match. Only the conversation text, user and metadata are left out (attribution that does not change the reply), and stream: false is folded into an absent stream. A field the gateway does not know splits the cache, which costs hit rate but never correctness. As a last check before replay, a hit whose stored content type disagrees with the caller’s stream flag is treated as a miss.

Requests whose meaning the text would not capture bypass semantic lookup before the embedding call, and so cost no embedding spend. They still use the exact cache.

Offering tools in the request (tools, tool_choice) is supported. The schema is a partition field, so a reply is only replayed to a request that offered exactly the same tools.

The index key carries a layout version (<namespace>:semantic:v2). Entries written before #1476 had no partition and live under the old <namespace>:semantic key, so after an upgrade they are never read again and expire on their TTL. Bump SEMANTIC_LAYOUT_VERSION whenever the partition rules change.

The HTTP regression suite is crates/rolter-gateway/src/semantic/http_tests.rs. It serves the gateway in-process against a stub upstream whose embeddings are identical for every input, with the in-memory cache backend (ResponseCache::in_memory, test builds only), so it needs neither Redis nor a paid model. Because similarity alone would match any two requests there, each miss it asserts is the partition or the bypass at work, and each hit shows the partition does not over-split.

Configuration & hot reload

A core requirement: operators change routes, providers, keys, limits and pricing from the UI and have them take effect without restarting the gateway.

Sources of config

1. Bootstrap file (rolter.toml) — used for first run, local dev, and IaC. Maps to rolter_core::GatewayConfig.
2. Database (Postgres) — the runtime source of truth once the control plane is running. The control plane composes a GatewayConfig-equivalent snapshot from normalized tables.

Propagation

sequenceDiagram
  participant UI
  participant Control as rolter-control
  participant PG as PostgreSQL
  participant Redis
  participant GW as rolter-gateway
  UI->>Control: PUT /api/v1/routes/... (RBAC checked)
  Control->>PG: write change in a transaction
  Control->>PG: bump config_version.version
  Control->>Redis: PUBLISH rolter.config {version}
  Redis-->>GW: message {version}
  GW->>Control: GET /internal/snapshot?version=N
  Control-->>GW: full snapshot (JSON)
  GW->>GW: build Snapshot, ArcSwap::store (atomic)

• The gateway keeps the routing table in an ArcSwap<Snapshot>. Swapping is atomic and wait-free for readers — in-flight requests keep using the old snapshot; new requests see the new one.
• Versioning: config_version in Postgres is the monotonic source of truth. The gateway also reconciles on an interval (and at startup) so a missed pub/sub message self-heals.
• Validation: the control plane validates a snapshot (every route target references a known provider, etc.) before bumping the version, so gateways never load a broken config.
• Per-row resilience: validation used to be all-or-nothing, so a single half-built row 500’d /internal/snapshot and froze config propagation for every tenant. Rows that are unservable only by themselves are now omitted from the snapshot instead. That covers both a route with no usable target (created before its targets are added, or pointing at a deleted provider) and a provider whose own definition is invalid — a malformed api_base, a hosted adapter missing its api_key_env. Dropping a provider drops the routes that depended on it, since they are then left with no resolvable target. Structural problems that span rows (duplicate names, a bad metrics_path, unreadable CA bundles) still fail the whole snapshot, since silently dropping one of two colliding rows would be worse than refusing.
• Saying what was dropped (#926): omitting an entry quietly is its own failure mode — every gateway keeps serving its last good config, so nothing looks broken until a gateway restarts cold or someone wonders why a change never took effect. So the reasons travel with the config:
  • /internal/snapshot carries a problems: [...] array alongside version and config, present only when something was dropped. It is absent from a healthy snapshot, which is what every gateway in the fleet transfers on every poll.
  • the gateway logs them once per change, at the point it applies a new version, rather than once per poll — a fleet polling every 5s would otherwise repeat the same complaint thousands of times a day. An older control plane sends no such field, so it defaults rather than failing the decode during a rollout.
  • the dashboard reads GET /api/v1/config/problems and shows them on the Providers screen. That endpoint runs the same computation the snapshot does, so the two cannot disagree; it also reports structural problems, which never reach a gateway at all.
  • that endpoint also appends ConfigStore::load_problems: rows the store’s loader could only map onto the config by guessing, such as a budget whose stored period it does not recognise and enforces as monthly (#1902). The snapshot carries the guess rather than the row, so only the store can say where one was made. Those lines are not in the snapshot’s problems, because nothing was dropped: the gateway serves the row, just not as the row says.
• GET /api/v1/config is the dashboard’s read-only view of the same document and sits on the open router, so it answers without a session. It is therefore redacted before it leaves the control plane (redact_config_for_dashboard): provider credentials, the plaintext of [[virtual_keys]], the digests of database keys and every MCP OAuth session are removed, and userinfo is stripped from the ClickHouse and egress-proxy URLs. Anything a gateway needs to authenticate with comes only through the token-guarded /internal/snapshot (#1212).

Why this design

• Redis pub/sub gives near-instant fan-out to many gateway replicas.
• Postgres versioning makes the system correct even if Redis drops a message.
• ArcSwap keeps the hot path lock-free; no read ever blocks on a config write.

Alternatives considered: Postgres LISTEN/NOTIFY (avoids a Redis dependency but Redis is already needed for cache/rate limits), and pure polling (simplest, higher latency). See ADR.

Config schema migrations

How a rolter.toml written for one release is carried across a version boundary, and why the mechanism is shaped the way it is (#326).

The operator-facing side of this is Upgrading Rolter. This page is the design.

The shape

crates/rolter-core/src/config_migrate.rs holds a chain of steps keyed by schema version:

#![allow(unused)]
fn main() {
pub static MIGRATIONS: &[Migration] = &[Migration {
    from: 1,
    to: 2,
    summary: "...",
    apply: tier_provider_sections,
}];
}

migrate() reads the file’s schema_version, runs every step whose from is at or above it, and stamps the result. N -> N+k is therefore the same code path as N -> N+1 — a fold over the chain — which is what makes skipping releases free rather than a separate feature.

GatewayConfig::from_toml_str calls it on every load, ahead of the tiered-section extraction, so a migration can still see providers and provider_groups as part of the document.

Three rules, each of which cost something to learn

A migration is behaviour-preserving or it is not a migration

The GatewayConfig parsed from the migrated document must equal the one parsed from the original. migration_is_behaviour_preserving pins this.

This is what makes it safe to migrate on every load rather than as a separate operator-invoked step. The moment a migration can change behaviour, running it implicitly at boot becomes the wrong design and the whole thing needs an apply/confirm cycle.

A file from the future loads, it does not fail

config_lint’s module docs already state the promise the config types make: no deny_unknown_fields, because a file written for X.y.z has to stay loadable by every other X.* build, including an older one it gets rolled back onto.

A version stamp is the obvious place to break that promise, and breaking it would be worst exactly when it hurts most — a rollback is what an operator reaches for when something has already gone wrong. So a schema_version newer than the build understands sets Report::ahead, logs a warning, and loads the document unchanged.

Migrate in memory, never in place

migrate() rewrites the parsed document. Nothing writes the file back.

A loader that edits its own input is a surprise in a crash-looping container, where the operator is least able to absorb one, and the config mount may be read-only anyway. Emitting a canonical rolter.toml is rolter config export’s job (and #1513 is making its output current).

What is deliberately not a migration

api_key / api_key_env / egress_proxy have plural successors and look like the obvious first migration. They are not one: rolter-control’s seed, snapshot and export paths read the singular fields directly rather than through resolve_api_key() and egress_proxy_pool(), so normalising the document would silently change what rolter-seed --import stores.

That is #1514. Until every reader goes through the accessor, these stay deserialize-time shims — which is how this repository has handled config evolution so far, and the reason the migration chain starts as short as it does.

Adding a step

1. Append a Migration to MIGRATIONS with from equal to the current CURRENT_SCHEMA_VERSION and to one higher.
2. Bump CURRENT_SCHEMA_VERSION. the_chain_is_contiguous_and_reaches_the_current_version fails if you forget, rather than the step silently never running.
3. Add a fixture in the old spelling and assert it parses identically to the new one.
4. Keep the step idempotent — migrating_twice_changes_nothing_the_second_time runs it against an already-migrated document.

If your step renames a key, config_lint needs to change with it

No current step consumes a key, so the lint and the chain do not interact. The first rename breaks that: the lint parses the file as written and would report the old spelling as an unrecognised key, telling an operator to fix something rolter already handled.

Do not fix this by linting the migrated document. The paths the lint prints are how an operator finds the line in their own file, and migrating first rewrites them into paths the file does not contain — providers[0].x becomes providers.readonly[0].x, which greps to nothing. Have the migration declare the key paths it consumes, and have the lint skip exactly those.

Previewing

rolter check --migrations renders the plan without writing anything, and adds a non-fatal finding so a CI job running --strict can notice a config drifting behind. It is opt-in because an operator who has never stamped their file would otherwise meet the warning on every run of a perfectly working deployment, and a warning everyone learns to skip past is worse than no warning.

Not covered here

The control-plane-to-gateway snapshot carries config_version, a change counter, not a format version — so a mixed-version fleet during a rolling upgrade has no negotiation. That is #1512, with its own ADR.

Data model

PostgreSQL is the source of truth. The initial schema lives in migrations/0001_init.sql; ClickHouse log schema in clickhouse/001_logs.sql.

Tenancy

erDiagram
  orgs ||--o{ teams : has
  teams ||--o{ projects : has
  projects ||--o{ routes : has
  routes ||--o{ route_targets : has
  projects ||--o{ virtual_keys : has
  orgs ||--o{ providers : has
  providers ||--o{ provider_keys : has
  users ||--o{ memberships : has
  orgs ||--o{ mcp_servers : registers
  users ||--o{ mcp_oauth_grants : consents
  mcp_servers ||--o{ mcp_oauth_grants : authorizes
  orgs ||--o{ mcp_tool_groups : governs
  orgs ||--o| mcp_gateway_settings : configures
  mcp_oauth_grants ||--o{ mcp_oauth_sessions : issues

• Org → Team → Project → Virtual Key is the hierarchy. Budgets and rate limits attach at any scope and combine most-restrictive-wins.
• Providers are owned at the org level and referenced by route targets. Upstream credentials live in provider_keys, envelope-encrypted (see security.md).
• Routes belong to a project and map a public model to route_targets with a strategy.
• Virtual keys belong to a project, store only a hash of the key plus a display prefix, and carry an optional model allow-list.
• MCP servers belong to an org and declare required OAuth scopes and exposed tool names. Only enabled servers reach gateway snapshots. Grants bind a user to a server; sealed token sessions belong to a grant. MCP tool groups persist named server/tool manifests, while MCP gateway settings hold organization defaults for registration and MCP-aware clients; neither is currently a request-path authorization boundary.
• Provider groups (provider_groups, provider_group_members) are org-scoped and address a fleet of same-kind providers as one group-slug/model name. The slug shares the provider slug namespace; a member’s weight and position feed the group’s balancing strategy, and a null upstream_model forwards the requested model unchanged.
• Labels (labels) attach a (key, value?) fact to a provider, provider group, route or model. source separates an operator’s custom label from an auto one Rolter derived, and is part of the uniqueness constraint so the two never overwrite each other; auto rows additionally carry observed_at/observation. subject_id is text because a model is addressed by name, which rules out a foreign key — three after delete triggers sweep a subject’s labels instead. See labels.md.

Cost & limits

• model_prices — USD per million tokens for input/output (+ cached input). Used to compute cost_usd per request, written to ClickHouse.
• budgets — spend caps per scope and period; enforced before forwarding and refreshed from spend aggregates. unpriced_policy is an optional per-budget override of the deployment-wide unpriced-traffic policy (ignore | warn | block, null inherits); the gateway resolves most-restrictive-wins across the scope chain, so a budget can tighten it but never loosen it (#996). period is free text with no check constraint. The snapshot loader reads it through BudgetPeriod::parse, which knows the spellings in BudgetPeriod::SPELLINGS (daily/1d/24h, monthly/30d, total/lifetime/all), and the control plane refuses anything else on create and edit (#1902). A row written before that check can still hold another value, 7d being the one the dashboard used to suggest. The loader keeps enforcing it as monthly, since failing the snapshot would withhold every tenant’s config over one row and moving a budget onto another window is not a guess to make for the operator, and ConfigStore::load_problems reports it so GET /api/v1/config/problems can say so. No migration rewrites those rows for the same reason. The dashboard keeps its own copy of the spelling table in ui/src/lib/budget-period.ts, which names the window on a card and offers the picker’s options; budget-period.test.ts reads BudgetPeriod::SPELLINGS out of config.rs and fails when the two differ.
• rate_limits — RPM/TPM per scope; counters live in Redis for multi-instance correctness.

Both cap tables are edited in place (#1285). PATCH /api/v1/budgets/{id} and PATCH /api/v1/rate-limits/{id} go through BudgetRepo::update and RateLimitRepo::update. Each locks the row with select … for update, works out whether the edit moves anything, and only then issues a single update statement in the same transaction, so the statement-level trigger from 0004_config_version_pricing_limits.sql bumps config_version once per edit. An edit that moves nothing issues no statement at all: a statement-level trigger fires on an update that matches no row too, so a no-op filtered out in the where clause would still bump and wake every gateway. limit_usd is compared as numeric(12,4), so 500.0 against a stored 500.0000 is no change. The rate-limit rule that a row keeps at least one positive cap is checked against the locked row as well, because two concurrent edits each lifting a different cap would both pass a check made on an earlier read. Both methods hand back the row before and after (Edited<T>), and the audit changes diff is built from that pair. Deleting and recreating a row instead would bump twice with no cap in between, and a gateway that polled in that gap would load a scope with nothing on it. The row keeps its id and created_at, and the scope is not editable. Redis spend counters are keyed by scope, scope id and period window rather than by budget row (rolter:budget:<scope>:<id>:<window>), so raising or lowering a limit keeps the spend already counted this period, while a changed period reads the new window’s counter. The handlers also publish the new version on CONFIG_CHANNEL, the same as every other snapshot input, so a Redis-subscribed gateway refetches at once instead of at its next poll.

Config versioning

config_version holds a single monotonic counter the gateways watch for reload-free updates (config-and-hot-reload.md). audit_log records who changed what.

The RBAC tables are split on exactly this question. access_profile_policies, access_profile_assignments, access_profiles and memberships all carry a bump_config_version() trigger, because the gateway resolves the model/route policy of each virtual key’s owner from them and enforces it on the request path (#791, ADR-0023). custom_roles and custom_role_grants do not and must not: they decide control-plane authorization, which is evaluated live per request, so a bump would wake the fleet for a change it cannot observe.

Since #1841 the snapshot also decides from users and access_profile_roles whether a personal virtual key (one whose created_by names the person who minted it for themselves) is served: its creator must be active and still hold a role that reaches the key’s project. Migration 0075 gives access_profile_roles a bump_config_version() trigger, and users one that fires only on deactivated_at and is_superadmin. It also adds a before delete trigger on users that disables the account’s personal keys: virtual_keys.created_by is on delete set null (0016), so without it deleting a leaver would turn their keys into ownerless shared keys. See RBAC & authentication.

Routes, providers and provider groups reach the snapshot with the org that owns them (and a route with its project) as tenancy, which the gateway uses to keep each org’s keys on its own rows (#1844). The columns already existed — providers.org_id, provider_groups.org_id, routes.project_id — so this took no migration; see RBAC & authentication.

provider_groups and provider_group_members carry one too. The data plane indexes config.provider_groups to resolve group-slug/model addressing, so a group is data-plane-visible the moment it exists — and until #1643 neither table bumped, which meant a group created in the dashboard was present in /internal/snapshot and still 404’d at the gateway until it was restarted or an unrelated config write happened to bump the counter for it.

plugin_instances is the control-plane registry for desired request/response middleware configuration. A row belongs to an org and may narrow to a project; slugs are unique within that scope. Endpoint credentials are environment-variable references and config is always a JSON object. The table deliberately has no bump_config_version() trigger until the allocation-light gateway dispatcher in #509 consumes it—an enabled registry row is configuration, not a false claim that middleware is running.

The guardrail registry is deployment-wide:

• guardrail_rules stores the ordered built-in and bounded-regex policy managed by the dashboard. Exactly one of builtin or pattern is present on every row.
• guardrail_providers stores external webhook endpoints and environment-variable credential references. A partial unique index permits at most one enabled provider because the gateway exposes one vendor-neutral webhook contract.

Both tables bump config_version in the write transaction. File-owned rules remain immutable and win name collisions; database rules extend that policy. An enabled file-owned webhook remains authoritative, otherwise the active registry provider supplies the snapshot webhook. The Guardrail Providers screen reports that merged webhook from GET /api/v1/config rather than the registry’s enabled flag (#2162). It tells the two sources apart by comparing the webhook field by field with the enabled row (ui/src/lib/guardrail-enforcement.ts), so a file webhook identical to that row is credited to the row until the API reports the source (#2249). A post_call webhook is reported as not enforcing, since the gateway runs only the pre-call stage. The Guardrail Rules screen reads the merged guardrails section of the same document (#2157): a rule no row accounts for is listed as the config file’s, a row whose name a differing effective rule takes is reported as overridden, and feature_flags.guardrails = false turns every card to not enforced (ui/src/lib/guardrail-policy.ts). It only trusts a document whose guardrails.enabled equals feature_flags.guardrails, which the store guarantees on every load; the default document get_config falls back to when the store fails breaks that, so it is reported as unreadable rather than as an empty policy. A file rule identical to an enabled row is credited to the row, as with the webhook, until #2249 reports the source.

Several deployment-wide settings are stored as singleton tables: one row keyed by id boolean primary key default true check (id), seeded by their own migration, so a read never has to handle “not configured yet”. runtime_policy, compatibility_policy, security_settings, logging_settings, client_settings and model_defaults all follow this shape and all bump config_version in the write transaction. logging_settings also holds ui_events (#1748), the deployment-level opt-out for the dashboard UX stream; the control plane reads it rather than the gateway, and the table’s existing per-statement trigger covers the column.

Two of them landed with the Settings screens (#564):

• client_settings — the base URL the dashboard advertises (advisory; the gateway never reads it), the allowlist of inbound client headers forwarded to the upstream provider, the static headers the gateway injects on every upstream request, and the request-id header. Trace-context headers propagate independently of the allowlist, and injected header values are treated as credential material: they reach the gateway through the snapshot but never the audit log, which records only the names.
• model_defaults — an enabled kill switch plus optional default_model, default_temperature, default_top_p and default_max_tokens. Defaults only ever fill a key the request omitted, so the table can be populated without changing the meaning of any request that was already explicit. default_temperature and default_top_p are double precision, not real: an f32 default serializes into JSON as 0.800000011920929 and that is what would reach the provider.

Data written by the data plane

Most tables flow control plane → gateway. Two flow the other way, written from the channel the gateway already holds and never read back by it:

• cluster_nodes — one row per node, upserted from the snapshot poll (#543).
• adaptive_routing_telemetry — one row per (node, model) holding the newest adaptive-routing sample that node pushed (#751): the decision split, the sanitized policy it runs, and a JSON document of per-target scores and signals. A scoreboard, not history — every report overwrites the row, so the table is the size of the fleet rather than of the traffic.

Neither carries a bump_config_version() trigger, and neither may grow one: the data plane does not consume them, and a version bump on every heartbeat or sample would make routine bookkeeping look like a config change and wake the whole fleet.

ClickHouse tables

PostgreSQL holds configuration; ClickHouse holds the high-volume append-only streams, all partitioned by day with a 90-day TTL and written in batches off the hot path. Schema lives in clickhouse/, applied by the container’s init directory:

• request_logs — one row per proxied request, with cost and token counts.
• provider_health_events — per-target health observations from every signal.
• mcp_tool_call_logs — MCP tool invocations.

Every one of these tables declares ts DateTime64(3) default now64(3), but that default is a fallback for old writers only. Because the writers batch, a row that relies on it is stamped with the flush time rather than its own, so each writer sends ts explicitly: request_logs and its request_payloads row use the instant the request began, provider_health_events the instant the observation was made, and mcp_tool_call_logs the ts the submitter reports for the tool call (falling back to ingest time). See Observability for what each instant means; rows written before #1210 carry flush time.

• ui_events — dashboard UX events (#805): screen views and time-to-interactive, navigation and back-outs, form submit/abandon and which validation rules fire, empty- and error-state impressions, save-to-confirmation latency. Carries trace_id, so a UX event and the gateway request it caused are one join apart. ts is the instant the interaction happened, stamped by the browser as the event is queued rather than when the batch reaches the control plane (#1224): the dashboard flushes on a timer and at unload, so ingest time collapsed a whole burst onto one instant and pushed an event minutes away from the request it joins on trace_id. Because that instant comes from a clock the server does not own, it is believed only within a window around ingest time — more than five minutes ahead or more than a day behind and the ingest instant is recorded instead. The window is deliberately not a clamp: pinning a skewed clock to the boundary would manufacture a cluster of rows exactly five minutes out, which reads as real traffic. Rows written before #1224, and any client that sends no ts, carry ingest time.

ui_events is structurally incapable of holding content: every column is a key, an enum, a duration or an id, so “no form values, no prompt text” is a property of the schema rather than a policy applied on write. None of these tables is read by the data plane, so none carries a bump_config_version() trigger.

Migrations are append-only

Migrations are embedded with sqlx::migrate! and sqlx stores a checksum of every migration it applies. If a shipped file’s bytes change — even a stripped trailing newline, as a formatting hook once did in #710 — every database that already ran it refuses to start on its next upgrade:

Error: store error: migration 18 was previously applied but has been modified

CI databases always start empty, so nothing else catches this: the change merges green and breaks deployments later. Never edit, delete or rename a file under crates/rolter-store/migrations/; add a new NNNN_*.sql instead. Numbers are append-only and never reused, even where the sequence has a gap.

scripts/check-migrations-immutable.sh enforces this. It runs as the migrations append-only step of the static checks job in quality.yml (inside the ci-ok gate) and as a prek hook locally, rejecting any modified, deleted or renamed migration relative to the branch’s fork point from master.

Mapping to the gateway

The control plane composes the normalized tables into the same shape as rolter_core::GatewayConfig (providers, routes, virtual keys and authorized MCP sessions), which the gateway turns into an immutable Snapshot.

Labels

A label is a short, visible fact attached to a provider, a provider group, a route or a model. Before #985 each such fact was built as a bespoke badge on one dashboard screen, which meant it existed nowhere else, could not be filtered on, and could not be added by an operator for their own purposes. This is the one primitive those facts are built on instead.

Two kinds, one table

They share a table because they are rendered and filtered together, and a screen that showed only one of them would be answering half the question. They do not share a write path: LabelRepo::create_custom and its siblings only ever touch source = 'custom', and upsert_auto / clear_auto only ever touch source = 'auto'. Nothing in the repository can turn one into the other.

Two consequences fall out of that, and both are deliberate:

• An operator cannot forge an observation. The labels_provenance check constraint requires an auto row to carry observed_at and forbids a custom row from carrying it at all, and no HTTP handler writes auto. A hand-written label can look like anything except a probed verdict.
• An auto label never silently overwrites a custom one. source is part of the uniqueness constraint, so (provider X, key "priced") can hold one of each. They are displayed side by side and marked; neither wins.

Editing or deleting an auto label answers 404, not 403. The where source = 'custom' clause on the update and delete statements is what enforces it, so there is no path — not even an internal one reachable from a handler — that edits an observation.

An auto label is a measurement, not a guarantee

observed_at is the point at which the fact was established and observation is what established it. Both are shown to operators. An auto label says this was true when we looked; it can be stale, and the UI must never present one as a promise about the next request.

The first producer is the pricing catalog: writing a model_prices row records priced on that model with the currency as its value, and deleting the row withdraws the label. It is written inside the same transaction as the price, because the moment the fact becomes true is that statement — a background job would only have to guess how stale it was allowed to be. New producers should follow the same rule: record the observation where the observed thing changes.

Subjects

subject_type is one of provider, provider_group, route or model, and subject_id is text rather than a uuid because a model is addressed by name while the other three are addressed by id.

That rules out a foreign key, so three after delete triggers sweep a subject’s labels when the subject goes. Without them a label would outlive the row it describes and — worse — be inherited by the next row issued that id.

Tenancy comes from the subject too: the table has no org_id, so LabelRepo::list_in_org joins back through providers, provider_groups and routes to decide what an org may see, and a write checks subject_in_org before it inserts. A label id belonging to another org answers 404 through this org’s path.

Scopes and the two API surfaces

The split follows where the subjects live. Providers, groups and routes belong to an org; models belong to the deployment-wide pricing catalog and to no org, so labelling one is a deployment-wide act and model_label mirrors model_price exactly.

label is its own capability rather than the subject’s, so an operator role can be allowed to annotate a provider without being allowed to re-point it.

Display and filter only, for now

This is why labels carry the experimental stability marker: the day a route selects on a label, a label stops being an annotation and becomes configuration the gateway consumes, which is a change of shape rather than an addition. See Stability markers for what that exempts labels from.

Labels are not read by the data plane. A route cannot yet select on one — a route that targeted eu-only would turn labels into config the gateway consumes, which is a materially larger change than displaying them.

The schema is shaped so that change needs no rewrite, and the labels_bump_config_version statement trigger is already in place. Bumping config_version on a label write costs a snapshot fetch today and buys the guarantee that the day a route does select on a label, propagation already works rather than silently serving stale config (see Config & hot reload).

RBAC & authentication

Two distinct auth surfaces:

1. Gateway (data plane) — virtual keys

Clients call /v1/* with a virtual key (Authorization: Bearer <key> or x-api-key). The gateway:

• looks the key up in the current snapshot
• checks the key is neither disabled nor past its expires_at
• checks the key’s model allow-list (empty = all)
• once the model resolves to a route, applies the route authorization contract below
• (roadmap) enforces budgets and RPM/TPM limits for the key’s scope chain

Keys are stored as hashes; the presented key is compared in constant time (rolter_auth::verify_key).

The route authorization contract (#1485)

Access to a route must not depend on how the request body is encoded. Every data-plane endpoint that addresses a model therefore authorizes through the same two functions in crates/rolter-gateway/src/handlers.rs, and no endpoint re-implements a check inline:

Before either gate sees a route, resolution (Snapshot::resolve_for) has already set aside every row outside the key’s tenancy: a route of another org, a route narrowed by advanced.visibility.project_only to a project the key was not minted in, and a provider or group address of another org. Such a row is a miss, so the request goes on to the builtin and the slug/model addresses as if it did not exist, and ends in the 404 model_not_found an unknown model gets (see One org never reaches another).

authorize_route checks, in order:

1. tenancy and visibility (model_visible_to); refused as model_not_allowed. It runs first so a hidden route never reveals whether the other gates would also have refused. The tenancy half (RouteEntry::in_tenancy_of) repeats what resolution did and only matters where a route was reached some other way, such as a complexity tier or a stored response’s route; then advanced.visibility.allowed_team_ids / allowed_key_ids narrow the route to the teams and keys they list. A request with no key at all is refused by any non-empty visibility list.
2. route policy — the access-profile policy of the key’s creator (KeyMeta::route_permitted, #791); refused as route_not_allowed.
3. provider allow-list — at least one target on the route (classic pool or any variant) is a provider the key may use (key_allows_route); refused as provider_not_allowed.

The per-target provider filter in pick_untried still runs during balancing; the route gate only guarantees that at least one target is reachable, so the caller gets an explicit 403 rather than a failed target selection.

The callers are the JSON pipeline (proxy: chat, completions, responses, messages, embeddings, rerank, image generation, audio speech), the multipart pipeline (proxy_multipart: audio transcriptions and translations) and the Realtime upgrade (realtime::realtime), which refuses before the WebSocket handshake completes and before an upstream socket is dialled. The complexity tier selector and GET /v1/models use the same authorize_route, so a tier never upgrades a request onto a route its key could not address directly, and the listing only shows routes the key could call. Before #1485 the multipart path checked only the model allow-list and the Realtime upgrade skipped visibility, so a hidden or policy-denied route was reachable through an audio upload or a socket. The HTTP regressions for every gate on every pipeline live in crates/rolter-gateway/tests/integration.rs (audio_uploads_*, realtime_upgrade_enforces_route_visibility).

visibility.allowed_user_ids and visibility.minimum_role are not enforced on the data plane: a virtual key carries no user identity or role. They remain control-plane concerns.

The Responses lifecycle calls (GET/DELETE /v1/responses/{id}, POST /v1/responses/{id}/cancel, GET /v1/responses/{id}/input_items) address a stored response rather than a model, so they carry no route to resolve. The registry entry records both the model the caller named and the route that served it (ResponseRoute::route: the configured route name, the tier route a complexity policy moved the request onto, or the provider-slug/model address itself). authorize_lifecycle re-runs the contract against the current snapshot on every call (#1779):

1. authorize_model on the recorded model id
2. the recorded route is resolved as the request path would resolve it (named route first, then provider-slug/model addressing), and authorize_route runs on it
3. the key must still allow the one provider holding the response; refused as provider_not_allowed, since the call can only go to that provider

It runs after the tenant-scoped registry lookup, so another key still gets the uniform 404 response_not_found and learns nothing; only the response’s owner can see these 403s. Without it, access revoked after creation (visibility, route policy, a narrowed key) would keep working for the registry TTL, which is 24 hours by default.

A removed route fails closed. When the recorded route no longer resolves, the call is refused with 403 route_not_allowed (“no longer configured”). With the route gone there is nothing left to evaluate visibility and route policy against. Allowing the call would let a deleted route keep serving provider-credentialed calls until the entry expires, and deleting a route is often exactly how an operator withdraws access. The cost is that a response created on a route that was then renamed or removed can no longer be managed through the gateway; recreating the route restores access. HTTP regressions: response_lifecycle_rechecks_route_authorization_after_revocation and response_lifecycle_reauthorizes_pinned_addresses.

A new endpoint that resolves a route must call both gates. A Realtime session also meets the budget, unpriced_policy and rate-limit admission of the HTTP path and is metered per response turn; see Realtime metering. Its key is checked once, at the upgrade, so revoking a key does not yet end a session already open (#1881).

One org never reaches another (#1844, #1845)

Routes, providers and provider groups live in the store under an org, and a route under a project as well. The snapshot carries that as tenancy (rolter_core::Tenancy) on every ModelRoute, ProviderConfig and ProviderGroupConfig it loads from the store; rows from a gateway-only config file carry none. Tenancy::admits decides whether a key may use a row:

Every way of addressing a model resolves through Snapshot::resolve_for, which applies the rule before anything else: a named route; then provider-slug/model, whose synthetic route takes the provider’s tenancy; then group-slug/model, whose synthetic route takes the group’s. A row the key’s org may not use is a miss, not a refusal. Another org’s route named edge/gpt-4o therefore never shadows this org’s own edge provider, and another org’s route named fake-llm never hides the builtin. The key gets the 404 model_not_found a model nobody configured gets, so the answer does not confirm the row exists. GET /v1/models lists providers, routes and groups through the same rule, so a key neither lists nor calls another org’s rows. Before #1844 any key could address any org’s provider by slug and spend that org’s credential.

advanced.visibility.project_only narrows a route further, to keys minted in the project the route lives in, and a key of another project gets the same miss. It is off by default (a route is visible to its whole org), and an admin turns it on as This project under the model’s Access & permissions. A key from the gateway’s config file is not narrowed. It narrows the named route only: the provider or group behind it stays reachable org-wide as provider-slug/model or group-slug/model, so confining a provider to one project takes the key’s providers allow-list (#1919).

The write path keeps the snapshot inside that rule:

• A route target or a provider-group member must name a provider in the route’s (or group’s) own org. Another org’s provider id gets the 404 an unknown id gets, so the refusal does not confirm that the provider exists. The snapshot loader also drops, with a warning, any cross-org target or member written before the guard existed, rather than serve it.
• Route names and provider names are unique across the whole deployment, because the gateway indexes them that way and Config::validate refuses a snapshot holding a duplicate, which would stop config propagation for every org at once.
• Provider slugs and provider-group slugs form one address namespace across the deployment, bootstrap-file rows included, because the gateway resolves provider-slug/model and group-slug/model from it. Config::validate does not check slugs; a second holder would silently take the address instead, and a group whose slug a provider holds would drop out of routing. So a provider slug is refused when any provider or group holds it, and the same for a group slug, on create and on a slug change. The gateway indexes first-wins in file order, bootstrap rows before database rows, so a slug two rows still share from before the guard resolves the same way on every build.
• A route name may contain / (Qwen/Qwen2.5-7B), but not as the address of another org’s or a bootstrap-file provider or group (edge/… while another org holds edge), and a database route may not be named fake-llm. The gateway already treats those routes as absent for other orgs’ keys; the refusal protects keys from a config file, which carry no org and would reach the route instead of the address. It runs one way only: a provider or group created later with the slug of another org’s existing slug/… route is accepted, because refusing it would let any org reserve slugs by naming routes after them. That org’s own keys get their address; org-less keys (and anonymous callers with require_auth off) resolve the named route first and reach the other org’s route. An operator may still replace the builtin: a bootstrap-file route, a [[models.default]] or a rolter-seed --import named fake-llm shadows it.
• Every refusal is a 409 (“… is already in use in this deployment; choose another”) that never says which org holds the name. rolter-seed --import refuses the same collisions, and the startup seeders ([[providers.default]], [[provider_groups.default]], [[models.default]]) skip a colliding default with a warning instead of writing it. Per-org namespaces are #1857.

The dashboard never sees tenancy: redact_config_for_dashboard clears it along with the credentials, and drops the database key records, which name every key’s org, team, project and creator. The unauthenticated GET /api/v1/config still lists every org’s providers, routes and groups; whether it may describe that topology at all is #1840.

One org never reaches another (#1844, #1845)

Routes, providers and provider groups live in the store under an org, and a route under a project as well. The snapshot carries that as tenancy (rolter_core::Tenancy) on every ModelRoute, ProviderConfig and ProviderGroupConfig it loads from the store; rows from a gateway-only config file carry none. Tenancy::admits decides whether a key may use a row:

Every way of addressing a model resolves through Snapshot::resolve_for, which applies the rule before anything else: a named route; then provider-slug/model, whose synthetic route takes the provider’s tenancy; then group-slug/model, whose synthetic route takes the group’s. A row the key’s org may not use is a miss, not a refusal. Another org’s route named edge/gpt-4o therefore never shadows this org’s own edge provider, and another org’s route named fake-llm never hides the builtin. The key gets the 404 model_not_found a model nobody configured gets, so the answer does not confirm the row exists. GET /v1/models lists providers, routes and groups through the same rule, so a key neither lists nor calls another org’s rows. Before #1844 any key could address any org’s provider by slug and spend that org’s credential.

advanced.visibility.project_only narrows a route further, to keys minted in the project the route lives in, and a key of another project gets the same miss. It is off by default (a route is visible to its whole org), and an admin turns it on as This project under the model’s Access & permissions. A key from the gateway’s config file is not narrowed. It narrows the named route only: the provider or group behind it stays reachable org-wide as provider-slug/model or group-slug/model, so confining a provider to one project takes the key’s providers allow-list (#1919).

The write path keeps the snapshot inside that rule:

• A route target or a provider-group member must name a provider in the route’s (or group’s) own org. Another org’s provider id gets the 404 an unknown id gets, so the refusal does not confirm that the provider exists. The snapshot loader also drops, with a warning, any cross-org target or member written before the guard existed, rather than serve it.
• Route names and provider names are unique across the whole deployment, because the gateway indexes them that way and Config::validate refuses a snapshot holding a duplicate, which would stop config propagation for every org at once.
• Provider slugs and provider-group slugs form one address namespace across the deployment, bootstrap-file rows included, because the gateway resolves provider-slug/model and group-slug/model from it. Config::validate does not check slugs; a second holder would silently take the address instead, and a group whose slug a provider holds would drop out of routing. So a provider slug is refused when any provider or group holds it, and the same for a group slug, on create and on a slug change. The gateway indexes first-wins in file order, bootstrap rows before database rows, so a slug two rows still share from before the guard resolves the same way on every build.
• A route name may contain / (Qwen/Qwen2.5-7B), but not as the address of another org’s or a bootstrap-file provider or group (edge/… while another org holds edge), and a database route may not be named fake-llm. The gateway already treats those routes as absent for other orgs’ keys; the refusal protects keys from a config file, which carry no org and would reach the route instead of the address. It runs one way only: a provider or group created later with the slug of another org’s existing slug/… route is accepted, because refusing it would let any org reserve slugs by naming routes after them. That org’s own keys get their address; org-less keys (and anonymous callers with require_auth off) resolve the named route first and reach the other org’s route. An operator may still replace the builtin: a bootstrap-file route, a [[models.default]] or a rolter-seed --import named fake-llm shadows it.
• Every refusal is a 409 (“… is already in use in this deployment; choose another”) that never says which org holds the name. rolter-seed --import refuses the same collisions, and the startup seeders ([[providers.default]], [[provider_groups.default]], [[models.default]]) skip a colliding default with a warning instead of writing it. Per-org namespaces are #1857.

The dashboard never sees tenancy: redact_config_for_dashboard clears it along with the credentials, and drops the database key records, which name every key’s org, team, project and creator. The unauthenticated GET /api/v1/config still lists every org’s providers, routes and groups; whether it may describe that topology at all is #1840.

Empty key sets

An empty effective key set does not mean “auth disabled” on a managed deployment. A gateway started with --snapshot-url (its config comes from the control plane) fails closed: with no virtual keys in the snapshot every /v1/* request gets 401. This keeps revoking the last key a lock-down rather than an accidental opening of the whole data plane.

A gateway running from a static bootstrap config with no keys stays keyless, so local fake-llm development needs no setup.

Key lifetime (#945)

virtual_keys.expires_at has always been read by the data plane — KeyMeta::is_valid refuses a key at or after that instant — but until #945 the mint path never wrote a value, so every key created through the dashboard or the API was immortal. The mint request now carries expires_in_days and the control plane turns it into an instant, rather than accepting one from the caller: a client with a wrong clock cannot mint a key that outlives what the operator chose. Omitting the field is the only way to ask for a key that never expires, and 0 is refused because it reads as “no expiry” while meaning “already expired”.

The same request requires a non-blank name. The plaintext secret is returned exactly once, so an unnamed key cannot be told apart from its siblings afterwards; the rule is enforced in the API, not only in the form, and applies equally to the admin (/api/v1/projects/{id}/virtual-keys) and self-service (/api/v1/me/projects/{id}/virtual-keys) paths.

Rotation replaces a secret without renewing the decision: the fresh key inherits the old one’s expires_at, and its purpose.

Personal keys follow their creator (#1841)

A key minted through the self-service routes (/api/v1/me/...) records its creator in virtual_keys.created_by; a key an admin mints for a project (/api/v1/projects/{id}/virtual-keys) records none. load_virtual_keys serves a personal key only while its creator:

• is active (users.deactivated_at is null), and
• still holds a role that reaches the key’s project — a membership at the project, its team or its org, or an access-profile role there, assigned to them or to a team they belong to — or is a superadmin.

Deactivating a leaver, through SCIM or PUT /api/v1/users/{id}, or removing their last such role therefore takes their keys off every gateway at the next snapshot. Reactivating them brings the same keys back, since nothing about the keys changed. A shared key is the project’s and never depends on who minted it, so offboarding cannot take an application down.

Migration 0075 makes this propagate. The snapshot now reads users and access_profile_roles, so both carry a bump_config_version() trigger (on users, only for deactivated_at and is_superadmin). Deleting the account disables its personal keys in the same statement, through a before delete trigger: created_by is on delete set null, which would otherwise turn them into shared keys, served without the access-profile policy that followed their owner. The audit row for a deactivation, reactivation or deletion records personal_keys, the number of keys it suspended, restored or disabled.

The playground key is scoped by the server

POST /api/v1/me/projects/{id}/playground-key mints a key for the dashboard’s own Playground, and it deliberately takes no request body (#1640). Both of the things a body would carry are the reasons this endpoint exists:

• Reach. The mint paths above take models from the caller, which is right for a key an operator is configuring and wrong for this one: a key the client scopes cannot be a key that “cannot address a model the user could not already address”, because the client can ask for everything. The list is computed here from the routes configured in the project being minted against, and written out explicitly — an empty models array means every model, so a project with no routes is a 400 rather than a key with the widest possible reach.
• Lifetime. expires_in_days has a floor of one day. A playground key lives PLAYGROUND_KEY_TTL_MINUTES (30) and the dashboard asks for a fresh one, so a credential does not outlive the tab holding it.

The row carries purpose = 'playground' so the Keys screen can say what it is rather than leaving a reader to infer it from a short expiry. The list is resolved once, at mint time: a route added afterwards is outside that key’s reach, which makes it a snapshot of what the caller could reach rather than a standing grant.

The dashboard calls this once per project as the Playground opens, and once more per Mint key / Renew key — never in a loop, since a refusal (a routeless project answers 400, no session answers 401) is a state the operator has to act on rather than one a retry can clear. The automatic call waits for my_virtual_key:create from /api/v1/rbac/effective and is not made on an explicit refusal, so a viewer is not sent into a 403 on arrival. The endpoint takes no body, so 400 is the one client error it gives, and the Playground reads that status as “this project routes nothing”: it explains the precondition, links Routing Rules and offers no retry (#2061). With no key and no mint due, the Playground asks GET /gw/v1/models once without a key; a gateway no control plane manages, holding no keys, answers it, and the screen then sends without a key rather than holding back its Send buttons. Both the Virtual Keys screen and the account’s own key list label a purpose = 'playground' row, so a half-hour expiry reads as the design rather than as somebody’s mistake.

The mint answers as soon as the row is written, but a gateway only learns about the key on its next snapshot poll (ROLTER_SNAPSHOT_POLL_SECS, 5 by default), so the Playground’s first GET /gw/v1/models with a fresh key answers 401 (#1853). The dashboard waits that out rather than treating it as a verdict: awaitingMintedKey in ui/src/lib/gateway.ts retries a minted key’s 401 with doubling backoff for about two poll intervals (10 s), and only then falls back to the control plane’s route list. A pasted key gets one attempt, and any status other than 401 falls back at once, since neither is something waiting can fix. The fallback drops the routes /api/v1/config/problems reports as omitted from the snapshot, and nothing is preselected from it until that list has answered, so the chat column cannot open on a route the gateway never received.

Override either default with server.require_auth:

2. Control plane (dashboard) — users + roles

Human users authenticate to the control plane. Two providers ship today: local accounts (argon2id password hashes, rolter_control::auth) and OAuth2/OIDC SSO (rolter_control::sso, authorization-code flow with PKCE, JWKS-verified id tokens, IdP group → role mapping). RBAC roles:

• admin — full control within scope (manage providers, routes, keys, members, budgets)
• member — create/edit routes and keys within scope
• viewer — read-only (dashboards, logs)

Roles are granted via memberships at an org / team / project scope. Permission checks resolve the most specific membership for the target resource.

Sessions are stateful rows (sessions, peppered token digest), so revocation is a delete. Deactivation, deletion, SCIM deprovisioning and a break-glass factor reset remove every session the account holds. A password set through PUT /api/v1/users/{id} does the same, except for the session that sent the request, so a superadmin resetting their own password stays signed in where they did it (SessionRepo::delete_for_user_except, #1936). The user.update audit detail carries password_changed and sessions_revoked.

flowchart LR
  U[User] -->|member of| Scope[org / team / project]
  Scope -->|role| Caps[admin / member / viewer]
  Caps --> Action{allowed?}

The capability table is the only source of truth

CAPABILITIES in crates/rolter-control/src/rbac_matrix.rs records, for every resource, what each of read / create / update / delete takes — a minimum scoped role, superadmin-only, or that the resource has no such action at all. Both the published matrix and the guard read it, so they cannot drift.

A guarded handler names a (resource, action) pair instead of a role:

#![allow(unused)]
fn main() {
authorize(&state, &principal, ScopeChain::org(org_id), cap!("provider", Create)).await?;
// deployment-wide, no scope to hold a role in:
authorize_superadmin(&principal, superadmin_cap!("feature_flags", Update))?;
}

cap! resolves the requirement through a const fn, so naming a resource the table does not define — or an action it marks as unsupported — is a compile error, and superadmin_cap! additionally fails to compile unless the table says the pair is superadmin-only. No handler names a Role; unit tests in rbac_matrix.rs scan every control-plane module to keep it that way, check the module list against src/ so a new file cannot slip past, and assert that every row in the table is claimed by at least one guard.

Two read-only endpoints publish that table, so a dashboard never assembles a permission matrix of its own:

• GET /api/v1/rbac/matrix — every role and, per resource, the minimum role each action takes (or that the action is superadmin-only, or unsupported entirely). Any authenticated caller may read it; it describes rules, not anyone’s access.
• GET /api/v1/rbac/effective?org_id=&team_id=&project_id= — the calling principal’s resolved role at that scope chain and the concrete resource:action pairs they may perform, evaluated from their memberships.

effective is advisory to the client and authoritative only on the server: a caller that ignores it and issues the request anyway gets the same 403. Scope precedence is unchanged — a project-scoped grant authorizes that project, not the whole org.

Read access is a viewer’s and mutations are an admin’s, with three deliberate exceptions:

• deployment-wide policy (feature flags, runtime/compatibility/adaptive policy, logging settings, cluster nodes, security settings, alerting, MCP tool-call logs) has no tenancy scope to be a member of, so it is superadmin-only;
• global account lifecycle — creating an org, editing or deleting a user account, and the model/pricing catalog — reaches across orgs, so it is superadmin-only too, while inviting a user into an org stays an org admin’s;
• a user’s own things — minting a virtual key for yourself takes member (a viewer cannot), and revoking your own MCP OAuth grant or session takes only a viewer membership plus ownership, which the handler checks after the guard.

Listing the pricing catalog (GET /api/v1/model-prices) and the effective model list (GET /api/v1/models) is every authenticated caller’s, with no membership anywhere. That is a third authority alongside a scoped role and superadmin, and the table names it rather than implying a role floor: both are deployment-wide catalogs of upstream capability and list price that carry no tenant’s data, and deployment is not a scope a membership can be held at, so a viewer floor there would have described a bar nobody could clear. GET /api/v1/rbac/matrix reports those cells as authenticated_only. The effective model list is still filtered per caller by the access-profile model policy (#534), so what a caller sees remains theirs alone.

Every cell in the table is now backed by the guard.

Listings answer with what the caller reaches (#1846, #1850)

A list endpoint guarded at its parent scope used to refuse the whole list to anyone without a role at that parent, so a team or project member could not find their own team or project, and a team admin could not see the people in their team. The list endpoints now answer with the rows the caller may read. ScopeFilter (rbac.rs) loads the caller’s memberships and access-profile grants once and decides each row by the rule authorize applies. “A role” below means one that meets the endpoint’s own floor in CAPABILITIES — viewer for every list here except invitations, which take admin:

A caller with no role anywhere inside the org still gets 403, and a plain signed-in account learns no org it does not belong to. GET /api/v1/rbac/matrix?org_id= follows the same rule for the org’s custom roles: the dashboard reads the table to say which role a disabled control needs, so a project member who could not read it saw every refusal explained as “your role does not permit this” and an empty Roles & Permissions screen. Revoking an invitation is authorized at the invitation’s own scope (its project, else its team, else the org), so a team admin can revoke an invitation they could send. GET /api/v1/auth/me returns each membership with scope_org_id and scope_team_id filled in, and the dashboard’s scope switcher uses them to land a project member on their own project instead of the first team of the first org.

A handler that starts from the session (CurrentUser) and then authorizes builds its principal with Principal::for_user, which turns is_superadmin into Principal::Superadmin. Before #1847 the self-service key routes built a plain user principal, so a superadmin with no membership — the operator seeded on a fresh deployment — could not mint a personal key or open the Playground.

Account events in an org’s audit log (#1854)

Account events — sign-ins and failed sign-ins, second-factor changes, break-glass resets, account edits and deletions — belong to a person rather than an org, so they are written with no org. GET /api/v1/orgs/{org_id}/audit-log returns them for the org’s own people: a row with no org is included when its actor, or its target user, holds a role in the org, its teams or its projects. user.delete is written once per org the account belonged to, because its memberships are deleted with it and nothing would tie an org-less row back to those orgs afterwards. Rows no org can claim — a superadmin’s own sign-ins, attempts against an unregistered address — are not readable through the API yet; the deployment-wide read is #1858. See security: who reads account events.

Custom roles and access profiles

The three built-in roles are a floor, not the whole rule set. An org may define custom roles: a base role plus a set of explicit (resource, action) grants drawn from the same CAPABILITIES table the guard reads. A grant can only widen — a custom role never takes away what its base role already allows, so the built-in roles keep behaving exactly as before and nothing has to be migrated.

Custom roles are not assigned to people directly. They are composed into an access profile, which names each role together with the org, team or project it applies at, and the profile is then assigned to users or teams. One profile can therefore say “auditor at the org, deploy admin on this one project” and be reused across an organization instead of re-granted per person.

flowchart LR
  P[Access profile] -->|role @ scope| CR[Custom role]
  CR -->|base| Built[admin / member / viewer]
  CR -->|grants| Pairs["resource:action pairs"]
  P -->|assigned to| Who[user or team]
  P -->|optional| Pol[model / route policy]

Evaluation order inside authorize is unchanged for the common case: memberships resolve first, and the configurable half is consulted only if the built-in answer was “no”. That keeps a plain deployment on exactly the old code path, and makes a custom grant strictly additive.

A profile may also carry a model and route policy — allow and deny lists over the models and routes its holders may reach. Deny wins over allow. Where a user holds several profiles the lists are unioned rather than intersected, since a second profile must never reduce access; one consequence is load-bearing: a profile with no model restriction at all makes the merged allow-list unrestricted, because that profile already permitted everything on its own.

That policy is published on GET /api/v1/rbac/effective as model_policy, and since #791 it is also enforced by the data plane. The bridge is the virtual key: a policy belongs to a person, but a request carries a credential, so the control plane resolves each key owner’s merged policy when it builds /internal/snapshot and publishes it on the key record. The gateway then applies it in KeyMeta::model_permitted and KeyMeta::route_permitted, alongside the key’s own model allow-list.

Both must permit a model, deliberately. The key list is what the key’s creator scoped that credential to; the policy is what an operator decided the person may reach at all. Neither can widen the other, so a key naming a model its owner is denied stays denied.

The shape, the merge rule and the allow/deny matching all live in rolter_core::ModelPolicy, which the control plane, the store and the gateway share — two implementations of “deny wins” free to drift apart would be a security bug.

Enforcement keys on the virtual key’s created_by. A key with no owner — admin-created and config-defined keys — carries no policy, because there is no person whose profiles could apply; restricting those is still the key’s own model list.

Because the gateway now reads them, four of these tables do carry a bump_config_version() trigger: access_profile_policies, access_profile_assignments, access_profiles and memberships (a profile assigned to a team reaches every member, so a membership change alters someone’s effective policy with no profile row changing). Since #1841 access_profile_roles does too, because a profile role is one of the ways a personal key’s creator still reaches the key’s project. custom_roles and custom_role_grants still do not and still must not: they decide control-plane authorization, which is evaluated per request against the live database, so there remains nothing to propagate and a trigger would only wake the fleet for a change it cannot observe. See ADR-0023 for why the policy is resolved at snapshot time rather than when a key is minted.

Changing one is safe by construction:

• deleting a custom role that a profile still references returns 409 rather than silently emptying the profile’s composition; detach it first;
• deleting a profile does cascade its own assignments, since the assignment has no meaning without it;
• every create, update and delete on a role, a profile, an assignment or a policy is written to audit_log with the before/after, because all of them change what real people can do.

GET /api/v1/rbac/matrix?org_id= returns the org’s custom roles alongside the built-in ones, so the dashboard’s matrix is API-backed and updates after any change instead of holding state of its own.

The dashboard edits both halves (#1184). ui/src/pages/Rbac.tsx carries the matrix on one tab and the org’s custom roles on another, with a resource x action grant grid built from the same resources[] the matrix returns — never a list held in the dashboard, so the grid can only offer pairs this build defines. A pair the resource does not have renders as a dash and a superadmin-only pair as a disabled checkbox, and grants naming a resource the build no longer defines ride through an edit untouched, since PUT replaces the grant set wholesale. ui/src/pages/AccessProfiles.tsx reads GET /api/v1/access-profiles/{id} per profile — the only call that answers roles, assignments and policy together — and sends the composition and the policy in the one request that saves the profile.

Endpoints:

• GET/POST /api/v1/orgs/{org_id}/custom-roles, GET/PUT/DELETE /api/v1/custom-roles/{id}
• GET/POST /api/v1/orgs/{org_id}/access-profiles, GET/PUT/DELETE /api/v1/access-profiles/{id}
• GET/POST /api/v1/access-profiles/{id}/assignments, DELETE /api/v1/access-profile-assignments/{id}
• PUT /api/v1/access-profiles/{id}/policy

Identity providers (ROL-35)

Local login and SSO both end at the same place: a verified identity that gets turned into a session and reconciled memberships. That shared shape is rolter_auth::IdentityProvider — an async trait with one method, resolve(Credential) -> Result<Identity, IdentityError> — so a new provider only has to prove who someone is; everything downstream (session issuance, group → role reconciliation, audit logging) is unchanged.

• rolter_control::auth::LocalIdentityProvider verifies a Credential::Password, preserving the original handler’s timing-safety property: every rejection path (unknown account, deactivated, sso-only, password login disabled by org policy, wrong password) still runs exactly one argon2 verification, so response time reveals nothing about which case applied.
• rolter_control::sso::OidcIdentityProvider verifies a Credential::AuthorizationCode, wrapping the existing code-exchange and JWKS id-token verification.
• Concrete providers live in rolter-control (next to the sqlx/reqwest they need), not in rolter-auth, which only defines the trait and stays free of those dependencies.

Roadmap

• LDAP — bind + group mapping for enterprise directories (#241), the next provider to implement IdentityProvider.
• JWT service auth and short-lived tokens.
• Audit log surfaced in the UI.
• Optional constant-time map / pepper for virtual-key lookup hardening.

Invitations

Invitations are how a rolter deployment onboards people without an identity provider. An admin mints a one-time link, the invitee opens it and chooses their own password, and the role the invitation carries is granted on acceptance.

This is deliberately not the same as POST /api/v1/orgs/{org_id}/users, which creates an account with a password the admin picked. That is right for seeding and for service accounts, and wrong for onboarding a colleague: it leaves someone else holding their credential — usually in a chat log.

The flow

1. POST /api/v1/orgs/{org_id}/invitations (admin at the target scope) returns the invitation, the token, and a ready-made accept_url. The token is shown exactly once; only its peppered SHA-256 digest is stored, the same treatment sessions and virtual keys get, so a database dump alone yields no usable link.
2. GET /api/v1/invitations/accept/{token} — unauthenticated preview returning the org name, the invited email, the role and the expiry. Nothing else: a link that leaked should not also leak a directory.
3. POST /api/v1/invitations/accept/{token}/accept with the invitee’s chosen password creates the account, grants the membership, and returns a live session so they land signed in.
4. DELETE /api/v1/invitations/{id} revokes a pending invitation.

Links expire after seven days.

Failing closed

Expired, revoked, already-accepted and simply wrong tokens all return the same 401. The caller learns whether their link works, not which of those it is.

Acceptance is single-use even under a race: the claim is an UPDATE … WHERE accepted_at IS NULL, which either affects one row or tells the loser they lost — before any account or membership is created.

One unaccepted, unrevoked invitation exists per email per org, enforced by a partial unique index (invitations_live_email_idx). The index does not look at expires_at, so an invitation that expired unaccepted still holds its address until it is revoked. InvitationRepo::create is a plain insert, so inviting an address that already holds one fails on that index and answers 500 with the database’s message, not the replacement the migration comment describes (#2324). The dashboard lists expired invitations for this reason, so they can be revoked.

Who may invite, list and revoke

Creating is authorized at the scope the invitation grants: scope_type and scope_id name an org, a team or a project, and the caller needs invitation create there or above, the same admin bar as granting a role directly. invited_by records the caller, and is null for the admin token.

GET /api/v1/orgs/{org_id}/invitations answers every invitation of the org, accepted and revoked ones included. An org admin gets all of them; below the org the answer is filtered to the teams and projects the caller administers (#1850), and a caller who reaches the org through neither gets 403. Revoking is authorized at the invitation’s own scope, and only touches an unaccepted invitation: an accepted one answers 404.

The dashboard

Governance → Users → Invite user (ui/src/pages/Users.tsx) picks the scope with the shared OrgScopePicker and posts scope_type and scope_id. The role field is named for the scope (pages.users.orgRole, teamRole, projectRole) and its options are labelled through shell.roles.*. The password method creates the account through POST /orgs/{org_id}/users, which has no scope, so the picker is locked to the org while it is chosen. The link dialog copies with the shared CopyButton: a clipboard the browser withholds (a plain-http dashboard) is reported as a failed copy, and the dialog stays open with the link.

PendingInvitations, below the users table, keeps the rows with neither accepted_at nor revoked_at, including expired ones (marked), and revokes through ConfirmDialog (invitation-revoke), gated on invitation:delete. The section waits for the capability answer and is absent on an explicit invitation:read refusal or a 403 from the list, so a viewer who may open the screen never meets a forbidden error for it. “Sent by” is the users-list entry for invited_by, since the row carries the id only (#2325).

Existing accounts

Acceptance adopts an account that already exists under the invited email rather than forking a second row for the same person. Someone may already hold a login in another org, or have arrived through SSO first.

• An account with a password keeps it. An invite link is not a password reset.
• An SSO-only account (no password) gains the password it was invited to set.
• A deactivated account cannot be revived by an invitation (403).

Relationship to single sign-on

Invitations and single sign-on co-exist; neither requires the other.

The membership an acceptance grants carries source = 'manual', and an SSO login only ever reconciles source = 'sso' rows. So an invited role survives every later IdP login, while roles that came from IdP groups are recomputed each time. A deployment can run invitations only, SSO only, or both at once.

Configuration

rolter does not send the email itself: it returns the link and lets the operator deliver it however their organization already delivers things. That keeps the control plane free of an SMTP dependency, which matters for air-gapped deployments.

Related

• Single sign-on (OIDC) — the IdP path, and how membership sources keep the two from fighting
• RBAC & auth — roles and scopes
• SCIM provisioning — IdP-driven lifecycle for deployments that want accounts created without anyone clicking a link

Single sign-on (OIDC)

rolter can authenticate operators against an OIDC identity provider — Keycloak, Okta, Entra ID, Auth0, Google Workspace — and turn the groups that provider reports into rolter roles.

Single sign-on is optional and additive. A deployment that never registers a provider behaves exactly as it did before: local accounts, passwords, and operator-granted roles. Nothing about SSO is reachable, and the login screen never mentions it.

The three supported deployments

The dashboard asks GET /api/v1/auth/methods — the one unauthenticated endpoint in this area — and renders whichever of the three it is told. That endpoint returns provider names, slugs and start URLs only; all of which are already visible in the login URL, and none of which are secret.

The flow

Authorization code with PKCE, no implicit grant, no client-side tokens:

1. GET /auth/sso/{slug}/start mints a state, a nonce and a PKCE verifier, stores them in sso_login_states, and redirects to the provider’s authorization_endpoint.
2. The provider redirects back to GET /auth/sso/{slug}/callback.
3. The callback consumes the state row (DELETE … RETURNING), so a replayed code + state pair finds nothing and is refused. States older than ten minutes are treated as absent and swept.
4. The code is exchanged at the token_endpoint with the PKCE verifier and the sealed client secret.
5. The id token is verified against the provider’s JWKS: signature by kid, issuer, audience (client_id), expiry, and the nonce from step 1.

Rules that hold on every path:

• The redirect URI comes from configuration (ROLTER_PUBLIC_URL), never from the request, so an attacker cannot point the callback elsewhere.
• Only asymmetric algorithms are accepted (RS256/384/512, ES256/384, PS256). HS256 and none are rejected — a symmetric id token signed with a value the attacker may know is not evidence of anything.
• The discovery document’s issuer must equal the configured issuer.
• A failed token exchange reports the HTTP status only. The provider’s error body can echo the client secret back, and that must not reach a log.

Groups become memberships

An operator maps an IdP group to a role at an org, team or project scope:

POST /api/v1/sso-providers/{id}/group-mappings
{"group_name": "platform", "role": "admin", "team_id": "…"}

The groups claim is read in every shape providers actually send it — a JSON array, a single string, or a space-separated list — and Keycloak’s leading / is stripped, so operators map the group name they see in the IdP’s own UI. The claim name is configurable per provider (group_claim, default groups).

If no mapping matches, the provider’s default_role applies. If there is no default_role either, the login is refused: SSO authenticates, it does not implicitly authorize.

Manual and SSO grants co-exist

Every membership records where it came from:

• source = 'manual' — an invitation, the admin API, the seed command.
• source = 'sso' — an IdP group mapping.

Each login reconciles only the sso rows inside that provider’s org. So:

• A role an operator granted by hand survives every SSO login, forever.
• Dropping a user from a mapped group revokes the role that group granted, on their next login — including the login that is then refused for having no grants left. That is how deprovisioning through the IdP works without SCIM.
• Another org’s SSO grants are untouched; they belong to that org’s provider.

An account is adopted by verified email: someone invited last month who now arrives through the IdP keeps the same user row, the same virtual keys and the same manual roles. SSO-created accounts get no password — an SSO identity must not silently gain a second, weaker credential.

Org login policy

PUT /api/v1/orgs/{org_id}/auth-policy (org admin) sets two flags:

• allow_password_login — when false, members of this org cannot use the password form.
• allow_sso — when false, callbacks for this org’s providers are refused without deleting the provider rows, so an IdP can be cut off in one request. An account a provider created has no password, so while this is off those members cannot sign in at all, whatever allow_password_login says. The dashboard confirms the change when the org has an enabled provider (#2326).

Two guard rails, both returning 409:

• Both flags off is not a policy, it is an outage.
• Password login cannot be disabled before an enabled provider exists.

And one exemption: a superadmin can always log in with a password, whatever the policy says. A mistyped issuer or an IdP outage would otherwise lock the deployment out with no way back in. That exemption is the reason the flag is safe to turn on at all; keep the superadmin’s password strong and stored somewhere the IdP does not gate.

Configuration

Register the redirect URI "$ROLTER_PUBLIC_URL/auth/sso/{slug}/callback" with the identity provider.

A provider’s slug is in that URI, so it is fixed at creation (PUT does not accept one) and has one rule, ^[a-z0-9][a-z0-9-]{0,62}$: lowercase letters, digits and hyphens, starting with a letter or digit, at most 63 characters. It is enforced in three places that have to agree (#2304):

• the sso_providers_slug_charset check constraint in 0047_sso_providers.sql, which is the rule itself;
• validate_slug() in sso.rs, which create_provider runs before the insert so a bad slug is a 400 that states the rule and not a store error carrying the constraint name. The slug is checked as sent, never trimmed or lowercased;
• ui/src/lib/sso-slug.ts, which the add sheet uses to mark the field, hide the redirect URI preview and block Save. It never rewrites the input, because the slug is registered at the identity provider and the admin has to see exactly what will be saved; it only suggests a corrected value.

sso_slug_outside_the_charset_is_a_400_that_states_the_rule in crates/rolter-control/tests/control_integration.rs asks the store and the endpoint about the same table of slugs and requires the same answer, and sso-slug.test.ts reads the migration and compares the pattern, so a change to one is caught by the others. Widening the rule means a new migration (the applied one is never edited) plus both mirrors.

The dashboard never assembles that URI, or the login URL, from the browser’s origin (#2083). Every provider row the admin API returns carries both, built by redirect_uri() and login_url() in sso.rs (the functions start_login itself uses), so the value on the card is byte for byte the one the flow sends:

{
  "slug": "okta",
  "redirect_uri": "https://rolter.example.com/auth/sso/okta/callback",
  "login_url": "https://rolter.example.com/auth/sso/okta/start"
}

The identity provider wants the redirect URI before the provider exists in rolter, since it issues the client id and secret the add form asks for. So the add sheet previews it from the slug as it is typed, on the base GET /api/v1/public-url returns (public_url.rs, capability public_url, readable by any authenticated caller), with ssoRedirectUri() in ui/src/lib/api.ts mirroring the Rust path. One read serves every keystroke. The endpoint also reports configured: false when ROLTER_PUBLIC_URL is unset, and the screen warns that the default only reaches rolter from a browser on the control plane’s own host. It is deployment-wide rather than SSO-specific so the User Provisioning screen can build its SCIM base URL from the same value (#2079). Both screens read it through usePublicUrl() (ui/src/lib/use-public-url.ts), the one place its query key and options are written, so they share one request.

Testing

Unit and Postgres-gated integration tests drive a stub IdP in-process, which covers rolter’s own logic. Interoperability is a separate question, so the e2e harness runs the same flows against a real Keycloak — genuine discovery document, real JWKS, real login form, real /-prefixed realm groups:

cd integration/e2e && uv run pytest tests/test_sso.py --idp

That suite is nightly and on-demand (.github/workflows/sso-e2e.yml), not part of the per-PR gate.

Related

• RBAC & auth — roles, scopes and how they are enforced
• SCIM provisioning — IdP-driven account lifecycle, which pairs with SSO but is independent of it
• Security — secret handling and the threat model

Two-factor authentication

TOTP (RFC 6238) as a second factor on local-account login (#1078). The operator-facing guide is docs/user-docs/security/two-factor-authentication.mdx; this page is the design and the reasons behind it.

Where the factor is checked

Once, on the login exchange, and nowhere else.

A session is the unit of trust in this control plane (see RBAC & auth): an opaque bearer token backed by a row, revocable immediately, presented on every request. Re-checking a factor per request would mean either keeping the secret somewhere a request path can reach it, or asking the user for six digits every few seconds — and it would buy nothing that revoking the session does not already give.

So POST /api/v1/auth/login does not issue a session when the account has an armed factor. It issues a challenge: a separate short-lived token that authenticates nothing and names only which login is in flight. POST /api/v1/auth/mfa/verify redeems it for a real session.

password ok ──► armed factor?
                 │
                 ├── yes ──► challenge (verify) ──► /auth/mfa/verify ──► session
                 └── no ──► policy requires one, and in force?
                             │
                             ├── no (or still in its grace window) ──► session
                             ├── yes, no ROLTER_KEK ──► 403 mfa_enrolment_required
                             └── yes ──► challenge (enrol) ──► /auth/mfa/enroll
                                                           ──► /auth/mfa/confirm ──► session

The challenge lives in its own table rather than as a flagged sessions row. A challenge grants nothing, and putting it in sessions would mean every future reader of that table had to remember to exclude it — a rule that holds only until someone forgets it.

Enrolment at sign-in

A required_* policy used to refuse a session to an account with no armed factor, and the only enrolment path needed a session, so the member could not fix it and an admin had to relax the policy or run the break-glass reset (#1852). The login exchange now hands such an account an enrolment challenge instead: a row in mfa_challenges with purpose = 'enrol', ten minutes to live, five codes to spend.

What makes it safe to hand out is how little it opens:

• It is not a session. CurrentUser and Principal look tokens up in sessions and virtual_keys, and it is in neither, so as a bearer it is refused everywhere.
• It is presented in the body of exactly two routes, POST /api/v1/auth/mfa/enroll (mint a pending secret) and POST /api/v1/auth/mfa/confirm (prove it). Every challenge read names a purpose, so /auth/mfa/verify does not find an enrolment token and the enrolment routes do not find a step-up token. Without that, a code from a secret the token minted but never armed would redeem the step-up.
• The confirm checks the code, then consumes the token with a delete … returning. Only the request whose delete removed the row goes on to arm the factor, mint the recovery codes and call issue_session, so a double submit cannot mint two batches, the second silently voiding the first.
• The session comes out of issue_session, the same path every other sign-in takes, so it is audited as auth.login, next to auth.mfa_enabled with at_sign_in: true. The password step writes after_lock on the challenge row (mfa_challenges.after_lock), and the redeeming request reads it back, so a sign-in that followed a lockout says so on its auth.login row even though a later request issued the session. The step-up carries it the same way.

Both writes an enrolment makes decide inside the statement rather than in a read before it, for the same reason the replay check does:

• MfaRepo::confirm arms the factor only while the row still holds the secret the code was checked against (secret_nonce, fresh per seal). Otherwise a second live enrolment token for the same account, or a second tab, could replace the pending secret between the check and the arm, and the confirm would arm a secret its caller never saw.
• MfaRepo::begin_enrolment upserts only over a row whose confirmed_at is null. The has_armed_factor check in front of it answers the common case; the condition covers an enrolment racing a confirm, which would otherwise silently disarm the factor that confirm had just armed.

A break-glass reset, a new password set through PUT /users/{id} and a deactivation (API or SCIM) all drop the user’s challenges in flight. An enrolment token is refused while a factor is armed, so without the purge, one minted before a reset would come back to life the moment the reset cleared the factor.

Asking for a secret does not charge the token’s budget; proving one does. The budget is not a guessing defence (whoever holds the token holds the secret) but a bound on how long one token stays useful.

A control plane with no ROLTER_KEK cannot seal a secret, so it keeps the old 403 mfa_enrolment_required. Handing out a challenge that fails one step later would only move the same dead end, and the fix is the operator’s. For the same reason PUT /orgs/{id}/auth-policy refuses a required_* value there with a 409 naming the key: without it nobody can enrol, and every account the policy binds, the admin saving it included, would be refused at its next sign-in.

KEK availability comes from Kek::from_env in production. The integration suite installs one KEK process-wide (#1351) and cannot unset it without racing other tests’ in-flight requests, so ControlState::mfa_without_kek, set only by test_app_without_kek, is how the tests reach these branches.

Both challenge responses carry expires_in next to expires_at, and the dashboard times its prompts from that. Comparing expires_at with the browser clock expired the prompt on arrival for anyone whose laptop clock ran more than the TTL fast, and under required_* that meant never getting in.

The trade is the one every enrol-at-sign-in flow makes: until a member enrols, their password alone gets whoever holds it through enrolment. That is no worse than the policy being off, the grace window below gives members a way to enrol from their own session first, and every enrolment at sign-in is audited. A factor armed by someone else is cleared with break-glass plus a new password; the reset alone leaves the password with whoever used it.

Why SHA-1

RFC 6238 permits SHA-1, SHA-256 and SHA-512, and rolter uses SHA-1. Not because it is the safe default in general, but because it is the only interoperable one here: Google Authenticator ignores the algorithm parameter in the otpauth:// URI and computes SHA-1 regardless, so arming SHA-256 would hand a large share of operators a factor that silently never verifies.

The construction is HMAC rather than a bare digest, so SHA-1’s collision weaknesses do not apply, and the secret is 160 bits of CSPRNG output. The practical attack on a TOTP factor is guessing a six-digit code, which is what the replay and attempt rules below defend.

Replay

A TOTP code is valid for its whole 30-second step, and rolter accepts one step either side to absorb clock skew — so without further defence, one shoulder-surfed code stays usable for up to 90 seconds.

user_totp_factors.last_used_step records the highest step the factor has accepted, and a verification is accepted only for a step strictly greater than it. The comparison is a where clause on the update, not a read followed by a decision in Rust: two logins racing with the same stolen code would both read the same value and both conclude it was fresh, whereas as a conditional update exactly one of them affects a row.

The visible consequence is that the code used to confirm an enrolment is spent, so the first sign-in afterwards needs the next code. That is the rule working. It is called out in the operator docs so it does not read as a fault.

Guessing

Two independent budgets, because they defend different things.

• The password is guarded by login_throttle (#1079), unchanged: a wrong password still costs the attacker its escalating delay and lockout.
• The challenge carries its own attempts counter, capped at three, and the attempt is charged in the same statement that reads the challenge — so a caller cannot buy extra guesses by issuing requests in parallel. Once the budget is spent the challenge is dead to the correct code too, which is what stops “hold one challenge open and grind” from working.

The throttle counter is cleared when the password succeeds, before the step-up. Holding the failed-password lock open across the challenge would make a wrong code look like a wrong password and lock the account on the strength of a mistyped digit.

Storage

The secret is sealed with the deployment KEK — it is a bearer credential, so a database dump alone must not yield one — and is registered in SEALED_COLUMNS so rolter kek verify reports a restore that cannot open it. Recovery codes and challenge tokens are peppered digests, the same construction virtual keys and session tokens use.

There is no model type that serialises the secret. The only path that unseals it is MfaRepo::open_secret, which hands back a non-Serialize struct; MfaRepo::status describes the factor without it, and that is what the API returns.

None of these tables carries a bump_config_version() trigger, and none may grow one: the data plane never reads them, so there is nothing for /internal/snapshot to propagate, and bumping the version on every enrolment would wake the whole gateway fleet for a change it cannot observe. Same reasoning as custom_roles in 0058.

Enforcement

mfa_policy is off / optional / required_superadmin / required_all, and a user’s effective policy is the strictest across their orgs. Taking the first match instead would let a relaxed membership soften a hardened one.

Under a required_* policy the password sign-in never admits an unenrolled account unprotected: it gets the enrolment challenge above, or, on a control plane with no KEK, a distinct mfa_enrolment_required refusal. Telling that user to retype their password would be a lie. Invitation acceptance does not go through this decision yet and issues a session directly (#1935).

mfa_enforce_after lets an org announce the requirement before it applies. SetPolicy reads it as a double option: an explicit null clears it, an absent key keeps the stored value, so a client that predates the field cannot cancel an announced window by re-sending mfa_policy. EffectivePolicy in mfa.rs reduces every org policy that binds the user: strictest wins for the policy, and the earliest start wins for the window, so one hardened org already enforcing is enough and a later window elsewhere cannot postpone it. While every binding requirement is still in its window the password alone signs in, and the session carries mfa_enrol_by so the dashboard can say the date on the way in. required stays true through the window, so removing an armed factor stays refused; the window lets the unenrolled in for a while and never lets the enrolled back out.

Unlike allow_password_login, required_all does not exempt superadmins. The exemption there exists because a broken IdP has no other way back in; here the way back in is rolter mfa reset, which needs no exemption to work.

Break-glass

rolter mfa reset --email ... --reason ... clears the factor and its recovery codes, revokes every live session for the account, drops its challenges in flight, and writes an audit entry carrying the reason. It runs on the host against the database, which is the same privilege level as reading the rows it deletes.

Deliberately not an API endpoint: an endpoint that clears a second factor is a second factor that anyone holding a session can clear.

Sessions go with it because the reason to run it is that the account is in unknown hands; leaving a week-long session alive would leave the factor bypassed regardless.

Not covered

WebAuthn and passkeys. Stronger, and worth their own issue — TOTP is the factor that needs no browser API work and the one every security review asks for.

The dashboard builds on this surface: ui/src/components/SignInEnrolment.tsx is the enrolment step in the sign-in card, loaded lazily so the QR encoder stays out of the chunk every signed-out visitor downloads, and the grace window is the Start requiring it control on the SSO screen’s sign-in policy card.

MCP servers, OAuth grants and sessions

rolter stores the OAuth state behind Model Context Protocol access: which MCP servers an org has registered, which users consented to what, and the token sessions held against those consents. Transport and the authorization-code / on-behalf-of exchange belong to the MCP proxy; this layer owns persistence, listing, revocation and audit, so it is useful to a stdio, SSE, streamable-HTTP or WebSocket implementation alike.

Model

mcp_servers (org-scoped)
   ├── mcp_oauth_grants   one live grant per (server, user)
   │      └── mcp_oauth_sessions   token material, sealed at rest
   ├── mcp_tool_groups    named server/tool policy manifests
   └── mcp_gateway_settings   organization defaults

• Server — name, slug (unique per org), URL, transport and the OAuth scopes every proxied call requires. Deleting one cascades to its grants and sessions: withdrawing the server withdraws access to it.
• Grant — a user’s consent against a server, with the scope set they agreed to. A user holds at most one live grant per server (a partial unique index on revoked_at is null); revoked grants are kept so the audit trail survives. Re-consenting updates the scopes in place rather than accumulating rows.
• Session — the tokens issued under a grant, with expires_at, an optional refresh token and refresh_expires_at.

Token handling

Access and refresh tokens are sealed with AES-256-GCM under the deployment KEK (ROLTER_KEK), the same mechanism as upstream provider credentials — there is deliberately no plaintext column for either. Ciphertext and nonce sit side by side; the KEK never reaches the database.

The store exposes tokens through two credential-only paths. McpOAuthRepo::open_session opens one explicitly selected live session for lifecycle code. The Postgres config store opens only the newest live session per (server, user) whose scopes remain within its live grant and cover the server’s required scopes; those records travel through the token-guarded /internal/snapshot channel already used for decrypted provider credentials. Public API DTOs carry only metadata and a has_refresh_token boolean.

The gateway indexes servers by (org, slug) and sessions by (server, user). A request to /mcp/{server} must authenticate with a database-backed virtual key whose created_by user owns the selected session. The gateway repeats the required-scope check before connecting and replaces the caller’s virtual key with the downstream bearer token. Revocation, expiry and policy writes bump config_version, so snapshot polling removes authorization without a restart.

Who sees what

Every listing is joined through mcp_servers.org_id, so a cross-tenant read is not expressible, not merely filtered out.

The curated catalog

LIBRARY in crates/rolter-control/src/mcp_oauth.rs holds the four reviewed server definitions the dashboard offers under MCP Library: GitHub, Sentry, Notion and Linear. Each entry names the server’s URL, transport, tool manifest and the OAuth scopes an authorization request must ask for.

The manifest matters beyond the card it renders on. create_server admits a source: "library" request only when it reproduces the catalog entry element for element, so what is written here is exactly what an installed server ends up storing — and therefore what the tool tally counts, what a tool group can select, and what allow_unlisted_tools is deciding about. While the lists were empty (#1252) an install produced a server that declared nothing at all.

Two properties are worth stating plainly:

• The lists are a hand-taken snapshot, not discovery. Nothing re-checks them against the live server, so a tool renamed upstream is stale here until someone edits the file. Calling tools/list on the server after install is the real fix and is tracked separately; this is the cheap step that stops the catalog from asserting something false.
• An empty required_scopes can be the accurate answer. Notion’s remote server grants access to the pages the user picks on its consent screen rather than to named scopes, so it asks for none. Every other entry declaring an empty list would be an unfilled row, and a test enforces that distinction.

Endpoints

• GET/POST /api/v1/orgs/{org_id}/mcp-servers, PATCH/DELETE /api/v1/mcp-servers/{id} — viewer reads, admin writes. A server URL must be http(s); the transport must be one of stdio, sse, streamable_http, websocket. PATCH updates registry metadata, enabled state, declared tools and required scopes without destroying grants or sessions.
• GET /api/v1/orgs/{org_id}/mcp/library — curated definitions annotated with whether the slug is installed. Installing one uses the ordinary server create endpoint, so the registry remains the source of truth. See The curated catalog for what an entry declares.
• GET/POST /api/v1/orgs/{org_id}/mcp/tool-groups, PUT/DELETE /api/v1/mcp/tool-groups/{id} — exact server/tool policy manifests. These definitions are not yet enforced by the proxy, which is why mcp_tool_groups carries the experimental stability marker.
• GET/PUT /api/v1/orgs/{org_id}/mcp/settings — organization transport and request defaults. The current HTTP proxy still uses deployment-level transport timeouts, which is why mcp_settings carries the experimental stability marker.
• GET /api/v1/orgs/{org_id}/mcp/grants, DELETE /api/v1/mcp/grants/{id}
• GET /api/v1/orgs/{org_id}/mcp/sessions, DELETE /api/v1/mcp/sessions/{id}
• GET/PUT /api/v1/mcp-servers/{id}/oauth-client — the OAuth client rolter presents to the server’s authorization server. Admin-only in both directions: the row names a third party the tenant has chosen to trust. Only client_id is required: authorize_url and token_url are the fallback for a server that publishes no metadata and must be sent together or not at all, issuer pins the authorization server’s issuer identifier for RFC 9207 validation, and discovery is auto (the default) or manual. The client secret is sealed with the deployment KEK on write and never read back; PUT with an empty secret downgrades a confidential client to a public one. The response also reports the RFC 8707 resource every request for this server will carry, and what the last discovery resolved.
• POST /api/v1/mcp-servers/{id}/oauth/authorize — begin consent. Returns the authorization URL rather than a 302, because the caller is the dashboard over fetch and cannot usefully follow a cross-origin redirect.
• GET /auth/mcp/callback — where the browser returns. Authenticated by the one-shot login state, not by a session bearer token. A browser is redirected on to the dashboard and any other caller gets JSON; see How the callback answers.
• POST /api/v1/mcp/sessions/{id}/refresh — renew a session from its stored refresh token.
• POST /api/v1/mcp/sessions/{id}/exchange — RFC 8693 token exchange for a narrower, downstream session.
• GET/POST/DELETE /mcp/{server_slug}/{path...} — Streamable HTTP/SSE proxy on the gateway, authorized by virtual-key owner, server and required scopes

Revoking a grant revokes every session under it in the same transaction, so consent and tokens can never disagree. Server creation/deletion and both revocations are written to audit_log.

Which specification revision this targets

The flow implements the MCP authorization specification revision draft, as published at modelcontextprotocol.io/specification/draft/basic/authorization and read on 2026-09-08 (#1347). That date is the thing to check first when the next drift is suspected: the specification moves, and #707 shipped a client that was correct against the revision of its day and had gone three MUSTs stale by the time #1347 was filed.

Three client-side requirements are load-bearing, and all three fail closed — a check that does not pass ends the flow rather than logging and continuing.

RFC 9728 — discovery of the authorization server

An MCP server publishes protected resource metadata naming its authorization servers, and a client is required to use it rather than to be told where to go. crates/rolter-control/src/mcp_oauth_discovery.rs walks the specification’s order:

1. an unauthenticated GET of the server URL, reading resource_metadata out of the WWW-Authenticate challenge on a 401;
2. /.well-known/oauth-protected-resource with the server’s path inserted after the suffix (https://h/.well-known/oauth-protected-resource/mcp, not https://h/mcp/.well-known/… — RFC 9728 §3.1 puts it the unusual way round);
3. the same document at the root.

The document must declare the resource it was fetched for (RFC 9728 §3.3), or a server could hand rolter somebody else’s authorization server. Each listed authorization server is then probed for RFC 8414 or OpenID Connect metadata in the required priority order, and the document’s issuer must be identical to the identifier the URL was built from (RFC 8414 §3.3).

Only the interactive POST .../oauth/authorize probes. What it resolves is cached on the mcp_servers row, and the background refresher and the token exchange read that cache, so nothing off a user’s request reaches out to an upstream. The cache write is skipped when the values have not changed: mcp_servers carries a statement-level bump_config_version() trigger, and an unconditional write would wake every gateway on every consent.

Moving a server’s URL invalidates the cache (#1416). The cached endpoints belong to whatever authorization server the old URL’s metadata named, so PATCH /api/v1/mcp-servers/{id} clears all four oauth_discovered_* columns whenever the update actually changes url. Without that, a refresh landing between the edit and the next interactive authorize would post to the previous server’s token endpoint while naming the new canonical URI in resource — it fails closed, since a compliant authorization server refuses the mismatched audience, but it fails for the wrong reason and re-discovery is the right answer. The clearing is part of the same update statement that writes the URL, for the trigger reason above: a second statement would bump config_version twice for one edit, and an edit that leaves url alone clears nothing.

Hand-configured endpoints remain the fallback. authorize_url and token_url are now optional on PUT .../oauth-client, and the preference order is discovery, then the last cached discovery, then what an operator typed. A row configured before #1347 keeps working untouched: discovery is attempted, finds nothing for a server that publishes nothing, and the configured pair is used. "discovery": "manual" pins a server to the configured pair and skips the probe entirely, which is worth setting for a server known to publish no metadata.

Re-pinning the issuer or the discovery mode invalidates the cache the same way (#1432). The cached triple belongs to whatever authorization server the old issuer and discovery choice named, so PUT .../oauth-client clears all four oauth_discovered_* columns whenever the update actually changes oauth_issuer or oauth_discovery — same trigger, same reasoning as the URL case above, and folded into the same statement for the same bump_config_version() reason. Without it, an operator correcting a rotated oauth_issuer while staying on "discovery": "auto" saw no effect on the refresher or the on-behalf-of exchange, which never probe and were still reading the stale cache; only the next interactive POST .../oauth/authorize happened to re-run discovery and overwrite it. An edit that leaves both fields alone — a secret rotation, say — clears nothing.

RFC 8707 — resource indicators

Every authorization request and every token request — code, refresh and exchange alike — carries resource, the canonical URI of the MCP server the token is for. Without it the token is not audience-bound, and a compliant server, which MUST reject a token that was not issued for it, is entitled to refuse every token rolter mints.

The canonical form is computed in exactly one place, ResourceUri::parse:

• scheme and host lowercased (uppercase is accepted on the way in);
• no fragment — RFC 8707 §2 forbids one outright;
• no trailing slash, the form the specification asks implementations to settle on;
• the query preserved, since RFC 8707 allows one where it is what scopes the resource;
• userinfo refused, since a credential has no business in an authorization URL.

The two requests cannot disagree about it, and that is structural rather than a convention: ResourceUri is the only way to name a resource in the crate, its only constructor is that parser, and both authorization_url() and post_token() take one. The value used in the authorization request is recorded on the login state and re-parsed at the callback — parsing is idempotent, so the token request carries the identical string. post_token() then refuses outright to send a form with no resource, which is the guard that catches a future refactor rather than a present bug.

RFC 9207 — issuer validation

Before the browser leaves, the issuer of the validated authorization-server metadata is recorded on mcp_oauth_login_states beside the sealed PKCE verifier, together with whether that metadata advertised authorization_response_iss_parameter_supported. The callback applies the RFC 9207 §2.4 table before the authorization code goes anywhere:

The comparison is byte equality. No case folding, no default-port elision, no trailing slash, no percent-decoding — reaching for a URL parser here would re-introduce exactly the equivalences a mix-up attack needs. A response that fails the check is refused whole: its error, error_description and error_uri are not acted on or displayed either, which is why the callback resolves the login state and validates the issuer before it reads anything else in the response.

Which line applies is visible over the API: GET /api/v1/mcp-servers/{id}/oauth-client returns discovered_iss_supported beside the rest of the discovery cache (#1569). Without it a client reading only that endpoint can see what was discovered but not how a callback with no iss will be judged, which is the difference between “proceed” and “reject”. Note the naming: the row and the server list use oauth_discovered_*, the client view drops the prefix.

A row with no recorded issuer at all — hand-configured, no metadata, no pinned issuer — lands on the last line of the table and keeps working exactly as it did. But an iss that arrives for such a row is refused, because there is nothing authentic to compare it against and accepting it would be validation in name only. The fix an operator is told about is to pin issuer on the OAuth client, or to let discovery resolve one.

Fetching from the control plane, and SSRF

Discovery means the control plane now fetches URLs it did not fetch before: one derived from the operator-configured server URL, and — the new exposure — the authorization_servers and endpoints of a document a third-party MCP server served. An operator registering an MCP server is already trusted to point the control plane at a token endpoint, so this widens an existing trust rather than creating one, but it widens it to a value the upstream chooses. Three guards bound it, and they are the reason this is acceptable rather than merely small:

• every URL is https, with http allowed only on loopback;
• every URL goes through the deployment’s egress policy, which denies link-local (instance metadata) by default and private and loopback ranges when configured to;
• discovery uses its own HTTP client with redirects disabled, so a host that passed the egress check cannot hand the request to one that would not have, and with a 5-second timeout and a 64 KiB body cap.

What remains uncovered is what the egress policy itself does not cover: a hostname that resolves to a denied address, since the policy classifies IP literals and does not resolve DNS. That is the same connect-time gap every other upstream in rolter has, and it is not made worse here.

The session lifecycle

Consent runs as an ordinary authorization-code flow with PKCE. POST .../oauth/authorize mints a verifier, seals it into a one-shot mcp_oauth_login_states row keyed by state, and returns the authorization URL. The callback consumes that row — a replayed state finds nothing and fails — verifies the code against the sealed verifier, then writes the grant and its first session in one transaction.

A background refresher sweeps every 60 seconds and renews up to 100 sessions per pass, 5 minutes before expiry, so the skew between rolter’s clock and the authorization server’s plus one round trip is always covered. It handles refresh-token rotation by replacing the stored refresh material whenever the response carries a new one. A permanently refused refresh revokes the session rather than retrying: a 4xx carrying invalid_grant or invalid_scope is a final answer about this grant — consent was withdrawn or the token was rotated away — and a retry loop would only hammer the upstream. Transient failures (network errors, 5xx) leave the session alone for the next sweep.

Token exchange (urn:ietf:params:oauth:grant-type:token-exchange) is the server-to-server half. A service acting for a user gets its own session row descending from the same grant, so it can be revoked independently without taking the user’s interactive session with it, and its scopes are intersected against the grant’s — an exchange can never widen consent.

How the callback answers

The callback negotiates on Accept (#2166). A request that accepts text/html is a browser: the authorization server sending the user back. It gets a 303 See Other to the dashboard’s Auth Sessions screen, built on the deployment’s public URL like the redirect URI itself, with the outcome in the query string:

• consent=completed&session=<id>&server=<id> when the grant and its first session were written;
• consent=failed&reason=<code> when anything refused, plus &server=<id> once the login state has named the server.

Every other caller, reqwest, curl and fetch included, gets what it always did: ConsentCompleted as JSON, or the usual error body. The response carries Vary: Accept either way, and GET /openapi.json documents both answers.

Only identifiers the caller could already list go in that URL. The authorization code, the state, the tokens and anything the authorization server said stay out: a failure travels as its family, and the dashboard supplies the words. That is also what RFC 9207 asks of a response whose issuer did not check out, which may not be displayed at all. The codes are ConsentFailure in crates/rolter-control/src/mcp_oauth_flow.rs, and they are published values, so one is never renamed:

A browser learns only the family, so the handler logs every failure at info with the code, the server and the full error, which is where an operator finds the detail. ui/src/lib/mcp-consent.test.ts reads the Rust enum and fails when the dashboard’s list of codes drifts from it; a code the dashboard does not know still renders a generic sentence.

In the dashboard

The whole flow is drivable from the SPA (#1194), so an operator never has to reach for curl to register a client:

• MCP Catalog → Configure carries an Authentication section (#1447) that shows only what the selected auth_kind uses: nothing for none, a write-only credential for bearer, a header name plus credential for header, and the OAuth client section below for oauth. It writes through PUT .../auth after the row is saved, and only when the kind, the header name or a typed credential actually moved. That keeps untouched saves out of audit_log, and it keeps a deployment without ROLTER_KEK able to edit a server’s other fields, because the route needs the KEK even for a kind that seals nothing. The header name is checked against the same reserved list as RESERVED_AUTH_HEADERS before sending. Clear stored credential is a confirmed, immediate move to none, because the shape constraint allows no bearer/header row without a credential. Saving a credentialed server as none or oauth asks for the same confirmation. A ROLTER_KEK refusal is shown as a deployment problem next to the control plane’s own message. The dialog also edits the three transport overrides: a blank field is null, and a field left as it was is omitted from the PATCH, so absent and null stay distinct on the wire. The rules live in ui/src/lib/mcp-server-auth.ts with unit tests.
• The OAuth client section (the oauth branch above) has: a discovery-mode picker (auto | manual), client id, a write-only client secret, the authorize/token URL pair, a pinned issuer and the default scopes (#1415). Only the client id is required under auto, where the pair is an optional fallback; manual requires the pair, and half a pair is refused in either mode. discovery and issuer are sent on every PUT — the handler reads an omitted mode as auto, which used to reset a manual server on every re-save. Under auto a read-only panel shows the row’s oauth_discovered_* columns with oauth_discovered_at, or says nothing has been discovered yet, and warns that saving a new server URL, issuer or mode clears it — the same three changes the store clears the cache on. A moved URL is also flagged beside the URL field whatever the auth kind, since the store drops the cache on a URL change even when the panel is not on screen (#1572). Connect is enabled for a client id alone under auto, and needs the pair under manual. The rules live in ui/src/lib/mcp-oauth-client.ts with unit tests. It reads GET .../oauth-client purely for redirect_uri — the callback is deployment-derived and cannot be worked out from the browser’s origin — and writes through PUT .../oauth-client after the server row itself is saved, which is also how a client is registered on a server in the same action that creates it. The secret is never echoed back: a badge says whether one is stored, and a Clear stored secret toggle sends "" to downgrade the client to a public one. The PUT is skipped entirely when nothing in the section changed, so re-saving a server does not fill audit_log with mcp_oauth_client.update entries nobody made.
• MCP Catalog → Connect calls POST .../oauth/authorize and opens the returned URL in a new tab. The dashboard never navigates itself there: the consent screen belongs to a third party, and a blocked pop-up is reported rather than left silent, because the request has already succeeded by then.
• Where consent ends (#2166). The new tab comes back through the callback and lands on Auth Sessions, which reads the outcome from its query string (ui/src/lib/mcp-consent.ts) and states it above the listing, before the listing has loaded: the server by name and the new session highlighted, or the failure in the dashboard’s own words for each reason, with a link back to the catalog. On success it posts one message on the rolter.mcp-consent BroadcastChannel. The catalog tab that started the flow listens and invalidates its mcp-grants and mcp-sessions queries, so the card marks Connected without a reload; a browser without BroadcastChannel falls back to the refetch a query does when its window regains focus. The mark means the signed-in account holds an active grant with an unrevoked, unexpired session under it. It is filtered to that account on purpose, since an admin’s listings carry every member’s rows.
• Auth Sessions → renew calls POST /api/v1/mcp/sessions/{id}/refresh for one row, beside the background sweeper. Only a session that stored a refresh token offers it. A refusal is worth reading rather than retrying, since the control plane has already revoked the session by the time the error arrives.

Static credentials, beside OAuth

OAuth is one of four things mcp_servers.auth_kind can name (#952). The others are none, bearer and header — one deployment-wide credential rather than a token minted per user — and they exist because most hosted MCP servers today issue a long-lived token or an API key rather than running an authorization server.

The schema, not the handler, is what keeps the four coherent. mcp_servers_auth_kind_shape requires bearer and header to hold a sealed credential and forbids one to none and oauth, and requires a header name for header alone. So a row cannot claim to be unauthenticated while holding a secret, and has_credential in the read API cannot contradict what the row says it does. The handler checks the same rules first only so an operator gets a 400 naming the problem instead of a 500 from the constraint.

mcp_servers_auth_header_name_shape restricts the header to an RFC 9110 field name and refuses ten reserved ones. Authorization is the one that matters: without that clause, header mode could present an API key exactly where the bearer path puts a token.

The credential is sealed with the deployment KEK through the same Kek helper as the OAuth client secret, registered in SEALED_COLUMNS, and projected as has_credential — the plaintext has no field on any Serialize type. Writes follow the same three-shape convention the client secret uses: absent leaves the stored value, "" clears it, anything else replaces it. That is what lets an operator move a server from bearer to header without re-typing a secret the API gives them no way to read.

Moving to a kind that carries no credential clears it rather than orphaning it, whatever the caller sent. A sealed secret nothing can use is one rolter kek verify would audit forever, and it is a credential still sitting in a backup for no reason.

A stored credential reaches the data plane through /internal/snapshot (#952). load_mcp_servers unseals it with the KEK and puts the plaintext on McpServerConfig, the same route an OAuth access token already takes — that endpoint is the credential-bearing control-to-data-plane channel, so this adds a secret to it rather than opening a new path for one. McpServerConfig’s Debug redacts the field so formatting a snapshot cannot spill it into a log or a crash dump, and the header form marks the value sensitive before it reaches the wire.

A row whose credential cannot be unsealed — ROLTER_KEK unset or rotated — keeps its other fields and loses only the secret, and the proxy then refuses that server with mcp_credential_unavailable rather than calling it unauthenticated. Dropping the server from the snapshot instead would make a misconfigured KEK look like a server that was never registered.

Per-server transport overrides

connect_timeout_ms, request_timeout_ms and max_retries are nullable on mcp_servers, and the proxy reads them (#952). Null means inherit rather than a copy taken at creation, so the value a server never overrode still moves when the value it inherits does.

What it inherits is the deployment transport timeout, not the org’s mcp_gateway_settings: those org defaults are still not read by the data plane, which is what the mcp_settings stability marker records and what #1404 closes. Until then a per-server override is the only way to give one server its own budget.

reqwest bakes the connect timeout into the client rather than the request, so a per-server connect_timeout_ms means a client per distinct value; they are built once and cached, and there are only ever a handful. max_retries repeats only a send that failed to connect: an MCP call is frequently a tool invocation and is not idempotent, so a request the server may have seen — including one that timed out — is reported rather than replayed.

On the wire the PATCH distinguishes absent from null — leave the override versus drop it — which an Option alone cannot express. serde collapses both to None for an Option<Option<T>>, so the fields deserialize through explicit_null, which returns Some(None) for a present null. Without it “stop overriding” silently means “leave it”.

Not supported, and why

• stdio — removed as a transport in #783, since a hosted control plane cannot dial a local subprocess. The MCP specification agrees from the other side: stdio implementations “SHOULD NOT” use its authorization flow and should “retrieve credentials from the environment”, so there is nothing here to store for one.
• mTLS — a client certificate is an identity, not a string, and there is no certificate store to put one in.
• Dynamic Client Registration (RFC 7591) — the specification now marks it deprecated, retained only for authorization servers that do not support Client ID Metadata Documents. The decision recorded in #1347 is that rolter does not implement it and will not: if client registration is ever automated here it will be Client ID Metadata Documents (draft-ietf-oauth-client-id-metadata-document-00), which the specification now says clients SHOULD support, and DCR would be a second registration path to keep secure for a mechanism its own specification is walking away from. A client_id therefore stays operator-supplied; it is the one part of the client the flow will not discover.

header mode is deliberately outside the specification, which defines only Authorization: Bearer. It is an accommodation for real servers, and both the operator docs and this page say so rather than implying conformance.

SCIM 2.0 provisioning

An identity provider can create, update, deactivate and reconcile rolter accounts and groups over SCIM 2.0 instead of an operator doing it by hand. Users answer who exists; groups, through an operator-written mapping, answer what they may do.

Tokens carry the tenant

There is no org id anywhere in the SCIM path. An operator mints an org-scoped provisioning token, and that token is what resolves the tenant:

POST   /api/v1/orgs/{org_id}/scim-tokens   { "name": "okta" }   # org admin
GET    /api/v1/orgs/{org_id}/scim-tokens
DELETE /api/v1/scim-tokens/{id}

The plaintext token (rolter_scim_…) is returned once, at creation. Only its peppered SHA-256 digest is stored — the same treatment as session tokens — so a database dump alone cannot be replayed, and a lost token is rotated rather than looked up. Revoking takes effect on the very next request; there is no cache to invalidate.

Because the token resolves to exactly one org, an IdP cannot address another tenant’s users: a foreign resource id is a 404, not a redacted row.

The base URL an IdP is pointed at

The resource endpoints below are mounted at the root of the control plane (scim.rs, scim_groups.rs), so the base URL a connector needs is the control plane’s public URL plus /scim/v2. The control plane builds every address it gives an outside caller from ROLTER_PUBLIC_URL, never from the request, and the dashboard can be open under another name than the one the IdP must call, so the User Provisioning screen does not derive the URL from window.location (#2079).

It reads GET /api/v1/public-url (public_url.rs, capability public_url, readable by any signed-in caller) and appends the path with scimBaseUrl() in ui/src/lib/api.ts, which mirrors the mount; ui/src/lib/scim-urls.test.ts pins the pair. The read goes through usePublicUrl(), which the Single Sign-On screen calls too, so both screens share one request and agree on whether the variable is set.

The address is shown twice with a copy button each: above the token list, and in the reveal step beside the token, where an operator holds a secret that is never shown again. With configured: false the control plane’s default http://localhost:4001 stands in for the variable. The screen still shows it and says, under it, that an IdP can reach it only from the control plane’s own host. The address appears only beside a token list that was read, since a caller refused the list or a control plane with no store has no connector to point anywhere. A failed read of the public URL shows a LoadError with a retry instead of a guessed host.

Users

GET    /scim/v2/Users?filter=userName eq "ada@example.com"
POST   /scim/v2/Users
GET    /scim/v2/Users/{id}
PUT    /scim/v2/Users/{id}
PATCH  /scim/v2/Users/{id}
DELETE /scim/v2/Users/{id}

• Filters. Only userName eq "value" is supported — the one shape reconciliation needs. Any other filter is a 400 with scimType: invalidFilter rather than being ignored: silently returning the whole directory reads to an IdP as “no such user”, and it then re-creates the account.
• Idempotence. IdPs retry. Creating an existing userName returns 409 with scimType: uniqueness instead of a second account. If the email already belongs to a local account (invited by an admin, or provisioned elsewhere), that account is adopted rather than failing on the unique email, so provisioning converges.
• PATCH. The active toggle is implemented, in both the path: "active" and bare {"active": false} forms, and with the string "true"/"false" some IdPs send. Any other operation is a 400 — an IdP that gets a success for an operation nothing applied would believe the change landed.
• Deactivation logs the user out. Setting active: false stamps deactivated_at and deletes the account’s live sessions, because an IdP disabling a leaver expects them out now, not merely unable to log in again.
• DELETE deprovisions. The account is deactivated, its sessions dropped, and the org’s SCIM identity mapping removed. The users row itself stays: its memberships and audit trail must outlive any one IdP, and a later POST adopts it again.

No password path

A password attribute in a SCIM body is ignored, not honoured. Provisioned accounts are SSO-shaped: password_hash stays null, and there is no code path through which SCIM can set or read a local credential.

Groups

GET    /scim/v2/Groups?filter=displayName eq "platform"
POST   /scim/v2/Groups
GET    /scim/v2/Groups/{id}
PUT    /scim/v2/Groups/{id}
PATCH  /scim/v2/Groups/{id}
DELETE /scim/v2/Groups/{id}

Same token, same tenancy rule, same ScimError envelope as Users.

• Filters. Only displayName eq "value" is supported, for the same reason userName eq is the only Users filter: answering an unsupported filter with the whole directory reads to an IdP as “no such group”, and it then creates a duplicate.
• Members must already be provisioned here. A members entry has to be a user with a SCIM identity in the token’s org. Anything else — an unparsable id, a local account the IdP never created, another tenant’s user — is a 400 with scimType: invalidValue. A group can therefore never be a side door into an account the Users surface did not make.
• PATCH. The operations IdPs actually send are implemented: add/replace on members (array of {"value": id}, array of bare ids, or a single entry), remove on members both with a value list and in the filtered members[value eq "…"] path form Okta uses, remove on the whole attribute to empty the group, replace on displayName/externalId, and the pathless {"op": "replace", "value": {…}} shape. Anything else is a 400.
• Idempotence. A replayed create is a 409 with scimType: uniqueness. A replayed add of an existing member, or a PUT re-sending the same member list, changes nothing.
• DELETE. The group row goes and its members are reconciled in the same request, so the roles it granted disappear immediately rather than at the next sync. A second DELETE is a 404.

Group→team mapping

A mapping is written by an operator, not by the IdP — the IdP may not decide what its groups are worth:

POST   /api/v1/orgs/{org_id}/scim-group-mappings   # org admin
       { "group_name": "platform", "role": "member", "team_id": "…" }
GET    /api/v1/orgs/{org_id}/scim-group-mappings
DELETE /api/v1/scim-group-mappings/{id}

This is deliberately the SSO group-mapping model, down to the vocabulary: a group_name grants one of admin/member/viewer at the most specific non-null scope id (team_id, project_id, or the org itself when neither is named), and a mapping may only grant inside its own org — a team or project belonging to another tenant is a 400. Two divergent group-mapping models in one control plane would be two things for an operator to learn and two places for a privilege bug to hide.

Mappings key on the group’s displayName, which is what an operator sees in the IdP UI. Renaming a group in the IdP therefore detaches it from a mapping written against the old name; rewrite the mapping, or keep the SCIM display name stable.

A mapping may be written before the IdP has ever mentioned the group. Creating or deleting one reconciles that group’s current members straight away, so a mapping change does not wait for the next sync.

Reconciliation converges

Every group write recomputes the affected users’ memberships from the database, never from the request body. The wanted set is “every mapping of every group this user is currently in”; source = 'scim' rows that are not in it are deleted, missing ones are created. Three consequences:

• A scheduled sync re-sending the same group state is a no-op.
• Dropping a user from a group revokes exactly what that group granted, in the same request.
• A grant an operator made by hand carries source = 'manual' and is never touched — the same rule SSO logins follow, so the enrolment paths can be used side by side. An equivalent manual grant also suppresses creating a duplicate scim row.

DELETE /scim/v2/Users/{id} drops the account from every group in the org before removing the identity, so a deprovisioned account cannot keep a group-granted role.

Roles

A provisioned account is granted a viewer membership at the org it was provisioned into, merely for existing — least privilege on purpose. Anything beyond a read comes from a group mapping an operator wrote.

Audit

scim.user.create, scim.user.update, scim.user.deprovision, scim.group.create, scim.group.update and scim.group.delete are written to audit_log, scoped to the org, with the provisioning token’s id in the detail — the actor is a token, not a human. Operator actions carry the acting operator instead: scim_token.create / scim_token.revoke, and scim_group_mapping.create / scim_group_mapping.delete.

Security

Secret handling

• Upstream provider keys are never stored in plaintext in the database. They are envelope-encrypted with AES-256-GCM: a per-record data key/nonce, wrapped by a master key (KEK) supplied via ROLTER_KEK (env/file). Pluggable backends (HashiCorp Vault, cloud KMS) are a roadmap item.
• In the bootstrap file, prefer api_key_env over inline api_key so secrets stay in the environment, not on disk.
• Virtual keys are stored as hashes with a short display prefix; the raw key is shown once at creation.
• Secrets are never logged. The gateway redacts auth headers from traces.

Transport

• Upstream calls use rustls (no OpenSSL). HTTP/2 keep-alive with connection pooling.
• Optional per-provider egress proxy (egress_proxy, HTTP/HTTPS/SOCKS5) for networks where providers aren’t directly reachable.
• Optional global or per-provider custom CA bundles add private PKI roots to outbound upstream clients while retaining public roots, certificate-chain validation, and hostname verification.
• Terminate TLS at the gateway or a fronting proxy/ingress in production.

Cross-origin policy (CORS)

security_settings.allowed_origins and allowed_headers are enforced by a middleware on the control plane (crates/rolter-control/src/cors.rs). The policy is read from an ArcSwap per request, so an edit through PUT /api/v1/security-settings applies immediately rather than at the next restart.

Only the control plane has one, because it is the only origin a browser talks to: it serves the dashboard, and the Playground reaches the data plane through the /gw/* reverse proxy rather than calling the gateway directly. A gateway CORS layer would govern requests browsers do not make, and would need SecuritySettings in the snapshot to do it.

Behaviour:

• No origins configured is the default, and the same-origin deployment everyone runs. The middleware adds no headers at all and the request path is unchanged — CORS is not needed when the control plane serves both the dashboard and the API.
• Matching is exact against the configured list, modulo case and a trailing slash. There is no wildcard and no suffix match, so evil-dash.example.com and dash.example.com.evil.test do not inherit an entry for dash.example.com. validate_origin already rejects * at write time.
• A denied origin gets a normal response with no CORS headers; the browser enforces the block. Vary: Origin is set on every answer so a shared cache cannot serve one origin’s response to another.
• Preflights are answered by the middleware with 204 and never reach the router, since they carry no credentials.
• authorization, content-type, traceparent, tracestate and x-request-id are always allowed, unioned with whatever is configured. The trace headers are what let the dashboard’s spans join the gateway’s on a split-origin deployment; leaving them to configuration means browser trace propagation silently does not work until someone remembers them.
• x-request-id is exposed via Access-Control-Expose-Headers, so a browser can read the correlation id the API already echoes.

Gateway ingress policy (#1162)

The rest of security_settings — the part that is not CORS — reaches the gateway through /internal/snapshot as GatewayConfig::security (rolter_core::SecurityPolicyConfig), loaded by PostgresConfigStore::load_security_policy. The table’s own trigger bumps config_version, so an edit propagates on the next poll like any other config change, with no restart.

Only the policy columns are selected. The dashboard credential ciphertext and nonce are not named by the query at all — the migration promised snapshots would never carry them, and a query that cannot see a column cannot leak it.

Load-bearing properties, each with a test that fails if it stops holding:

• Every rule can only close, never open — except auth_bypass_routes, which an operator writes out path by path. The default for the whole struct is “no extra rules”, so a store that cannot be read leaves the deployment where it was rather than opening one that was closing.
• server.require_auth in the config file still wins, in both directions. A file is a deliberate local override by whoever runs the process, who may not be the person holding the dashboard.
• Bypass matching is exact. /v1/models does not open /v1/models/gpt-4o, and validate_bypass_route already refuses wildcards and non-/v1 paths at write time. The MCP surface passes a literal /mcp into authenticate rather than its real path, so no bypass entry can ever name it.
• /healthz and /readyz are exempt from required_headers. A probe is the deployment’s own traffic; filtering it takes the gateway down rather than protecting it.
• A refusal names the header and never its value. An operator may well have put a shared secret in a required header, and echoing it would hand it to the one caller who did not know it.

allow_direct_provider_keys is gone from the API and the dashboard. The gateway has no direct-provider-key passthrough, so the column controlled nothing; the store now pins it to its default. A toggle that reads like a security control and does nothing is worse than an absent one, because it converts into a false belief during exactly the review where it matters.

The Security screen (#2103, #2114)

The dashboard’s Security screen (ui/src/pages/Security.tsx) saves every field in one PUT /api/v1/security-settings, and the control plane refuses the whole save when one entry breaks a rule in validate_settings. Three pieces of the screen exist to keep that refusal from being the first the operator hears of it.

• The list rules are mirrored in ui/src/lib/security-lists.ts. Every list is one entry per line, and an entry that does not parse is kept in the field and named with its line, never dropped. The functions accept what validate_settings accepts and refuse what it refuses, no stricter and no looser. The one extra rule is a required header named twice, since the store keeps the pairs as a map and one of the two values would vanish. security-lists.test.ts reads security.rs and fails when the forbidden-character sets or the /v1/ prefix change there, so a rule widened on one side shows up as a red test rather than as a toast.
• ui/src/lib/security-loosening.ts decides which saves ask first. Only three edits loosen: virtual_key_required going on to off, dashboard_auth_enabled going on to off, and a path added to auth_bypass_routes. It compares the draft with what the store held at the last load or save, and a save that only tightens goes out without a dialog (see destructive actions).
• ui/src/lib/gateway-pickup.ts reads the fleet after a save. The write bumps config_version (the table’s trigger), each gateway reports the version it runs on its snapshot poll, and GET /api/v1/cluster/nodes compares the two as converged. The screen asks for the inventory under a key of its own per save, so an answer read before the save is never taken for one after it, and counts the live gateways that converged. With no live gateway on record, or a failed read, it says pickup cannot be confirmed and when a gateway applies it (next poll, ROLTER_SNAPSHOT_POLL_SECS, 5 s by default; at once with Redis pub/sub).

The virtual-key switch is narrower than its label in the gateway: authenticate in crates/rolter-gateway/src/handlers.rs reads virtual_key_required || managed_auth for a gateway with an empty key set, and every gateway that receives the setting through a snapshot is managed, so the switch never changes what one of them does (#2357). The confirmation words the consequence as the documented one (the gateway “decides by how it was started”) and claims no more. dashboard_auth_enabled and the dashboard credential are stored and returned, and no code path in the control plane reads them yet (#2356), so the dashboard-protection row says what the setting asks for and nothing about an effect.

Egress policy (SSRF)

A provider’s api_base decides where the gateway sends traffic, so an admin surface that accepts an arbitrary one turns the proxy into an SSRF primitive aimed at whatever the gateway’s network position can reach.

rolter is built to run self-hosted and air-gapped, so upstreams legitimately live on loopback, RFC1918 and container networks — blanket-denying private destinations would break the core use case. What no legitimate LLM upstream needs is the link-local range, which is where cloud instance metadata lives (169.254.169.254, fe80::/10). That is denied by default; everything else is opt-in:

[egress]
block_link_local = true   # default — cloud instance metadata
block_loopback   = false  # sidecar / single-host deployments
block_private    = false  # on-prem clusters
allow_hosts      = []     # exact-host escape hatch

Enforced in two places: the control plane rejects a denied api_base at write time (400, so it never reaches the database), and snapshot validation re-checks it, so a bootstrap toml can’t smuggle one in either.

Config-time validation classifies IP literals only. Resolving hostnames there would make validation depend on live DNS — reintroducing the “one bad row freezes every gateway” failure mode described in config-and-hot-reload.md — and would be bypassable by DNS rebinding regardless.

A literal is read the way the HTTP client reads it. EgressPolicy::check_url parses with the WHATWG url crate that reqwest uses, because that parser normalizes the host before anything connects: http://2852039166/, http://0xa9fea9fe/, http://169.254.43518/, http://0251.0376.0251.0376/ and http://169.254.169.254./ all dial 169.254.169.254, and a backslash ends the authority, so the host of http://169.254.169.254\@example.com/ is the address. A hand-rolled host split saw none of those as that address, and hyper never calls a resolver for an IP literal, so the connect-time check below did not see them either. deny_reason also classifies an IPv4-mapped IPv6 address (::ffff:169.254.169.254) as the IPv4 address it carries and treats the unspecified address (0.0.0.0, ::) as loopback, since connecting to it reaches the local host. A socks proxy URL keeps its host opaque to the parser, so that host is parsed again as an http host would be. allow_hosts entries are parsed the same way and match the address, so an entry for 169.254.169.254 also covers 0xa9fea9fe.

So the same policy is enforced a third time, at connect time, by a custom resolver on every upstream client: whatever DNS actually returns is classified immediately before the connection is made. That covers the two cases config validation cannot see — a hostname like metadata.internal, and a name that resolved to a public address when it was configured but resolves to 169.254.169.254 at request time. A denied destination surfaces as a policy error naming the host, not an opaque connect failure.

A name that resolves to several addresses keeps the permitted ones: refusing the whole name would take down a legitimate multi-homed upstream. Only a name left with nothing is refused — which is exactly what rebinding to a denied address produces. The resolver reads the policy from a live handle, so a hot reload re-tunes enforcement without discarding pooled connections.

The control plane’s own requests to operator-supplied URLs (the connector test probe, alert channel delivery, MCP OAuth discovery) check the same policy on the stored URL when it is saved and again before each request. Alert delivery and MCP OAuth discovery also refuse redirects, so a 3xx cannot hand the request to a host the check never saw. None of them has a connect-time resolver yet, and the connector probe still follows redirects; both are tracked in #1949.

Control-plane input validation

Every control-plane mutation body is decoded through a SafeJson extractor rather than axum’s Json. Before the body is deserialized into its typed struct, every string in it — nested objects, arrays and object keys included — is screened for control characters, and the request is rejected with a 400 naming the offending field.

The concrete failure this prevents: Postgres text columns cannot store a NUL byte, so a field carrying one failed deep inside the store and surfaced as an unhandled 500 instead of input validation, violating the “bounded error, no unwrap/expect on a request path” invariant. Screening the whole C0/C1 range (and U+007F) also closes the log-injection vector a raw escape or newline would otherwise open in operator-facing logs.

Tab, newline and carriage return stay allowed — multi-line values are legitimate, a PEM CA bundle being the obvious one. Malformed JSON now also comes back in the same OpenAI-style error envelope as every other failure instead of axum’s default rejection body.

Open mode (no admin token)

With no ROLTER_ADMIN_TOKEN set, Principal short-circuits to Superadmin for every request: the management API and /internal/snapshot have no authentication step to fail. This is the zero-credential local-dev shape, and crates/rolter-control/src/open_mode.rs is what keeps it from being anything else. Before either listener is opened, it evaluates “is a token set” against every address about to be bound:

--internal-addr counts as a listener here: an exposed credential channel is no better than an exposed API, so either one alone is enough to refuse.

The decision also rides into the dashboard through window.__ROLTER_CONFIG__ (openMode: true), which renders a persistent banner. Without it the dashboard looks identical whether the control plane is gated or wide open, which is the property that made this dangerous rather than merely permissive (#970).

This is why ROLTER_CONTROL_HOST defaults to 127.0.0.1 rather than 0.0.0.0: containers and clusters set it explicitly, and by then they have a reason to have set a token too.

The published image is one of those containers. Its default command runs rolter easy-up --host 0.0.0.0, since a published port cannot reach a container’s loopback (#1891), and leaves the refusal above in charge: a bare docker run of the image with no token exits rather than serving. easy-up applies the same rule before it writes, seeds or binds anything, to the host its control plane binds (--control-host/ROLTER_CONTROL_HOST, else --host/ROLTER_HOST) and, in database mode, to the gateway’s host as well, because the gateway then serves the management API at /admin/* by proxying to the co-hosted control plane. The bind is a flag on that command, not an image-wide variable, so a rolter-gateway or rolter-control run from the image keeps its own default.

Who reads the request log (#1820)

/api/v1/analytics/* (the request log, usage, spend and attribution rollups) and /api/v1/health/* (provider uptime, MTTR, the failure timeline) are merged onto the part of the router that also serves the probes and /api/v1/config, and until #1820 that made them open: neither resolved a principal, so a caller with no credentials read every tenant’s request logs — captured prompt and completion bodies included — on a deployment whose CRUD API was enforcing RBAC.

Every handler in both modules now takes an AnalyticsAccess (crates/rolter-control/src/analytics_access.rs). It authenticates the way the CRUD API does and turns what the caller holds into a filter each query binds as ClickHouse parameters — never spliced SQL:

Two details carry the design:

• Rows are filtered and bodies are masked in the database. Viewer is the lowest role, so a row is visible when any role reaches it. A body needs the role resolved most-specific-first exactly as rbac::resolve_role does — an org admin who is a viewer on one project reads that project’s rows without its bodies — and raised by custom roles as rbac::custom_base_role does. A withheld body comes back empty with payload_withheld = 1, so the dashboard says “hidden for your role” instead of “payload capture is off”.
• The floors live in the capability matrix. analytics:read (viewer), request_payload:read (member) and provider_health:read (viewer) are rows in CAPABILITIES, and the extractor reads its thresholds from them through cap!, so GET /api/v1/rbac/matrix publishes the rule the filter applies. The per-project exception is project_settings:update (admin), stored in projects.payload_min_role; GET /api/v1/rbac/effective folds it in when the scope it is asked about names a project and the caller holds a role on that project’s own chain, read from the project row rather than from the query string, so naming another org’s project beside one’s own org reads nothing back.

A captured body is joined to its log row on (request_id, ts), never on the id alone. The gateway keeps whatever x-request-id the caller sent, so two tenants’ requests can share an id; an id-only join let a caller log a bodiless request under an id seen on another project’s rows and read that project’s prompt through their own row, which the mask passed because it judges the row, not the body. PayloadLog copies its log row’s ts through the same serializer, so the pair names one request.

The pair is not a perfect key. Two requests that share an x-request-id and were logged in the same millisecond (ts is DateTime64(3)) still get one body between them, which takes a client sending predictable or constant ids. Keying the join on something the caller cannot choose is #1937. The pair also drops bodies written by gateways from v0.1.0 or earlier, which let ClickHouse stamp the log row and the payload row separately: after the control plane is upgraded those bodies stop showing until payload retention removes them. The user docs’ upgrade page says to upgrade gateways first for that reason.

Two limits are known and tracked:

• Provider health matches by name. provider_health_events carries the provider’s display name and no org, and names are unique per org only. A name that another org used and then deleted brings its history (uptime, latency, error kinds, target_id) to whichever org creates it next, until the 90-day TTL drops it. Provider names are unique across the deployment, so two orgs never hold one at the same time. Recording the org in the rows is #1908.
• No database, no dashboard. A control plane with ROLTER_ADMIN_TOKEN and no store has no sessions, so the admin token is the only credential these routes accept, and the dashboard’s e-mail-only sign-in cannot present it. The routes stay closed on purpose; letting the dashboard take the token is #1909.

What keeps this from regressing is every_route_the_spec_does_not_mark_public_refuses_an_anonymous_caller in crates/rolter-control/tests/control_integration.rs: it walks the served /openapi.json with no credentials and a forged bearer and requires a 401 from every GET the document does not mark public. The routes that are open by design (/api/v1/ping, /roles, /provider-kinds, /currency, /config, /config/problems) are marked .public() in openapi.rs for that reason, and crates/rolter-control/tests/analytics_scoping.rs pins the row and body rules against a real ClickHouse.

One org never reaches another (#1844, #1845)

A provider’s credential belongs to the org that stored it, and until #1844 the gateway did not know that: the snapshot carried routes, providers and provider groups with no owner, so a key from any org could call another org’s routes, or address its providers directly as provider-slug/model or through a group as group-slug/model, spending that org’s credential. Two orgs that picked the same provider name also froze config propagation for everyone, because the snapshot refuses duplicate names.

Every row loaded from the store now carries its org (and a route its project) as tenancy, and the gateway resolves a model only among the rows the key’s org may use, on all three address forms and in GET /v1/models. Another org’s row is a miss: it shadows nothing of the caller’s own, and the caller gets the 404 an unknown model gets. On the write path a route target or group member must name a provider in the same org. Names the gateway indexes deployment-wide (route and provider names, and the one slug namespace providers and groups share) are unique across orgs, and so is a route name that would sit on another org’s slug/model address. A collision is refused with a 409 that does not say which org holds the name. The route-name check runs one way: a slug created after another org’s slug/… route is accepted, and keys that carry no org then reach that route rather than the new address (the ADR-0017 addendum explains why). An admin can also narrow a route to its own project (project_only), which narrows that route but not the slug/model address of the provider behind it (#1919). The contract, the table of which keys admit which rows, and the write-time guards are in RBAC & authentication; per-org namespaces, which would let two orgs reuse a name, are #1857.

Control↔data-plane trust boundary

GET /internal/snapshot returns provider api_keys decrypted. That is by necessity — the data plane needs the upstream credential to authenticate to the provider — but it makes the snapshot channel a different trust boundary from the operator-facing management API, and it should be configured as one:

ROLTER_INTERNAL_TOKEN=...            # gates /internal/*, distinct from ROLTER_ADMIN_TOKEN
ROLTER_INTERNAL_ADDR=127.0.0.1:4002  # serves /internal/* on its own socket

With ROLTER_INTERNAL_TOKEN set, the operator admin token no longer opens the snapshot — only the gateway’s own credential does. With ROLTER_INTERNAL_ADDR set, /internal/* is not mounted on the public API router at all, so it is absent from the port the dashboard and management API are served from rather than merely gated on it. Bind it to loopback or a private interface. The gateway sends ROLTER_INTERNAL_TOKEN on snapshot polls, falling back to ROLTER_ADMIN_TOKEN.

Both are optional. Unset, the historical behavior holds — /internal/* shares the public listener and accepts the admin token — and the control plane logs a warning at startup saying so. The tenant-facing CRUD surface never returns a provider key in any configuration.

Two things this deliberately does not do. There is no mTLS between the planes: a shared secret over a private interface is comparable strength when the network path is already trusted, and mTLS adds certificate lifecycle to an air-gapped deployment. And the snapshot still carries plaintext rather than sealed ciphertext: envelope-passing would require the gateway to hold the KEK, moving the master secret onto every data-plane node, which is a worse trade for most deployments. Revisit both if the planes ever cross an untrusted network.

Wire transparency

• Outbound requests to upstream providers carry no rolter-identifying marks: no User-Agent, no added X-*/Via headers, no metadata injected into the JSON body, no marks in SSE framing. The only headers sent are functionally required ones — content-type, the provider’s auth header, and anthropic-version for Anthropic.
• Responses back to clients likewise gain no rolter-added headers.
• This is a tested guarantee: golden wire tests in rolter-proxy capture the raw outbound request head and fail on any unexpected header (see openai_wire_carries_no_rolter_signature).

External PII sanitization (#848)

The gateway can hand request content — and optionally response content — to a self-hosted de-identification service before it reaches a provider. Configured under [pii_sanitizer]; see docs/user-docs/security/pii-sanitizer.mdx for the operator-facing contract.

The design property worth stating precisely: the placeholder→plaintext mapping never enters the gateway process. The sanitizer substitutes deterministic placeholders and returns an opaque restoration token; rolter holds the token and nothing else. There is therefore no log line, metric label, span attribute or cached body from which the original values can be recovered, and no in-memory table for a crash dump to expose.

Consequences that fall out of that choice:

• RestorationTicket is not printable. Its Display renders <restoration token redacted>, and the token is reachable only through token_for(&scope), which returns None unless the caller’s org/team/project/route match the scope the ticket was minted under. A token from one project cannot restore content in another.
• Restoration is opt-out, not opt-in-by-default. RestorationPolicy::Never is the default; CallerAuthorized honours x-rolter-pii-restore only because the policy allows it, never because the caller asked.
• The response leg never requests reversibility. Restoring provider-generated content would return the very data that leg exists to remove.
• A streamed response with an active response leg is refused (pii_streaming_unsupported, HTTP 400), mirroring guardrails.streaming_post_call. Correct streaming restoration would need either a network round trip per chunk — destroying TTFT — or the mapping held in-process, destroying the property above. streaming = "passthrough" waives the response leg instead.
• A malformed sanitizer reply is a failure, unlike the guardrail webhook’s decision parsing which defaults to allow. Defaulting would forward content the gateway believes is sanitized and is not.
• Restore runs after the cache store, so what is cached is what the upstream said. A later hit on the same entry is re-evaluated against its own request’s policy and ticket rather than replaying someone else’s restored plaintext.
• fail_open is the default and is the risk to monitor. It forwards unsanitized content when the service is down. rolter_pii_sanitizer_errors_total rising under fail_open means personal data is reaching providers; deployments where the sanitizer is a compliance control should set fail_closed.

A response-leg failure never fails the request even under fail_closed: the upstream call already happened and was already billed, so refusing to deliver would spend the caller’s money and return nothing. The counter records it.

Streamed responses and post-response controls (#1776)

Output guardrails, post_response plugins and the PII sanitizer’s response leg all judge a whole response body, and a stream reaches the caller before any whole body exists. A setting that the client flips with "stream": true must not switch off a control the operator configured, so each of the three refuses the stream up front with a 400 and param: "stream", before the cache lookup or the upstream call. Each has an explicit way to waive the refusal.

A plugin’s waiver is its own failure mode rather than a new setting. fail_closed already says “never deliver what I have not approved”, and a stream is exactly a response the plugin cannot approve. fail_open already says “deliver when I cannot answer”, which is all a stream permits. The check (handlers.rs, beside post_response_plugin_list) refuses when any applicable plugin is fail closed and names it in the message. Before #1776, every post_response plugin was silently skipped for a stream. Tests: the *_post_response_plugin_*stream* cases in crates/rolter-gateway/tests/integration.rs, including a cached stream refused after a hot reload adds the plugin.

The dashboard states the consequence where the choice is made (#2178). ui/src/pages/Plugins.tsx shows a note in the plugin dialog once the stage is post_response, worded by the selected failure_mode and wired as the description of both pickers, and a post_response card with fail_closed carries a one-line warning. Unlike the guardrail rule screen, which has to read streaming_post_call from the effective config, both follow from the plugin’s own fields. The stories WarnsThatAFailClosedPostResponsePluginRefusesStreams and DialogNoteTracksStageAndFailurePolicy pin it.

Failed-login throttling (#1079)

POST /api/v1/auth/login is unauthenticated and runs one argon2id verification per request. That cost is deliberate against guessing and accidental against denial of service: an attacker who does not care about the password can saturate the control plane by posting junk credentials. crates/rolter-control/src/login_throttle.rs counts rejected attempts and refuses further ones before the password is verified, which is what closes the CPU half rather than only the guessing half.

Four properties are load-bearing:

• The counters key on the submitted address, never on a row that was found. An unregistered address is counted, delayed and locked exactly like a real one, so the throttle does not undo the constant-cost verification in LocalIdentityProvider::resolve by becoming an enumeration oracle.
• A lock is a clock, not a state. It is a TTL nobody clears by hand. A lock an attacker could set would be a denial-of-service primitive against the operator — anyone who knows an email address could park the account. The worst case here is a bounded outage capped by --login-max-lock-secs.
• Two independent subjects. The account budget stops guessing one password; the client-address budget stops spraying one guess across many accounts, which a per-account counter never sees. X-Forwarded-For names the client only when --login-trust-forwarded-for is set, because an unverified forwarded header hands out unlimited budgets and lets one client spend another’s.
• A backend outage degrades to absent, not to locked. A redis error counts as zero failures. The alternative — treating an unreachable counter as a reason to refuse — would turn one redis blip into a fleet-wide lockout.

The same guard covers the invitation preview/accept endpoints, which are the same primitive with a different token.

Rejected attempts, engaged locks and sign-ins that follow a lock are written to audit_log, and rolter_control_login_attempts counts them by outcome (success, invalid, throttled, locked, error) with no account or address label. The audit write is spawned rather than awaited: resolving the org an entry belongs to costs a query that only exists when the account does, and awaiting it would make a registered address measurably slower to reject.

Counters live in redis when one is configured, so every replica shares a budget, and in-process otherwise. That fallback is documented rather than hidden: with N replicas and no redis, an attacker gets N times the allowance, and the control plane says so at startup.

Who reads account events (#1854)

Sign-ins, failed sign-ins, lockouts, second-factor changes, break-glass resets and account edits belong to a person, and a person can belong to several orgs, so these rows are written to audit_log with no org. Until #1854 the only read path, GET /api/v1/orgs/{org_id}/audit-log, filtered on the org column, so none of them reached an API or a screen.

The org read now joins on membership: a row with no org is returned to an org when its actor, or its target user, holds a role in the org, one of its teams or one of its projects. That puts an org member’s failed sign-ins, a second factor being removed and a break-glass reset in front of the people who administer that org, in Governance → Audit Logs, where the auth.* actions are filterable. It never shows one org another org’s people. user.delete is the exception to “written with no org”: it is written once per org the account belonged to, because the account’s memberships are deleted with it and the join would have nothing left to match.

The alternative was to write each account event once per org the person belongs to. It was not taken because it fans one sign-in out into a row per org, and it would leave every row already written unreadable, while the join makes a deployment’s existing history readable as soon as the control plane is upgraded. The cost is that visibility follows current membership: an org sees the account events of the people it has now, including events from before they joined, and stops seeing someone’s once they hold no role in it. Deactivation keeps memberships, so a deactivated leaver stays visible; user.delete, which removes them, is written per org for that reason.

Rows no org can claim are still not readable through the API: the account events of someone with no membership anywhere — above all a superadmin’s own sign-ins — and attempts against an address nobody registered, which are recorded with no actor and otherwise reach only the logs and rolter_control_login_attempts. A deployment-wide read for the superadmin (and the security-auditor role of #1834) is #1858.

Three credentials, one word (#943)

Three unrelated secrets pass through rolter, and each is an “API key” to someone. Keeping them apart is a security property, not a documentation nicety — a provider key pasted into a client is a credential leak that no budget, allow-list or audit row constrains.

The gateway therefore refuses a provider-shaped key with a message that names the mistake (“this looks like an Anthropic provider key…”) rather than a bare invalid api key. provider_key_vendor in crates/rolter-gateway/src/handlers.rs does the shape match, and it never classifies our own sk-rolter- prefix: a revoked or mistyped virtual key is a different problem and gets the plain message. The hint is derived purely from the prefix — no lookup, no timing difference between a known and an unknown key, and the presented secret is never echoed back.

The dashboard follows the same vocabulary: Virtual Keys and My Virtual Keys mint the client credential, the provider sheet says Provider key, and the Playground’s key field says which one it wants. docs/user-docs/security/which-key is the user-facing version of this table.

The Playground does not ask for that key first. Opening it mints a session key through POST /api/v1/me/projects/{id}/playground-key (see RBAC and auth), and the dashboard holds the plaintext in a module variable in ui/src/lib/gateway.ts — never in localStorage, which is where it used to go and where a long-lived production key then sat until somebody cleared it (#944). The screen renders the key’s state, never the secret: a badge, the expiry, and a Renew key button that asks for a fresh key rather than extending the one in hand. The paste field stays for testing one specific key on purpose, and a key pasted there carries no expiry, because the dashboard did not choose one.

Threat model (high level)

• Tenant isolation: virtual keys are scoped to a project and reach only their own org’s routes, providers and provider groups (#1844); model allow-lists prevent access to unconfigured models; cache keys are namespaced to avoid cross-tenant cache poisoning.
• Offboarding: deactivating a person, or removing their last role that reaches a project, takes the keys they minted for themselves off every gateway, and deleting the account disables them (#1841); keys an admin minted for an application are the project’s and stay. See RBAC & authentication.
• Abuse: RPM/TPM rate limits and budgets bound spend and load (roadmap enforcement); failed control-plane sign-ins are throttled per account and per client address (see above).
• AuthZ: control-plane mutations are RBAC-checked and recorded in audit_log.
• Supply chain: cargo deny/advisory scanning in CI is a roadmap item.

Operational guidance

• Always set a strong ROLTER_KEK (e.g. openssl rand -hex 32) and rotate provider keys periodically.
• Run the control plane on a private network; expose only the gateway publicly.
• Back up Postgres; treat the master key as the most sensitive secret. It is not in the dump, so a database backup without it restores into a store nothing can read — rolter kek verify catches that at restore time, and the runbook covers backup, restore and rotation.

LDAP authentication

The third identity provider behind rolter_auth::IdentityProvider, after local password login and OIDC SSO (#241). It is compiled in behind the ldap cargo feature and is off unless configured.

How a login is verified

A user’s distinguished name is rarely their login name, so verification is a two-phase bind:

1. Find. Bind as a read-only service account and search base_dn for the entry matching user_filter, with {login} substituted.
2. Verify. Bind a second time as the entry that was found, using the password the user submitted. That bind is the authentication.

rolter never reads, compares or stores a password hash from the directory. It also never caches the password.

A filter matching more than one entry is a hard failure rather than a login: binding as an arbitrary one of several matches would authenticate the wrong person.

Group mapping is least-privilege

group_role_map maps a directory group onto a rolter role. It is explicit and additive:

• a user in several mapped groups gets the union of their roles;
• a user in no mapped group is rejected, even with a correct password.

There is no implicit default role and no wildcard. Authenticating proves who someone is, not what they may do — the same rule SSO already follows.

Group names are matched case-insensitively and with surrounding whitespace trimmed, because directories are inconsistent about DN case and an operator who copies a group from their directory’s UI should not silently get no access.

What a client is told

Every authentication failure returns the same generic rejection:

The first four are deliberately indistinguishable. A login form that distinguishes “no such user” from “wrong password” is a user-enumeration oracle against the corporate directory, which is a worse leak than the login itself.

A directory being down is distinguishable, because that is an operational fact the operator needs and reveals nothing about any account. Distinguished names, filters, bind credentials and raw LDAP result codes are logged for the operator and never returned.

Configuration

url = "ldaps://dir.example.com"          # ldaps:// strongly preferred
bind_dn = "cn=svc-rolter,dc=example,dc=com"
bind_password = "…"                       # read-only service account
base_dn = "ou=people,dc=example,dc=com"
user_filter = "(uid={login})"             # must contain {login}
email_attr = "mail"
display_name_attr = "cn"
group_attr = "memberOf"
require_tls = true

[group_role_map]
"cn=rolter-admins,ou=groups,dc=example,dc=com" = "admin"
"cn=rolter-users,ou=groups,dc=example,dc=com" = "member"

Configuration is validated before it is used to authenticate anyone. These are rejected at startup rather than at first login:

• a URL that is neither ldap:// nor ldaps://;
• require_tls with a plain ldap:// URL — the service-account bind password would cross the network in cleartext;
• a user_filter without {login}, which would authenticate whoever the filter happens to match, for any login submitted;
• an empty base_dn;
• an empty group_role_map, under which no directory user could ever be granted a role.

Filter injection

The login is escaped per RFC 4515 before substitution. Without it a login of * matches every user in the subtree, and )(uid=admin rewrites the filter — LDAP injection, the directory analogue of SQL injection. \, *, (, ), / and NUL are escaped.

Operational limitations

• Group membership is read from the user entry via group_attr (typically memberOf). Directories that do not populate a reverse-membership attribute need a nested group search, which is not implemented.
• Nested groups are not expanded. A user in a group that is itself a member of a mapped group is not granted that role. Map the groups users are directly in.
• No connection pooling. Each login opens and closes its connections. This is correct but not fast; an interactive login is not a hot path, and pooling bound connections has its own correctness hazards.
• No referral chasing. A referral from the directory is not followed.
• ldaps:// only for transport security. StartTLS on a plain ldap:// connection is not implemented.

Health and readiness

Both rolter binaries serve two operational endpoints that answer two different questions. They are unauthenticated, cheap, and safe to probe on every pod.

The words mean the same thing on rolter-gateway and rolter-control.

Why the split exists

Collapsing both probes onto one endpoint has no correct answer, only a choice of which failure to accept:

• If the single endpoint answers as soon as Axum binds, Kubernetes marks the pod Ready before it can serve. A rolling update then routes dashboard and /internal/snapshot traffic — which every gateway in the fleet polls — at a control pod whose migrations have not run and whose pool has no connection.
• If the single endpoint instead checks Postgres, a database blip fails the liveness probe and Kubernetes kills every control pod at once. A recoverable dependency outage becomes a restart storm, and the restarts make the outage worse by reconnecting a stampede at the recovering database.

Splitting the endpoints lets a dependency failure drain traffic without touching the process.

/healthz — liveness

Returns 200 OK with the body ok whenever the HTTP server is accepting connections. It checks no dependency: not Postgres, not Redis, not ClickHouse, not an upstream provider. That is the point — nothing outside the process may cause a restart.

/readyz — readiness

Gateway

Returns 200 OK normally, and 503 Service Unavailable with the body draining once the instance has been drained from the dashboard’s cluster screen — the state arrives on the /internal/snapshot response the gateway already polls. A drained gateway therefore leaves the Service and finishes its in-flight requests, instead of being killed with them in progress.

Control plane

Returns a JSON body describing each check:

{
  "status": "ready",
  "checks": {"database": "ok", "migrations": "ok", "kek": "ok"}
}

200 when status is ready, 503 otherwise. The checks are:

• database — the Postgres pool hands out a connection within 2 seconds. The timeout is deliberately shorter than a typical probe timeoutSeconds so an exhausted pool answers a clean 503 rather than hanging until kubelet reports a timeout, which looks like a dead process.
• migrations — every migration embedded in this binary is applied to the database. A pod running ahead of its schema would serve wrong answers on the CRUD API and /internal/snapshot, so it is not ready.
• kek — ROLTER_KEK parses when it is set. A KEK set to an empty value seals nothing and decrypts nothing, so the pod cannot serve credentials. Reported as not configured (and not a failure) when the variable is unset.

Redis and ClickHouse are deliberately not checked. The control plane serves configuration without either — losing Redis costs the config-bump fan-out notification, losing ClickHouse costs the analytics screens — so their absence is degraded, not unready. Removing every control pod from the Service because analytics is down would be a strictly worse outage.

Running without --database-url (bootstrap config only), there is nothing to wait on and the answer is always ready.

The dashboard’s gateway pill

The status pill in every screen header (ui/src/components/ScreenHeader.tsx) polls the gateway’s /readyz every 30 seconds through the control plane’s /gw/* proxy (#1973).

/readyz is the one gateway signal every dashboard role can read: the proxy takes no admin token and the endpoint takes no key. /api/v1/cluster/nodes knows more but is superadmin-only, and /api/v1/health/* is provider health from ClickHouse, not the gateway’s. The probe sends no Authorization header.

readinessFrom in ui/src/lib/gateway.ts reads the answer strictly:

A 200 that is not ok is an SPA fallback in front of the control plane, and a 502 that is not the proxy’s JSON comes from an ingress that could not reach the control plane. Both read as unknown, so the pill never sends an operator to the wrong process.

One failed check does not blank a known answer. gatewayHealthFrom in ui/src/lib/gateway-health.ts keeps it for up to one and a half poll intervals, without the pulse and with the answer’s time in the tooltip, then drops to unknown. The poll pauses while the tab is hidden, and the header’s refresh button re-asks along with everything else without spinning for the background poll.

What the pill cannot see:

• other gateways behind the same ROLTER_GATEWAY_URL, and whether a gateway runs the current config version. Both need a fleet summary every signed-in caller can read (#2013).
• an unreachable gateway host. The proxy’s HTTP client has no connect timeout, so a host that drops packets hangs the request instead of answering 502, and the pill reports unknown after its own 10 second timeout rather than down (#2014).

Probe wiring

charts/rolter wires this for you:

readinessProbe:
  httpGet: {path: /readyz, port: http}
livenessProbe:
  httpGet: {path: /healthz, port: http}
startupProbe:               # control plane only
  httpGet: {path: /healthz, port: http}

The startupProbe on the control plane holds liveness off while first-boot migrations run against a cold database, so a slow migration is not mistaken for a wedged process. Tune the periods under control.probes in values.yaml.

What to alert on

• /readyz down for longer than a failover — the control plane cannot reach its database, or is running against a schema it has not migrated. Page.
• /healthz flapping — the process itself is crashing or wedging. Page.
• /readyz down on a single gateway — usually just a drained instance, which is expected during a rolling update. Do not page on this alone.

Observability

Metrics

• The gateway exposes Prometheus metrics at GET /metrics: counters (rolter_requests_total, rolter_upstream_errors_total, rolter_auth_failures_total, reload/log/budget/rate-limit/retry/cooldown/health/breaker/scrape counters), gauges (rolter_config_version, rolter_breaker_entries), and per-model latency histograms — rolter_request_latency_ms (total) and rolter_request_ttft_ms (time-to-first-token), each labelled {model=...} with the standard _bucket/_sum/_count series. Histograms are observed once per completed request from the log sink, off the response hot path.
• The exporter is hand-rolled (atomic counters + non-cumulative histogram buckets cumulated at render) rather than the metrics facade + global recorder, which does not fit the lock-free arc-swap design where an explicit Arc<Metrics> is threaded through the request path.
• Passive per-target SLA signal: rolter_target_requests_total{provider,target,outcome} (a counter, outcome = ok for 2xx else error) is tallied once per completed request from the log sink — free, derived from real traffic, no extra upstream calls. A per-target error rate / uptime is sum(rate(rolter_target_requests_total{outcome="error"}[5m])) / sum(rate(rolter_target_requests_total[5m])). This is the first slice of provider stability tracking (ROL-123); the ClickHouse provider_health_events table and the dashboard land in later slices. The active prober is guarded: bounded probe concurrency with per-provider jitter, consecutive-failure/-recovery thresholds gating the unhealthy flip (no single-probe flapping), and exponential probe backoff when a probe itself gets a 429.
• Client disconnects (#1083): rolter_client_disconnects_total counts requests whose caller left before the response completed, and rolter_inflight_requests is the live in-flight gauge those requests must return to zero. Abandoned requests are logged with status 499 and are never retried — see Client disconnects.
• Provider queues (#1855): rolter_provider_queue_depth{provider} and rolter_provider_inflight{provider} gauges and the rolter_provider_queue_wait_ms{provider} histogram, kept by RAII guards in metrics.rs. A job carries a QueuedGuard from the moment it is built until a worker takes it (picked() observes the wait and hands over an InflightGuard) or it is dropped unrun, so a shed or stranded job can never leave the depth raised; the in-flight guard lives until the upstream’s response headers arrive, which is also when a worker is released, and covers the direct path when queueing is off. The steady-state lookup borrows the provider name, so the path allocates nothing. Over OTLP (#1862) the two gauges are a labelled gauge family (LABELLED_GAUGE_FAMILIES, registered as observable gauges beside the labelled counter families by register_labelled in rolter_core::telemetry), and each wait is also recorded into an OTLP rolter_provider_queue_wait_ms histogram on the Prometheus boundaries, through a per-provider QueueWaitRecorder whose provider attribute is built once when the provider’s counters are. The Redis consumers’ connection state (#1772) rides the same two families; see Redis connections. The rolter · gateway capacity board in integration/dogfood/signoz/dashboards/ charts all of it.
• Billed but withheld (#1478): a response an output guardrail or post-response plugin refuses to deliver is still billed. Its row carries the caller’s 403, withheld = 1, the policy in error, and the provider’s tokens and cost, and rolter_withheld_responses_total counts it. usage_unknown = 1 marks any successful row whose upstream reported no usage. See Billed but withheld.
• Multi-key providers: rolter_key_cooldowns_tripped_total counts api keys parked after a key-level failure (429/401 on a provider with several keys); the request retries in-flight on a sibling key.
• A/B attribution: rolter_variant_requests_total{model,variant} (a counter) tallies requests per chosen variant, so traffic splits are visible in Prometheus/Grafana without querying ClickHouse. Classic single-pool routes (no variant) emit nothing. Observed from the same log-sink funnel (ROL-195, part of ROL-188).
• Adaptive routing (#544): rolter_adaptive_routing_decisions_total{model,mode} splits a route’s picks between blend (the latency/cost/load blend), exploration (the bounded random share that keeps starved targets sampled) and fallback (the deterministic pipeline stack served while the kill switch is off or the evidence is too thin), and rolter_adaptive_routing_engaged{model} is 1 while the blend is actually routing. Only routes on the adaptive strategy emit these. A config reload rebuilds the balancer and so resets the counters, which lines up with a rolter_config_version bump — alert on rate(), not the absolute value. A route sitting at engaged 0 with all picks on fallback is the expected steady state before an operator enables the policy. rolter_adaptive_routing_target_score{model,target} (#751) adds the blended score each target currently carries — the same numbers the control plane serves at GET /api/v1/adaptive-routing-telemetry, computed at scrape time rather than on the request path.
• Roadmap: add per-provider/route labels on the histograms, in-flight gauges, cache-hit ratio, and circuit-breaker state gauges.
• Roadmap: scrape/federate upstream engine metrics from vLLM/SGLang/TGI /metrics and correlate them per target (queue depth, KV-cache usage, running/waiting requests) to feed load- and cache-aware routing and the dashboard.

Where request logs go (#929)

The dashboard’s Usage, Costs and Logs screens read ClickHouse through the control plane. The rows they read are written by the gateway. For a long time nothing connected the two: the control plane took CLICKHOUSE_URL, the gateway took [logging].clickhouse_url from its own bootstrap TOML, and a deployment that set only the first one logged nowhere while the screens queried a table nobody filled. The snapshot even carried the field, as "clickhouse_url": null, so the control plane was telling every gateway to log nowhere.

There is now one variable and one order of precedence. The gateway resolves its destination, at startup, as:

1. [logging].clickhouse_url in its own config file — an explicit local decision is never overridden.
2. CLICKHOUSE_URL in its environment. Both processes read this one variable, which is what a compose file, a Helm release or rolter init sets.
3. The value the control plane publishes in /internal/snapshot, when the gateway is started with --snapshot-url. The control plane puts its own CLICKHOUSE_URL there, so a fleet writes where the dashboard reads without anyone configuring the gateways individually.

Step 3 is read before AppState is built, not on the first poll, because the ClickHouse writer is spawned once at startup — the hot-swappable snapshot carries routing, not background tasks, so a destination arriving on a later poll would arrive too late to open a sink. It is best-effort by construction: a control plane that is not up yet costs one five-second timeout and the gateway starts exactly as it did before.

When all three come up empty on a gateway that has a control plane, startup logs a warning naming both fixes. An empty analytics screen and a quiet deployment look identical, and rolter check says the same thing before anything starts.

Tracing & context propagation

• tracing + tracing-subscriber with RUST_LOG filtering; TraceLayer logs each HTTP request.
• Inbound: accept W3C traceparent/tracestate (and b3) from clients and continue the trace; honor x-request-id / x-correlation-id.
• Outbound to engines: inject the active trace context into upstream requests so vLLM/SGLang/TGI spans join the same distributed trace. vLLM and SGLang support OpenTelemetry tracing (e.g. vLLM --otlp-traces-endpoint); point them at the same OTLP collector so engine prefill/decode spans line up with rolter’s request span.
• A per-request request_id is echoed in a response header and stamped on logs, metric exemplars and spans for correlation.
• Events: rolter calls no span-event API directly, but tracing-opentelemetry turns every tracing event fired inside an active span into an OTel span event. That API is deprecated upstream in favour of log-based events; the target model, and the sequencing it forces, are recorded in ADR-0025.

How the context actually moves

rolter-core::telemetry owns both directions, and both are inert unless an OTLP endpoint is configured:

• Extract. GatewayMakeSpan (rolter-gateway::trace) is the TraceLayer’s span-maker: it builds the request span and makes the extracted inbound context its parent. A B3-only caller is normalized into an equivalent traceparent first, so one W3C propagator serves both wire formats. Without this the gateway’s spans were disconnected roots — the trace id reached the request log, but nothing joined the caller’s trace.

  It has to be the span-maker rather than a middleware layered inside the TraceLayer: DefaultMakeSpan builds the request span at DEBUG, so under the default RUST_LOG=info it is disabled, and setting a parent on a disabled span silently does nothing. With no pipeline installed it falls back to that stock DEBUG span, so the untraced path costs what it always did.

• Inject. The context handed to the provider is injected from the current span, inside the per-attempt upstream.request span, rather than copied from the caller. Copying it verbatim made the provider call a child of the caller’s span and therefore a sibling of the gateway’s own work, which silently invalidated every waterfall built from the data. The allowlisted client headers from Forwarder::forwarded_header_names are unaffected; only the trace headers changed hands.

• Log correlation. RequestLog.trace_id is read off the span context when a pipeline is installed and falls back to parsing the inbound header otherwise, so ClickHouse and the trace backend agree by construction.

Pipeline spans

One span per stage, so a slow request is attributable rather than merely slow:

queue.wait spans enqueue→dequeue only: the span travels with the queued job and the worker closes it the moment it picks the job up, so it measures the wait and not the wait plus the upstream call. The job carries the caller’s span alongside it, and the worker instruments the forward with it — the queue worker runs on its own task where nothing is in scope, so without that the forwarder’s own translate.request span becomes an orphan root in a trace of its own. Names follow the OTel GenAI semantic conventions where they fit, so a backend’s built-in GenAI views work.

Spans never carry prompt or completion content, API keys, virtual-key plaintext, or injected header values — those are credential material, and redaction stays owned by the existing logging_settings machinery rather than a second policy.

Control-plane spans

The control plane runs the same pipelines as the gateway (telemetry::init()), but until #845 emitted no spans of its own — everything it did was invisible beyond what the HTTP layer produced by default.

http.route is the matched template (/api/v1/providers/{id}), never the concrete path. control.request comes from one middleware rather than an attribute on each of ~90 handlers, so a route added tomorrow is instrumented the moment it is mounted.

snapshot.build is the one to watch. Snapshot latency is fleet-wide config-propagation delay: every gateway waits on it, so when an operator changes a route and the fleet serves stale config, this span is what says whether the delay is in generation or downstream of it. config_version on the span answers “which config was this” without putting an unbounded value on a metric.

Tenant attributes

The gateway.request span carries rolter.org.id, rolter.team.id and rolter.project.id, recorded once the virtual key resolves. The span itself is built by the tower layer, which runs before auth, so the fields start empty and are filled in rather than passed at construction.

These exist so per-tenant telemetry destinations are routable. ADR-0026 decided that fan-out to tenant-owned backends belongs in an OpenTelemetry Collector rather than in-process exporters — one egress path in the gateway no matter how many tenants, and no data-plane process POSTing to operator-supplied URLs — and that rolter’s job in that design is to stamp the attribute the collector routes on. Until this landed there was nothing to stamp: the request logs in ClickHouse always carried tenant identity, but no exported span ever did, which made the routing half unimplementable.

An unattributed request — a config-defined key, which has no org — records nothing rather than an empty string. An attribute that is present-but-blank on some spans and absent on others is harder to write a routing rule against than one that is consistently absent.

The names are deliberately rolter-local. The GenAI and HTTP conventions define nothing for tenancy, and a convention-shaped guess like tenant.id would be worse than an obviously-local name if the spec later defines it differently.

Running it locally

The observability overlay starts a collector and a trace UI alongside the normal stack:

docker compose -f docker/docker-compose.yml \
               -f docker/docker-compose.observability.yml up

Traces land at http://localhost:16686; the collector takes OTLP on 4317/4318 and re-exposes collected metrics on 8889. The overlay sets OTEL_EXPORTER_OTLP_ENDPOINT on the gateway and control services, so bringing it up is the only step — without it that variable is unset and tracing stays off.

The collector binds 0.0.0.0, not localhost: one bound to loopback inside its container is unreachable from the gateway container.

Choosing an overlay

There are two, and they are mutually exclusive — both publish OTLP on 4317/4318.

docker compose -f docker/docker-compose.yml \
               -f docker/docker-compose.signoz.yml up      # SigNoz on :8080

Jaeger is the default because it is two containers and starts in seconds, and because reading a correctly-parented waterfall is what the tracing work needed. Reach for SigNoz when “which stage is slow across all requests” matters, which Jaeger cannot answer.

Nothing in the Rust differs between them: the gateway speaks vendor-neutral OTLP and only the destination changes. Any other OTLP backend works the same way — repoint the exporter in infra/otel/collector.compose.yaml.

Querying SigNoz from an agent (MCP)

The SigNoz overlay also starts SigNoz’s MCP server on http://localhost:8000/mcp, so an agent can query traces, run ClickHouse queries, and manage dashboards against the local instance. Point an MCP client at that URL; the SigNoz agent-skills plugin ships a signoz server entry to fill in.

It needs a SigNoz API key, which is created in the UI (Settings → API Keys, admin only) and supplied to the server, not the client:

set -x SIGNOZ_API_KEY (pass show rolter/signoz-api-key)
docker compose -f docker/docker-compose.yml \
               -f docker/docker-compose.signoz.yml up -d signoz-mcp

The key is read from the environment and never written to a tracked file. With no key set the container still starts, but every call returns Authorization or SIGNOZ-API-KEY header required.

The MCP dashboard tools need SigNoz v0.135.0 or newer, which is why the signoz image is pinned ahead of the collector/ClickHouse pair.

The SigNoz overlay is a pinned equivalent of what SigNoz’s Foundry CLI generates, since SigNoz deprecated its own compose manifests in v0.130.0. Two things to know before touching it: its four images are a tested set and must be bumped together (a newer migrator emits ClickHouse settings an older server rejects), and it vendors 3.6 KB of ClickHouse config — the cluster topology and the {shard}/{replica} macros that ReplicatedMergeTree needs — rather than SigNoz’s full 56 KB config.xml, which the stock image defaults cover.

Metrics over OTLP

The counters and gauges rolter-gateway::metrics computes are exported over OTLP as well as served on /metrics. The Prometheus endpoint is unchanged — this is a second exporter over the same numbers.

Both read one list, Metrics::scalars(), so they cannot drift: a counter added there reaches Prometheus and OTLP without a second edit. Counters export as counters and gauges as gauges, since exporting a counter as a gauge would break rate() on the backend.

The instruments are observable: nothing is pushed on the request path. The SDK invokes the callback on its own schedule (OTEL_METRIC_EXPORT_INTERVAL, default 60s) and reads the same atomics the Prometheus renderer does, so the hot path still only does fetch_add. With no OTLP endpoint configured no meter provider, exporter or callback is built at all.

Per-model histograms and label-bearing counters stay Prometheus-only for now; the scalar set is what OTLP carries.

Control-plane metrics

The control plane has no Prometheus registry to mirror, so these are the one place it measures itself (#845). They are real histograms — measurements taken as they happen — not observable instruments, because a duration cannot be reconstructed from a counter after the fact.

And one counter, for the endpoint an unauthenticated attacker can reach (#1079):

A counter rather than a histogram: the question it answers — “is somebody running a credential-stuffing run against this deployment” — is a rate, not a distribution. It carries no account or address label; either would be unbounded cardinality and would put the identity an attacker is guessing into the metrics pipeline. A rising invalid with a rising locked behind it is the throttle working; a rising invalid with no locked means the run is spread thin enough to stay inside the per-account budget, and the per-address budget is the one to tighten.

And one for the telemetry the control plane ingests but cannot store (#1747):

Both callers swallow the failure by design — the dashboard drops the batch and keeps flushing — so without this a missing ui_events table loses the whole UX stream with no signal at either end. insert is a store that refused or could not be reached; unconfigured is a control plane with no CLICKHOUSE_URL. The store’s error text is deliberately not a label (it is unbounded); it is in the matching telemetry ingest failed warning, which crates/rolter-control/src/ingest_failure.rs rate-limits to one per minute per stream and which carries a suppressed count of the failures it stands for. Any sustained non-zero rate is worth an alert.

And the connection pool, as observable gauges (#1052):

One pool serves /internal/snapshot, the whole CRUD surface and every RBAC membership lookup, so a ceiling that is too low presents as “the control plane got slow” with nothing to attribute it to. The three gauges together are what separate the two cases: pool-bound is connections == max while idle is zero and rolter_db_pool_acquire_ms shows waits; acquire waits without a pinned pool mean the database itself is slow, and raising the ceiling there makes it worse.

rolter_db_pool_acquire_ms is sampled on a 15-second timer rather than instrumented per call. Wrapping every acquire() would mean touching every repository method to answer a question that is a distribution over time, not a per-request fact. The probe takes one connection and drops it: if that is disruptive, the pool is already far too small, and that is the finding.

Boundaries are deliberately not the gateway’s. A gateway request is dominated by an upstream model call and is interesting out to tens of seconds; a snapshot build and a CRUD write are database work where “is this 2 ms or 40 ms” is the whole question, so reusing the gateway’s boundaries would put nearly every observation in the first bucket.

Cardinality is bounded by construction. http.route is the matched template, so a thousand providers are one series. Status is recorded as a class, not a code: twelve statuses across ninety routes would be over a thousand series to answer a question five buckets answer. config_version is unbounded — a new value on every config write — so it lives on the snapshot.build span and never on a metric.

rolter_snapshot_payload_bytes skips the 304 case rather than recording a zero: a not-modified poll transfers no body, and folding zeroes in would drag the size distribution down and misreport what the fleet actually moves.

Payload size matters on its own. It is what every gateway transfers on every poll, and it is the first thing to look at when propagation gets slower without generation getting slower. The snapshot bench in rolter-core shows why the encode is the stage to watch: at 1000 routes it costs ~2.8 ms against ~150 µs for sanitize and ~120 µs for validate.

The control plane also registers the same process and runtime metrics as the gateway — it degrades for the same reasons and previously answered none of those questions either.

Logs over OTLP

Logs export alongside traces and metrics, gated on the same environment (OTEL_EXPORTER_OTLP_ENDPOINT, or the logs-specific OTEL_EXPORTER_OTLP_LOGS_ENDPOINT). With neither set no exporter is built and logging is stdout-only, exactly as before.

It is additive, not a replacement: the stdout fmt layer is untouched, so docker logs shows what it always did and the collector additionally receives the same records.

Records carry trace_id and span_id, which is most of the value — a log joins the trace it came from instead of being a separate pile of text. tracing-opentelemetry publishes the OpenTelemetry context when a span is entered, and the logs SDK stamps it onto each record. That correlation is asserted by a unit test rather than assumed from a dependency default, since it would fail silently if the default ever changed.

Redaction is unaffected: this exports the same tracing events the stdout layer renders, so anything already kept out of logs stays out of them.

Process metrics

Alongside the domain counters, the process reports its own vitals: resident and virtual memory, CPU time, open file descriptors, thread count and uptime. These are the numbers an operator reaches for first when a node degrades, and the gateway previously answered none of them.

They are observable instruments read from /proc, so nothing touches the request path. On a platform where /proc is unavailable the instruments are not registered at all, rather than registered and always zero — a metric that reads zero forever is worse than an absent one, because a dashboard cannot tell it from a healthy process. Names follow the OTel process.* conventions.

Runtime metrics

The process metrics describe the machine; these describe the scheduler. tokio.runtime.queue.depth is the one that matters: when latency rises, no other signal separates “the provider is slow” from “the request sat in our own run queue before we ever called the provider”, and the two have opposite fixes. It pairs directly with the queue.wait span. Alongside it are tokio.runtime.workers, tokio.runtime.tasks.alive and tokio.runtime.worker.busy.time.

Busy time is exported, not a busy ratio: a ratio computed in-process would average over whatever interval the SDK happens to use and would not re-aggregate across instances. As a monotonic counter the backend derives utilisation with rate(tokio.runtime.worker.busy.time) / tokio.runtime.workers, which does.

None of this requires --cfg tokio_unstable. #834 assumed it did, and that is true only of part of tokio’s surface: num_workers, num_alive_tasks and global_queue_depth are stable, and worker_total_busy_duration sits behind cfg_64bit_metrics!, which is #[cfg(target_has_atomic = "64")] — a property of the target, not an instability gate. What remains genuinely gated is the blocking pool and the per-worker steal/poll counters, which are therefore not exported; the workspace-wide flag decision stays unmade rather than being smuggled in with a telemetry change.

Outside an async runtime the instruments are not registered, for the same reason the process metrics are absent without /proc.

Connection-pool metrics

rolter_upstream_connections_total and rolter_upstream_connect_errors_total count connections established to providers, and attempts that never got that far.

Pool exhaustion presents as latency with healthy providers — the failure mode the other metrics cannot explain. reqwest and hyper expose no pool introspection at all, so this is instrumented rather than read: a tower layer over the connector (ClientBuilder::connector_layer) sees every connection hyper builds because it had none to reuse. The signal is the ratio rate(rolter_upstream_connections_total) / rate(rolter_requests_total) — near zero when the pool is working, approaching one when every request is paying a fresh TCP and TLS handshake.

Idle-versus-active counts stay unavailable: that needs the connection object’s drop, and the connector layer’s response type is opaque outside reqwest.

Resource attributes

Every signal carries service.name, service.version (the crate version), and where configured service.instance.id and deployment.environment.name. OTEL_RESOURCE_ATTRIBUTES is honoured by the SDK for anything else.

service.instance.id is the very same value the cluster watcher sends, read from rolter_core::node_identity rather than derived a second time, so a node in cluster_nodes and a node in the trace backend are the same node by construction. The precedence is ROLTER_NODE_ID, then HOSTNAME, then the host’s own name from gethostname (#1644) — the syscall is last so an operator who names a replica keeps that name, and it exists because a gateway started from a shell, a systemd unit or a launchd job has no HOSTNAME in its environment. When even the syscall fails the attribute is omitted rather than invented per restart, which would churn the identity on every deploy. A value the control plane’s ingest would reject — blank, longer than 128 bytes, or carrying a control character — counts as not resolved, so nothing is ever reported under an id that is discarded on arrival.

Wrapping audit (#815)

OpenTelemetry’s Don’t wrap OpenTelemetry argues that a house abstraction over the instrumentation API costs performance, maintainability and developer education. Three anti-patterns are named: wrappers that force callers to allocate an attribute collection, wrappers that look an instrument up by name per measurement, and general API abstraction that teaches a proprietary interface instead of the standard.

rolter has four things sitting between its code and the OTel API. Each was audited against that post; the verdict is keep for all four, for the reasons below. The post is guidance rather than a mandate, and it asks for a refactor only where there is a measured cost or a real maintenance burden.

Note up front that rolter instruments through tracing + tracing-opentelemetry. That is the ecosystem bridge, not a bespoke house wrapper, and it is not what the post argues against. This audit is not a proposal to remove tracing.

stage_span! expands to a direct tracing::info_span! call guarded by an is_active() check, so it is closer to the code generation the post recommends than to a runtime wrapper. It takes tracing’s own compile-time field syntax and never asks a caller to build a Vec or a slice of attributes, so the force-allocation anti-pattern does not apply. The guard is the point of the macro: with no pipeline installed it yields Span::none(), which allocates nothing.

The scalar-metrics list is the shape most at risk, since install_metrics registers one observable instrument per scalar and each instrument’s callback calls collect() and finds its own entry by name. That is a by-name lookup, but it is not on a hot path: the instruments are observable, so the SDK invokes those callbacks on its own export interval (OTEL_METRIC_EXPORT_INTERVAL, default 60s). The request path only ever does fetch_add on a named AtomicU64 field — there is no map, no lookup and no lock between a request and its counter. The post’s performance argument therefore does not bite here even though the gateway hot path is where it would bite hardest.

What the export path does cost is one Vec<ScalarMetric> allocation per instrument per cycle and a linear scan of it, so the work is quadratic in the number of scalars. At the current eight scalars, once a minute, that is immaterial. It is left as-is deliberately: the OTel Rust 0.33 API offers only per-instrument with_callback, so collecting once per cycle for all instruments is not expressible, and matching by name rather than by index keeps the callbacks independent of the order scalars() happens to return.

RequestHistograms::record is the one place a wrapper does force an allocation on the request path — model.to_string(), to build the single KeyValue both histograms take. It is unavoidable rather than incidental: OTel’s Value::String holds an owned or 'static string and model names are neither. The obvious way to avoid it is to cache a prebuilt attribute set per model, which would put a sharded-lock map lookup on the request path — precisely the lookup-based anti-pattern the post names, and something AGENTS.md forbids on the data-plane hot path. One small allocation is the cheaper of the two, and it is paid only when an OTLP endpoint is configured.

GatewayMakeSpan is a tower_http::trace::MakeSpan implementation: layer configuration, which the post explicitly separates from instrumentation and calls not wrapping. Recorded here only so the audit is complete.

Turning telemetry off explicitly

ROLTER_TELEMETRY_ENABLED=false hard-disables every export — traces, metrics and the dashboard’s browser tracing — regardless of which OTEL_* endpoints are set (#812). Unset means enabled, which changes nothing for an existing deployment: with no endpoint configured nothing is exported anyway.

The switch can only subtract. It never turns export on by itself, and an unrecognized value leaves export on rather than silently blinding a deployment; only 0, false, no and off disable it.

It exists because “off” was previously implicit — achieved by leaving an endpoint unset — which does not survive somebody setting the endpoint for one signal and gives an operator nothing to point at in a security review. It is deliberately environment-only and has no config-file equivalent; see ADR-0026, which also records why per-tenant telemetry destinations belong in the collector rather than in rolter.

Cost when tracing is off

With no OTEL_EXPORTER_OTLP_ENDPOINT (the default) behaviour and hot-path cost are unchanged: telemetry::is_active() is a single relaxed atomic load, stage spans are Span::none() (no allocation, and instrumenting a future with one is a no-op), no carrier is built, and outbound trace headers are copied verbatim exactly as before.

Exporters (OTel-compatible)

rolter emits traces and metrics via OpenTelemetry OTLP (gRPC/HTTP), so any OTel-compatible backend works without code changes — just set an endpoint and headers:

• SigNoz, Grafana Tempo/Mimir, Honeycomb, Datadog (OTLP intake or the OTel Collector datadog exporter).
• Langfuse for LLM-specific observability (prompt/response, token usage and cost as traces), ingested via its OTLP endpoint or SDK.

Recommended topology: rolter → OpenTelemetry Collector → fan-out to the chosen backends. The collector also scrapes the upstream engines’ /metrics and rolter’s /metrics, keeping vendor specifics out of rolter. Configure via env, e.g. OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME=rolter-gateway.

infra/otel/collector.yaml is a ready-to-run local collector example: it accepts OTLP/gRPC and OTLP/HTTP, scrapes the compose gateway’s Prometheus endpoint, exposes collected metrics on :8889, and prints telemetry via the debug exporter. It intentionally has no external backend configured, keeping the default example safe for air-gapped use. Replace the debug exporter with an internal OTLP-compatible destination for production.

Connector delivery: rendered collector config (#836)

The observability_connectors table (#511, docs/dev-docs/architecture/data-model.md) lists the sinks a deployment wants its telemetry shipped to in addition to ClickHouse, but the connector’s endpoint is never called directly by rolter. Consistent with ADR-0026 — fan-out belongs in the collector, not N in-process SDK exporters each carrying its own connection/queue/retry buffer, and terminating an operator-supplied URL in the collector keeps that SSRF surface out of the data plane — GET /api/v1/connectors/collector-config (superadmin-only) renders an OpenTelemetry Collector config document from the enabled rows instead: one otlphttp exporter and one traces/metrics/logs pipeline set per connector, receiving from the otlp receiver rolter’s own OTEL_EXPORTER_OTLP_* export targets.

The endpoint answers a superadmin principal only, and a collector has no session to present one, so a collector is not pointed at it: that would put the admin token in the collector’s own deployment. The dashboard’s Collector config dialog (#1195, #2106) shows the document and the full endpoint URL, built from the control plane’s public base, and the operator saves the document as the collector’s config file, or an automation fetches it with its own credential and reloads the collector. A connector that is switched off is left out of the document, and the dashboard creates connectors switched off unless its add sheet’s start switch is on (#2349). sampling_rate becomes a probabilistic_sampler processor scoped to that connector’s own pipeline, since sampling is now the collector’s decision, applied independently per destination rather than once for the whole deployment. A managed secret (managed_auth_secret) is decrypted server-side and rendered as a literal Authorization: Bearer header; an external reference (auth_secret_ref) instead renders as a ${env:...} placeholder the collector’s own environment must resolve, since rolter never holds that secret’s value.

ROLTER_TELEMETRY_ENABLED wins over every connector row: when it is off, the rendered document has no exporters and no pipelines regardless of what is enabled in the registry, the same way it already silences every in-process exporter.

Backpressure to a slow or dead sink is the collector’s problem now, not rolter’s: it already queues and retries per exporter, which is the reason fan-out moved here in the first place rather than something this endpoint needs to build.

The remaining sink kinds from #511 (Datadog, Prometheus remote-write, Langfuse) become an exporter-block case in this renderer, not a new in-process delivery adapter — widening KINDS in connectors.rs and the kind check constraint in migration 0062 is the only rolter-side change; everything else is collector configuration.

Alert evaluation and delivery (#1871)

crates/rolter-control/src/alerting.rs owns alerting end to end: channels, rules, a 60-second evaluator started by every control-plane process that has a database, and the alert_notification_history table (migration 0025). The user-facing behaviour is in docs/user-docs/observability/alerting.mdx; this section is the reasoning behind it.

One evaluation is one transaction around the rule’s row lock. The ClickHouse read runs first, unlocked and bounded to 30 seconds, because it is the slow part. Then the evaluation takes select … for key share on the rule’s channel, select … for update on the rule, decides whether the reading is a transition, POSTs it, writes the rule and the history row, and commits. The scheduled pass takes the rule lock with skip locked and and enabled, so a rule another replica is evaluating is left to that replica and a rule disabled mid-pass is skipped; Evaluate now waits for the lock instead. That is what keeps control.replicaCount > 1 from reporting each transition once per replica. An edit that lands between the read and the lock and changes the signal, window or channel discards the reading, since it measures a query the rule no longer asks for or would go to a channel that was not locked.

The channel is locked before the rule because deleting a channel takes the locks in that order: the delete locks the channel row, then its on delete set null locks every rule naming it. With the evaluator taking the rule first and meeting the channel only at the history insert’s foreign-key check, a delete issued during a POST deadlocked with it (40P01), and either the delete returned a 500 or the evaluation rolled back after the receiver already had the alert. In the same order, the delete simply waits for the delivery to commit. key share rather than share keeps the channel editable meanwhile, and update_channel locks with for no key update so an edit does not queue behind a delivery either.

An older reading never overwrites a newer one. The unlocked read also takes clock_timestamp(), the database clock rather than the replica’s, as the reading’s time, and that is what last_evaluated_at stores. Once the rule is locked, a reading older than the rule’s last_evaluated_at is dropped: another evaluation started later and has already recorded its result, so applying the older one would report a transition pair that never happened. This matters most for a query that runs into the 30-second bound while a later pass succeeds.

A transition is decided against the history, not the state column. state also holds unknown and error, and neither says what an operator was last told. The newest history row does: firing is reported when the reading is high and the last row is not firing, resolved when the reading is low and the last row is firing. That makes three cases fall out without special handling: a rule enabled while its condition holds fires on its first pass; an evaluation error between two high readings does not fire twice; and a condition that cleared while evaluation was failing still resolves. It also makes a crash between the rule update and the history insert self-healing, since the next pass sees no row and reports again. The history insert uses clock_timestamp() rather than the column’s now() default, because now() is the transaction’s start and a transaction that waited on the lock started before the one it waited for.

Delivery never fails an evaluation. Every way a POST can go wrong becomes a failed row with a short detail (the HTTP status, or timed out, could not connect, endpoint denied by the egress policy, channel secret could not be unsealed, channel secret is not a valid header value), and the rule’s state still moves. The response body is never read, since a receiver’s error body can echo the bearer secret it just rejected.

A state that was never delivered is retried. One dropped POST at the moment a rule fires would otherwise mean no page for the whole incident, since every later pass reads firing against a newest row of firing. So transition() treats a holding state as unreported while every row for it since the last change is failed or skipped, and sends it again once the rule has an enabled channel. Each attempt is a new history row with a new id. The scheduled pass spaces attempts out by the number that failed: the next pass, then 2, 4, 8, 16 and 32 passes on, then hourly (retry_delay), so a dead endpoint writes a handful of rows an hour rather than one a minute, and the retries stop at the first delivered row or at the next change of state. A skipped row costs no delay, so a transition recorded while the channel was off goes out on the first pass after it is switched on or attached, and a rule with no enabled channel writes nothing more. Evaluate now retries without waiting, as an explicit operator request.

The delivery client is a dedicated reqwest::Client with redirects off, a 5-second connect and a 10-second total timeout. Redirects are off for the same reason as MCP OAuth discovery: a 3xx is how a host that passed the egress check hands the request to one that would not have. The endpoint is checked with EgressPolicy::check_url at save time and again before each POST, which matches connectors, and is stored as reqwest::Url serializes it, so the check and the request read one spelling of the host. A denial is logged with the channel id and the reason, never the endpoint, whose query string can carry a token. A save that only switches a channel off, with the endpoint unchanged, skips the check, so a channel a since-tightened policy denies can still be disabled. Nothing classifies what a hostname resolves to at connect time; that gap is shared by every control-plane outbound client and tracked in #1949.

A channel secret is bound to the endpoint’s origin. update_channel clears the sealed columns when the new endpoint’s scheme, host or port differs from the stored one and the request carries no new managed_secret; otherwise anyone with alert_channel:update could repoint a channel and receive its bearer secret. The audit entry records endpoint_changed, secret_replaced and secret_cleared, never the endpoint. A secret must also pass HeaderValue::from_str as Bearer <secret> at save time, since a trailing newline from a file would otherwise fail every delivery with no hint.

A failed evaluation writes state = 'error' and a last_error that is safe to show: a transport error is reduced to its class (reqwest’s message carries the URL, and CLICKHOUSE_URL can hold a password), while a ClickHouse error response keeps its status and exception text, cut at 512 bytes. The raw error goes to the log as alert signal query failed.

Request & cost logs

• Every proxied request is logged to ClickHouse (request_logs): identifiers, model, provider/target, status, token counts, cost_usd, latency, TTFT, cache flag, error.
• Sampling (logging_settings.sample_rate, default 1): LogSink::enqueue drops a row whose request-id hash bucket is at or above the rate (should_sample_request in crates/rolter-gateway/src/logging.rs) before it reaches the ClickHouse channel. LogSink::observe, the budget spend recorder and the rate-limit token recorder all run before that decision, so budgets, rate limits and /metrics count every request. The rate is not stored on the row and no read scales by it: below 1 the analytics endpoints, /api/v1/me/usage and the request_volume / spend_velocity alert signals return the sampled share (scaling them is #2239 and #2278). The Logs Settings screen warns with the resulting share whenever the form holds a rate below 100 % (#2088).
• ts is the instant the request began — the same instant latency_ms is measured from, and the one the passive health event derived from that request carries. It is reconstructed from the request’s monotonic start (Instant), so a clock step during a long request cannot reorder rows, and it is written by the gateway as an RFC 3339 literal at the column’s millisecond precision (the insert asks ClickHouse for date_time_input_format=best_effort so that literal parses into DateTime64(3)). Add latency_ms to ts for the completion time. The same rule applies to the matching request_payloads row, which copies its request’s ts.
• Gateways older than #1210 did not write ts at all and let the column’s default now64(3) stamp it, which recorded the batch flush time: every row in one flush shared a single millisecond, bursts collapsed onto one point, timeseries buckets were skewed by the flush interval, and keyset paging by (ts, request_id) had no order within a batch. The column default is kept as a fallback for those writers, so historical rows and any pre-#1210 gateway still land — but on such rows ts means flush time, not request time.
• Retention defaults to 90 days for metadata and seven days for captured payloads, set as the TTL in the ClickHouse schema. Both are admin-managed: PUT /api/v1/logging-settings accepts retention_days (1–3650) and payload_retention_hours (1–8760) and issues the matching alter table … modify ttl against ClickHouse, which then expires parts on its own schedule. Payload retention may not exceed metadata retention, so raw prompt bodies never outlive the row they belong to. A ClickHouse failure leaves the stored policy in place and is logged rather than failing the admin write — re-saving reapplies it.
• Payload capture is disabled by default. Set [logging.payload_capture] enabled = true to write redacted request and response payloads to the separate request_payloads table, which has a seven-day TTL (versus 90 days for request metadata). max_bytes bounds each body; redact_fields adds recursively redacted JSON keys before storage. Optional models and virtual_key_ids allow-lists make the deployment-level switch route- or key-specific.
• Request id / trace continuation: every request carries an x-request-id — the caller’s when supplied, otherwise a generated UUID — which is echoed on the response and stored on the log row for end-to-end correlation. An inbound W3C traceparent or B3 (b3 / x-b3-traceid) header is parsed and its trace id stored in request_logs.trace_id, so gateway logs join the caller’s distributed trace instead of starting a disconnected one.
• Outbound propagation: when the caller sent trace context, it is forwarded verbatim to the chosen upstream (traceparent, tracestate, and the b3 / x-b3-* family) so vLLM/SGLang/TGI continue the same trace. An untraced request adds nothing to the upstream wire — this is the caller’s own context, not a rolter fingerprint, so it preserves wire transparency.
• Writes are async and batched off the hot path so logging never adds request latency.
• The dashboard queries ClickHouse for usage, spend, latency percentiles and error rates, sliced by org/team/project/key/model.
• Who reads it is decided per row: every analytics and health query binds the caller’s tenancy, and captured bodies are masked below the request_payload floor (member, or viewer on a project that allows it). See Who reads the request log.

Time bounds on the read API

Every control-plane read over ClickHouse (the five /api/v1/analytics/* endpoints, the three /api/v1/health/* rollups, /api/v1/mcp/logs and its summary, and /api/v1/me/usage) takes a caller-supplied since/until, and the two keyset-paged lists also take a cursor whose first half is a timestamp. All of them are bound as ClickHouse parameters and parsed in SQL with parseDateTime64BestEffortOrZero. The OrZero variant is required: ClickHouse constant-folds both branches of the if that picks the default window, so a strict parse of an absent (empty) bound aborts the query (#1177). Its side effect is that anything the parser cannot read becomes 1970-01-01, so before #1192 a typo in since silently scanned the whole table.

crates/rolter-control/src/time_bounds.rs closes that gap before any SQL is built:

• is_time_bound accepts one grammar: YYYY-MM-DD, optionally followed by T or a space and hh:mm[:ss[.f{1,9}]], optionally followed by Z or ±hh:mm. That is RFC 3339 as the dashboard writes it (toISOString()), the YYYY-MM-DD hh:mm:ss.sss form ClickHouse returns for a DateTime64(3) under the default date_time_output_format=simple (so every cursor the API hands out), and a bare date.
• It is narrower than RFC 3339 exactly where ClickHouse misreads it. Measured against ClickHouse 24.10, each of these parses to the epoch: a lowercase t or z, year 0000, a day the calendar lacks (2026-02-30), and an offset whose + a client left unencoded, which the query string decodes to a space. Years outside DateTime64’s 1900-2299 range are accepted because ClickHouse clamps them to that range instead.
• The analytics, health, MCP-log and self-service modules import time_bounds::Query in place of axum’s Query. It deserializes the same way, then checks since and until through the TimeBounds trait every windowed query type implements, so a new query type that never names its window does not compile. An empty bound still means “the default”.
• analytics::parse_keyset_cursor applies the same grammar to the cursor’s timestamp half, for both the invocation list and the MCP call log.
• A refusal is a 400 in the gateway’s OpenAI-style envelope, {"error": {"message", "type": "invalid_request_error", "param", "code"}}, with code invalid_time_bound, invalid_cursor, or invalid_query for a query string that does not deserialize at all (axum’s own rejection is plain text). When the value is a valid bound with its + turned into a space, the message says to send %2B.

A valid bound is forwarded byte for byte, so it means what it always meant. The router-level tests in time_bounds.rs run every windowed route against a stand-in ClickHouse to prove both halves: a malformed bound never reaches the database, and a valid one arrives unchanged. every_documented_time_bound_is_on_a_checked_route fails when the served OpenAPI document gains a route with a since, until or cursor that the table there does not list. /api/v1/me/usage authenticates a session first, so self_service_key_lifecycle in tests/control_integration.rs covers it.

Provider health events

• Every health signal is written to ClickHouse (provider_health_events): target_id, provider, source, outcome, status_code, latency_ms, error_kind, and ts.
• ts is the instant the observation was made, stamped by the emitter, not by the batch writer: a passive event carries the ts of the request it was derived from, and a probe or status_page event carries the moment that poll completed. As with request_logs, gateways older than #1210 left this to default now64(3) and so recorded the flush time, which collapsed a whole sweep onto one millisecond and skewed the uptime/MTTR buckets below.
• source distinguishes where the observation came from: passive (real traffic completing through the request funnel), probe (active liveness sweeps), and the opt-in llm_call / status_page sources.
• outcome is ok / error / timeout; error_kind gives a coarse label (rate_limited, upstream_error, connect_error, timeout).
• Writes reuse the same async, batched, off-hot-path writer and ClickHouse endpoint as request_logs; when no clickhouse_url is configured the sink is a no-op.
• Counters rolter_health_events_written_total and rolter_health_events_dropped_total track the writer, mirroring the request-log counters.
• This event stream feeds uptime %/MTTR rollups and the dashboard health panel.

Passive events are per attempt, not per request

A passive event describes one upstream attempt, not one client request. A request that fails over makes several, and each one is an independent observation of the target it went to.

This used to be per request (#1646). The whole passive funnel — the health event, rolter_target_requests_total{provider,target,outcome} and rolter_upstream_errors_total — was derived from the request log row, whose provider/target are those of the attempt that finally answered the caller. So when a target 503’d a quarter of its requests and the gateway failed over, the record read like this:

Failover doing its job is exactly when an operator most needs to see that a target is sick, so a sick target was indistinguishable from a healthy one on every Health screen and in every metric. Now each superseded attempt is recorded against the target that produced it as it happens, and the same fleet reads ok=23 errors=4 on the target grain with a matching error series in Prometheus — “served 100%, but 25% of attempts to this target failed”.

The request-level row still describes the attempt that answered the caller, and the two never double-count: an attempt the failover funnel already recorded is logged with the request-level target attribution suppressed. request_logs and /api/v1/analytics/summary are unchanged — they are per request by definition, and the client really did get a 200.

Stability rollup API

Read-only, window-bounded rollups over provider_health_events, served by the control plane when --clickhouse-url is set (otherwise 503). All accept since/until (default last 7 days), checked as described in time bounds on the read API and then passed as ClickHouse query parameters, never interpolated.

• GET /api/v1/health/uptime — per provider/target: event counts, uptime, failure_rate, error_budget_burn and sla_breached against an sla target (query param, fraction in (0,1], default 0.99), and last_event.
• GET /api/v1/health/mttr — per provider/target mean time to recovery (mttr_seconds) and incident count, computed from downtime episodes (a run of non-ok events bounded by ok).
• GET /api/v1/health/timeline?bucket=hour|day|week|month — bucketed ok/error/timeout counts per provider/target for the failure timeline (default bucket hour).

Two grains, and why they are never summed

The three rollups group by (provider, target_id), and target_id means two different things depending on the source that wrote the row. A probe or status_page event watches the provider as a whole and carries the provider’s own name as its target_id; a passive event is derived from one completed request and carries that request’s real target. Both grains therefore land in the same rollup, describing the same provider, with numbers that are not comparable — a probe fires every probe_interval_secs regardless of traffic, while a passive observation only exists because a request happened.

Every row consequently reports a grain: provider when target_id is the provider itself, target when it is one route through it. uptime also returns sources, the sorted distinct source values behind the row.

Reading the numbers:

• A provider row answers “is this provider reachable at all, right now?”. Its instant is the last probe or status-page poll, and its denominator is the number of polls in the window.
• A target row answers “what did real traffic through this route see?”. Its instant is the last request that used it, and its denominator is the number of requests in the window.

The two are never added together. The dashboard renders one card per provider, headlined by the provider row when there is one, with the target rows nested inside it; a provider with no probes configured has no provider row, so the card sums its target rows — which is sound, because every request went through exactly one of them — and labels the headline as a roll-up. Before #1257 the screen laid every row out as a peer card, so a single dead provider appeared several times over with contradictory failure counts.

SLA states on the dashboard

The dashboard screen is labelled Provider Health. Its nav key is still circuit-breaker, so old links keep working, but it never reads the gateway’s circuit breakers: those live in gateway memory and leave it only as /metrics counters. Until #2113 the screen called sla_breached tripped, which read as live breaker state when it was a 7-day SLA verdict. Each card and target row carries one of three SLA states instead:

• breaching — sla_breached from the uptime rollup: the window’s failure rate is over 1 - sla.
• atRisk — the window meets the SLA, but the two newest hourly buckets of the timeline fail faster than it allows, i.e. error_budget_burn over those buckets alone is above 1. Over the whole window a burn above 1 is a breach, so a middle state has to come from a shorter span.
• within — neither.

“Newest” is measured from the latest bucket in the timeline response, not from the browser clock: ClickHouse writes buckets as zone-less 2026-08-06 10:00:00 strings in its own zone, which the browser cannot know. A probed provider’s card takes the state of its provider row and buckets; a rolled-up card takes the worst state of its targets, the same rule its breach flag already followed.

Health

• GET /healthz on both binaries for liveness probes.
• GET /readyz on the gateway for readiness. It returns 503 draining once the control plane marks the node as draining (PUT /api/v1/cluster/nodes/{id}/drain), so a load balancer stops sending new traffic while in-flight requests finish; /healthz stays 200 because the process is healthy. The drain reaches the node on the snapshot poll it already makes, and the control plane refuses to drain the last live gateway.

Client disconnects

A caller can leave at any point in a request: a browser tab closes mid-stream, a client-side timeout fires while the upstream is still thinking, a Kubernetes rollout kills the pod holding the connection. The gateway sees this as the response future — or the response body stream — being dropped, and it happens often enough at scale that “undefined” is not an acceptable answer. This page states what rolter does (#1083).

The three guarantees

1. The request is logged exactly once, with status 499. The status is nginx’s convention for client closed request: it is not a real HTTP status, it never reaches a caller (there is nobody left to receive it), and it exists so an abandoned request is distinguishable in analytics from both a success and an upstream failure. The row carries error = "client disconnected".
2. No retry is attempted. A disconnect is the caller’s decision, not an upstream fault. Failing over to a second provider would double the cost of every abandoned request and add load precisely when callers are timing out.
3. The in-flight slot is released. The balancer routes on live in-flight counts, so a leaked slot permanently skews every later routing decision for that model — a silent, restart-only degradation.

Tokens the provider already produced stay on the row. An abandoned stream that received a usage frame is real spend; dropping it would under-bill.

The row names the target the caller left (#1816). The guard is armed before a target is picked, so each attempt of the forward loop calls CancelGuard::attribute(provider, target, variant) as it picks one, and a retry that fails over moves the row with it. The classic loop, forward_variants and the multipart loop all do. A request abandoned before any pick keeps an empty provider, which is how that rarer case stays distinguishable. Without the attribution, the slow target that made callers give up was exactly the one missing from the logs: a dogfood run logged 17 timed-out requests as provider = ''.

A named row is charged to its target through LogSink::log, like any failure: rolter_target_requests_total{outcome="error"} and a passive health event. Whether a disconnect should be charged at all is open in #1933. What is settled is that one upstream failure is charged once. The forward loops funnel a failed attempt that a retry will supersede through CancelGuard::record_failed_attempt (#1646), which also marks the guard, and then back off. A caller who leaves during that backoff still gets a row naming the target, but the guard’s Drop logs it through LogSink::log_recorded_attempt, which skips the per-target counter and the health event, the same way the loop’s own error row does for an attempt it already funnelled. attribute clears the mark, since the next attempt has not been counted anywhere yet.

How it is enforced

Both paths are RAII rather than explicit cleanup on purpose: a cancelled future is never resumed, so anything written as “log after the await” would simply not run.

Metrics

crates/rolter-gateway/tests/client_disconnect.rs pins all of it: a non-streaming caller that times out, a caller that leaves during a retry backoff (one error on the target and one health event, not two), a stream abandoned mid-body, and a normal request that must not be marked or counted.

Billed but withheld

A post-call policy decides what the caller receives. It does not decide what the provider charged. By the time an output guardrail or a post-response plugin runs, the upstream has already generated the answer and billed for it, so a refusal at that point is a delivery outcome, not a billing one. This page states how rolter accounts for that case (#1478).

Before #1478 a blocked response returned its 403 before the accounting stream existed. Spend never reached the budget counters, tokens never reached the rate-limit windows, and no row was written. A caller could spend real money, repeatedly, while every budget and dashboard read zero.

The guarantees

1. Usage is read before any policy runs. On the buffered path (a translation, output guardrails, post-response plugins or the PII sanitizer’s response leg), the gateway parses usage from the upstream body right after dialect translation and before any policy touches it. That value is what gets billed. A guardrail mask, a plugin rewrite that drops or zeroes the usage object, or a sanitizer re-serialization changes what the caller sees and nothing else.
2. A refusal is billed exactly once. When a guardrail or plugin blocks, the refusal goes through the same UsageLoggingStream::finalize as a delivered response, which enqueues spend and token records and writes the row. Nothing else writes that request’s spend. A retried request is billed only for the attempt that answered, because superseded attempts produced no usage and write no row of their own.
3. The row keeps both outcomes. status is what the caller got (403), withheld = 1 marks the divergence, error names the policy (guardrail_blocked: <rule> or plugin_blocked: <reason>), and the token and cost columns hold the provider’s figures.
4. The rejected content is never recorded. The refused body is never handed to the accounting stream, so it can’t reach request_payloads even with payload capture on. The request payload is still captured as usual.
5. Cache hits stay free. A withheld response-cache hit writes a row with cache_hit = 1, withheld = 1 and zero tokens and cost. Nothing was generated upstream for it. The miss that populated the entry was billed when it happened.
6. Unknown usage is not zero usage. usage_unknown = 1 marks a row whose upstream answered successfully without reporting any usage, so its zeros are unknown rather than free. This covers every row, not only withheld ones. The usual cause is an OpenAI-style stream without stream_options.include_usage.
7. A body that never arrives is still logged (#1775). Both buffered paths (the policy branch of stream_response and the cacheable miss in proxy) read the whole upstream body before answering. When that read fails after a 2xx status line, body_read_failed in handlers.rs writes one row: 502, the read error, usage_unknown = 1. It goes through LogSink::log, so the target’s passive health funnel and rolter_target_requests_total{outcome="error"} count it, and rolter_upstream_errors_total moves. It used to return a bare 502 with no row, as if the request had never happened. Covered by withheld_usage.rs against an upstream that closes a 200 mid-body.

rolter_withheld_responses_total counts refusals of both kinds, cache hits included.

How it is enforced

A live SSE stream never reaches a post-call block. Output guardrails either refuse a streamed request before any upstream call (streaming_post_call = "reject", the default) or let it through unmasked ("passthrough"), and post-response plugins are skipped on streams. A stream buffered for the response cache is the exception, and it takes the cacheable-miss path above with its SSE usage frames parsed.

Tests

crates/rolter-gateway/tests/withheld_usage.rs drives the gateway over HTTP against mock upstreams and a stand-in ClickHouse. It covers redacted versus blocked output, a blocking plugin, a plugin that rewrites the usage object, an upstream that reports no usage, and a retried request. The budget-counter and response-cache cases need Redis. They read ROLTER_TEST_REDIS_URL, skip when it is unset, and isolate themselves by a unique org id and cache namespace. The nextest job in quality.yml runs a Redis service for them. To run them locally:

docker run -d --rm --name rolter-test-redis -p 127.0.0.1:26379:6379 redis:8-alpine
ROLTER_TEST_REDIS_URL=redis://127.0.0.1:26379/ \
  cargo test -p rolter-gateway --test withheld_usage
docker rm -f rolter-test-redis

Redis connections on the data plane

Three gateway components keep state in Redis: the budget enforcer (crates/rolter-gateway/src/budgets.rs), the rate limiter (rate_limits.rs) and the response cache (cache.rs). Each holds one multiplexed connection through ReconnectingRedis in crates/rolter-gateway/src/redis_conn.rs. This page covers what happens to that connection when Redis goes away and comes back.

The contract

• During an outage, fail open. Redis unreachable means requests pass unchecked, spend and tokens go unrecorded, and every cache lookup misses. A counter-store outage never takes the data plane down.
• After an outage, recover without a restart. Once Redis is reachable again, the enforcers and the cache that already exist resume on their own. A gateway restart or a snapshot update is not needed. Counters that Redis kept (it persists them, or it never went down and only dropped connections) are enforced again straight away.

Before #1483 only the first half held. The connection sat in a OnceCell, so the first connection was the only one the process would ever make. After Redis closed it (a restart, a failover, CLIENT KILL, an idle timeout on a proxy in between), every later command failed against the dead handle. Budgets and rate limits then stayed failed open, and recording stayed off, until the gateway restarted. That was true even though Redis itself was healthy again.

How a lost connection is replaced

The live connection sits in an ArcSwapOption. On the connected path, taking it is a lock-free load plus a clone of the MultiplexedConnection handle, which is the same cost the OnceCell had.

Commands go through a Lease, which implements redis::aio::ConnectionLike, so call sites pass it to query_async and the AsyncCommands methods as they did before. When a command fails with a connection-level error, the lease evicts its connection from the slot. Connection-level errors are:

• I/O errors, a broken pipe, or a reset
• a response timeout
• an error the client marks unrecoverable
• READONLY from a primary that a failover demoted

A server rejecting one command, such as WRONGTYPE, leaves the connection in place. Each connection carries a generation number, so a failure reported late against an old connection can never evict its replacement.

The next caller that finds the slot empty reconnects straight away. Losing a connection that worked does not count as a failed attempt, so no backoff applies to it.

Which commands are replayed

A command that fails because its connection died is replayed on a fresh connection only if it is read-only. The allowlist is GET, MGET, LRANGE, EXISTS, TTL, PTTL and PING. A pipeline is replayed only when every command in it is on that list.

The budget and cache admission reads are on that list: the budget MGET and the cache GET/LRANGE/MGET. So the first request after a drop is still checked against the counters, rather than failed open.

Writes are not replayed by default. The client library reports “never sent” and “sent, reply lost” as the same BrokenPipe, and replaying an INCRBYFLOAT in the second case would charge one request’s spend twice. A write that meets a dead connection is lost, the same way any write is lost during an outage. The write after it uses the new connection. In practice one request’s spend or token count per consumer can go unrecorded at the moment a connection drops.

A caller can opt a lease into replaying writes with Lease::replay_writes, but only after it has worked out what a double application costs. Rate-limit admission is the one caller that does. It is a single atomic script that both checks and charges, and replaying it can at worst over-count one request in a bucket that expires within two minutes. That is stricter, never looser. The trade-off is argued in Rate limiting.

The redis crate can report a closed socket before the next command is sent, through a synthesized Disconnection push. Rolter does not use it, because the push is only delivered on RESP3 connections. Rolter uses whatever protocol the url asks for (RESP2 by default), so it keeps working behind proxies and servers that do not speak HELLO 3.

Bounded reconnects during a real outage

Connection attempts are single-flight. Callers that arrive while an attempt is in progress wait on an async mutex for that attempt instead of opening their own. The mutex is only taken when there is no live connection, so the connected path never touches it.

Consecutive failed attempts back off exponentially: 100 ms after the first, doubling each time, up to a ceiling of 5 s. While a backoff window is open, no attempt is made at all. Callers get “unavailable” at once and fail open without waiting. Each consumer therefore makes at most one connection attempt per backoff window during an outage, however much traffic the gateway is serving, and a replica makes three in total (budgets, rate limits, cache).

Two timeouts bound how long a caller can wait:

These match the redis crate’s own defaults. ReconnectPolicy sets them explicitly so that a dependency bump cannot change them without anyone noticing.

What operators see

• On the first failure: redis connection lost; reconnecting on next use (warn), with the cause.
• During an outage: redis unavailable; failing open until it is reachable again (warn), with retry_in_ms. This is logged once per attempt, so at most once per backoff window for each consumer, never once per request.
• On recovery: redis connection re-established; enforcement resumed (info).

Each line carries consumer = budgets, rate_limits or response_cache, the same value as the metrics’ consumer label.

On /metrics and over OTLP (#1772), each consumer that has a Redis URL reports:

• rolter_redis_connected{consumer}: 1 while it holds a live connection, 0 once an attempt failed or the connection was evicted;
• rolter_redis_reconnects_total{consumer} and rolter_redis_connect_failures_total{consumer};
• rolter_fail_open_total{control="budget"|"rate_limit"}: admission checks that let a request through because Redis could not be reached, counted where exceeded and check_at give up (no connection, or the read or admission script failed). The response cache backs no control, so it has no series.

ReconnectingRedis holds these as RedisConnStats (plain atomics in metrics.rs) and updates them only on the slow path: in connect and invalidate, and in the consumers’ fail-open branches. A command over a live connection touches none of them. AppState::with_logging registers each consumer and warms its connection in the background (warm_up), so the gauge is truthful from startup rather than reading 0 until the first budgeted request. After that it is what the last use found: an idle consumer does not probe, which the user docs spell out next to the metric.

Tests

The contract is tested against a real Redis. The tests self-skip unless ROLTER_TEST_REDIS_URL is set; see Testing.

• budgets, rate_limits and cache each have a test that closes the component’s own connection with CLIENT KILL. The test then asserts that the same instance enforces against the counters that survived the kill, and that it records through the new connection.
• redis_conn puts a TCP forwarder in front of Redis and takes it down and back up. Those tests cover an outage followed by a restart, the fact that reads are replayed and writes are not, and single-flight connection attempts.
• Tests that point at a closed port cover backoff pacing and the fail-open latency.
• The outage-and-restart test also reads the three connection series back through Metrics::render: connected before, disconnected with failures counted during, reconnected once after. The fail-open window test asserts one counted failure for a burst of 500 callers.

Rate limiting

[[rate_limits]] caps a scope (org, team, project, key, business unit or customer) to rpm requests and/or tpm tokens per minute. The gateway enforces the caps in crates/rolter-gateway/src/rate_limits.rs, using counters in Redis that every replica shares. This page covers two separate things:

• the approximation: how the counters estimate “the last 60 seconds”
• the concurrency guarantee: why concurrent requests can’t all take the same free slot

They are independent. The estimate can be off at the edges of a bucket while admission is still exact under concurrency.

The window: a sliding-window counter

Each limit keeps one counter per fixed one-minute bucket (unix_seconds / 60) and per kind:

rolter:rl:<scope>:<id>:<req|tok>:<bucket>

A check reads the current bucket and the previous one, and weights the previous one by how much of it still lies inside the trailing 60 seconds:

estimate = current + previous × (60 − seconds_into_current_bucket) / 60

On a bucket boundary the previous bucket counts in full. Half way through it counts for half. This is an approximation: it assumes the previous bucket’s requests were spread evenly across its minute.

• Traffic bunched at the end of the previous minute is under-weighted. For example, suppose 60 requests arrived in its last second. Thirty seconds into the next minute they count as 30, although all 60 are still inside the trailing window.
• Traffic bunched at the start is over-weighted by the same logic.

The error is bounded by one bucket’s traffic and disappears as traffic evens out. In exchange, a check costs two reads per limit and no per-request data. The smoothing is still better than a plain fixed window, which would allow 2 × rpm across a bucket edge.

Retry-After on a 429 is the number of seconds left in the current bucket. That is when the weighting next changes in the caller’s favour.

Buckets expire two minutes after their last write, so idle limits clean themselves up.

Admission is one atomic step

A request is admitted only if every applicable limit passes:

• estimate + 1 ≤ rpm
• estimate < tpm (explained below)

The limits are checked in scope-chain order and the first refusal wins (most-restrictive-wins). An admitted request is charged to every applicable rpm bucket. A refused one is charged to none, so a request that the key’s limit rejects costs the org nothing.

The whole decision runs as one Lua script (ADMIT in rate_limits.rs) on the Redis server. The script reads the buckets, evaluates every limit and increments the request buckets. Redis runs a script to completion before it serves any other command, so no request can read a counter between another request’s read and its increment. Every replica talks to the same Redis, so this holds across replicas too.

Before #1484 the gateway ran the same logic as three client-side steps: MGET, decide, then an INCR pipeline. Requests arriving together all read the same counts before any of them wrote, so 32 concurrent requests against rpm = 1 were all admitted. Multiplexing and extra replicas made that more likely, not less.

The script is loaded once per server with EVALSHA. After a Redis restart the script cache is empty, so the first call gets NOSCRIPT, loads the script and retries. The redis crate’s Script::invoke_async handles that.

The script touches keys belonging to several scopes, so it needs a single Redis node (or a primary with replicas). It cannot run on Redis Cluster, where those keys would hash to different slots. The gateway connects with a single-node client in any case.

Tokens are reactive

Token usage is only known once the response has finished. So tpm is read in the same atomic step (the decision sees one consistent snapshot), but it is only charged afterwards, by record_tokens.

A request is refused once the trailing window has already reached the cap. Requests admitted together before that point can overshoot the cap by their own usage. This is the documented reactive behaviour of tpm, not a race: the tokens do not exist yet at admission time, so no atomic step can charge them. Reserving an estimate up front is a separate feature (#1464).

Realtime sessions

A /v1/realtime upgrade goes through the same admission step, keyed on the opening key’s scope chain, so a session costs one request against rpm and is refused while tpm is full. The process-local [realtime] max_connections cap is checked just before it, so a session the gateway would have refused anyway never takes a slot.

A session’s tokens are charged the way a stream’s are, only later: each finished response turn is added to tpm on the session meter’s next flush. A full window does not close a live session, for the same reason it does not cut off an HTTP response in flight. See Realtime metering.

When the Redis connection drops

Admission is the one write that opts out of the no-replay rule in Redis connections. If the script call meets a dead connection, it is replayed once on a fresh one, so the first request after a drop is still enforced.

The replay is exact when the first attempt never reached Redis, which is the usual case: the connection was already dead. If the script did run and only its reply was lost, the replay charges one extra request to a bucket that expires within two minutes, and may refuse a request that the lost reply had admitted. That errs by one towards stricter, for one window. The alternative is to admit the request unchecked and uncounted, and for a throughput control that is the worse failure.

Spend recording stays on the no-replay side. A double-applied INCRBYFLOAT would corrupt a budget for its whole period.

While Redis is unreachable, admission fails open, like every other Redis-backed control.

Tests

Tests against a real Redis live in rate_limits.rs. They self-skip without ROLTER_TEST_REDIS_URL; see Testing.

• 64 concurrent requests against rpm = 1 admit exactly one.
• Three limiters with separate connections, standing in for three replicas, race 96 requests against rpm = 5. Exactly 5 are admitted and exactly 5 are counted.
• The org/key scope chain under concurrency: the key’s cap wins, and the org is charged only for what the key admitted.
• Window boundaries, with the clock pinned through check_at: the previous bucket counts in full on the boundary and for half at mid-minute.
• A full tpm window refuses and charges no request slot.
• Recovery after CLIENT KILL: the first check after the drop is enforced through the replayed script.

Realtime metering

A /v1/realtime session is a WebSocket that can stay open for an hour and run hundreds of model responses. Until #1396 it was admitted against the process-local [realtime] caps and nothing else: no budget, no rate limit, no request-log row. This page states how a session is admitted, how it is metered and what happens when its budget runs out.

The code is in crates/rolter-gateway/src/realtime.rs (admission and the relay loop) and crates/rolter-gateway/src/realtime_metering.rs (the meter).

Admission

The upgrade request goes through the same checks as an HTTP request, keyed on the same scope chain (key, project, team, org, plus the business unit and customer the key is attributed to). They run before any upstream is dialled:

Opening a session is one request against rpm. The process-local cap is checked first so that a session this gateway could not hold anyway never costs the key a slot. The tokens a session’s turns use reach tpm as they are metered, so a key that spent its token window over a socket is refused its next session, or its next HTTP request, like any other usage.

A full tpm window does not close a live session. Rate limits refuse new work, and the HTTP path does not cut off a response in flight either.

The metering unit: the response turn

A session is metered in the unit the upstream bills and reports. Every response.done server event carries the usage of the response it ends, with input_tokens, output_tokens, total_tokens and input_token_details.cached_tokens, and that is what gets priced against the route model’s [[model_prices]] row.

The alternatives were weighed and rejected:

• Metering on close leaves a session that never closes cleanly unaccounted until it is killed, and a session that ends because the network dropped may never be accounted at all.
• Metering wall-clock time bills something no provider charges for, and cannot be reconciled with a provider invoice.
• Estimating from audio bytes duplicates the provider’s own tokenizer badly, while the provider reports the exact count on every turn.

Reading the event stream cheaply

The relay hands every upstream text frame to a TurnTracker. Almost every frame is a delta carrying base64 audio, often tens of kilobytes, so parsing each one would put a JSON decode of the whole audio stream on the relay’s path. The tracker reads the event type from the leading bytes of the object instead (the first key, or the second behind event_id, which are the two places the Realtime API writes it) and fully parses only response.created and response.done, which are small. A frame whose type sits anywhere else falls back to a substring scan: slower, but still metered correctly.

response.created opens a turn and starts its clock, and response.done closes it. Time to first token is stamped by the first response.*.delta that names the turn in its response_id, or by the first one naming none. Other deltas do not count: conversation.item.input_audio_transcription.delta streams the user’s own words, often well before the model answers. A done whose created was never seen is still billed, with zero latency.

The periodic flush

A closed turn goes to the session’s meter task over a bounded channel. The relay never waits on Redis or ClickHouse; it carries live audio.

Every [realtime] usage_flush_secs (default 15), and once more when the session ends, the meter flushes:

1. one request_logs row per turn, attributed like an HTTP row: org, team, project, key, business unit, customer, model, provider, target and variant. Its request_id is the upgrade request’s id with the turn’s ordinal appended (<id>:1, <id>:2, …), so every row is unique and one session’s rows share a prefix and a trace_id
2. the window’s cost added to every applicable budget counter, in one write
3. the window’s tokens added to every applicable tpm window
4. a re-read of the session’s budgets

Rows are per turn rather than per window so that latency, time to first token and token counts in analytics describe one model response, as they do for HTTP. The flush window only batches the counter writes and bounds how late enforcement can be.

usage_flush_secs = 0 flushes after every turn instead of on a timer. The meter still ticks once a second in that mode, but only to re-read the budgets while the session is quiet (see the next section), so 0 is the tightest setting on both counts. There is no setting that turns metering off.

The snapshot is read per flush, so a price, budget or limit edited mid-session applies from the next flush on.

A budget that runs out mid-session

The meter re-reads the budgets on every tick, not only after this session spent something: a budget is shared by its whole scope chain, so another session or plain HTTP traffic may be what used it up. A tick is one usage_flush_secs, or one second when that is 0. The re-read costs nothing when no budget applies to the session or Redis is not configured.

When a budget is spent the session is closed. The client first receives a Realtime error event whose error object is the HTTP refusal’s:

{
  "type": "error",
  "event_id": "event_rolter_…",
  "error": {
    "type": "insufficient_quota",
    "code": "insufficient_quota",
    "message": "budget exceeded for Org 'acme' (limit $500.00)",
    "param": null
  }
}

It then receives a close frame with code 1008 (policy violation), and the upstream leg is closed. rolter_budget_blocks_total counts the close.

Closing is the conservative choice of the three the issue named:

• Degrading has nothing to degrade to. The session is pinned to one upstream and model for its life, and switching mid-session is exactly the failover the relay refuses to do because replaying audio or tool events is unsafe.
• Allowing the session to completion turns a budget into a suggestion for an hour at a time, which is the bug #1396 was filed about.

The overshoot is bounded: at most the spend of one flush window, plus whatever turn was in flight when the budget ran out. An operator who needs a tighter bound lowers usage_flush_secs.

Responses cut short

A response still in flight when the session ends never reports usage, but the upstream generated part of it and will bill it. The last flush writes a row for it with usage_unknown = 1, zero tokens, and the reason the session ended as its status:

This follows Client disconnects and Billed but withheld: spend that cannot be counted is marked as unknown, never recorded as free. A row cut short by the client also counts on rolter_client_disconnects_total.

These rows do not count against the target’s health. A client leaving, a session limit or a spent budget is not the upstream’s failure, and when the upstream leg did break, the relay has already attributed that to the target once for the whole session.

A turn whose response.done says status: "failed" is logged with status 502. Its usage is billed if the upstream reported any.

Shutdown

A session outlives the HTTP request that opened it. axum’s graceful shutdown stops tracking a connection once it upgrades to a WebSocket, so on its own it would let the process exit under live sessions. The runtime would then cancel each relay and meter wherever it stood, and every turn still waiting for the next flush would lose its row, its budget charge and its tpm tokens. With the default usage_flush_secs that is up to 15 seconds of every session, on every rollout.

So the gateway drains realtime sessions itself (Sessions in realtime.rs, AppState::drain_realtime_sessions):

1. Every relay and meter task is tracked from admission on, including the moment between the upgrade response and the completed handshake.
2. When SIGTERM or Ctrl-C arrives, every session is told to close at once, in parallel with axum’s drain of plain HTTP requests. Each client receives a 1001 (going away) close, the code a WebSocket server sends when it stops, so an SDK reconnects rather than treating the session as refused. The upstream leg is closed too.
3. New upgrades are refused with 503 from that moment.
4. Each meter runs its last flush: rows for finished turns, usage_unknown rows with status 503 for responses in flight, budget and tpm writes.
5. Once the HTTP drain is done, the process waits up to 10 seconds for the tracked tasks to finish, then exits. The sessions were closed when the signal arrived, so the wait only covers a meter still waiting on Redis, and it fits well inside the 30 seconds an orchestrator usually allows before SIGKILL. A session still open after it is logged as a warning.

The budget and tpm writes are awaited by the meter, so they land before the process exits. The request-log rows are handed to the shared ClickHouse writer, which is not drained at shutdown yet, so rows from the last [logging] flush_ms can still be lost. That gap is not specific to realtime (#1924).

Failure modes

• Redis down. Admission and recording fail open, as on the HTTP path: the session is admitted, turns are still logged to ClickHouse, and spend is not counted until Redis is back. See Redis connections.
• The meter falls behind. A turn that finds the meter’s queue full is dropped and counted on rolter_usage_records_dropped_total, the policy the HTTP usage sink already has (#1051). The queue holds 1024 turns and each flush is bounded by the Redis timeouts, so reaching it needs a thousand responses inside one stalled flush.
• The relay task dies. The meter sees its channel close, flushes what it was already handed and stops.
• The process shuts down. Covered by Shutdown. A SIGKILL, or a drain that outlives its 10 seconds, still loses whatever the meter had not written.

What is not metered

These are tracked rather than silently missing:

• Guardrails and plugins do not run on realtime events (#1880). This is why realtime still carries its stability marker: the marker’s note named guardrails alongside metering, and a subsystem graduates in the pull request that closes the last gap its note names.
• A revoked or expired key does not end a live session (#1881).
• Audio and text tokens are priced at the same rate, because a price row has one input and one output rate (#1882).
• Input-audio transcription runs a second model whose usage arrives on conversation.item.input_audio_transcription.completed, which the meter does not read yet (#1883).

Tests

crates/rolter-gateway/tests/realtime_metering.rs drives real sessions against a mock Realtime upstream: a completed turn is logged with its usage and cost, a session that stays open is accounted on the flush timer, a response cut short is logged as unknown, turns waiting on the timer are flushed when the session ends, an unpriced model is refused under block, a spent budget refuses a new session, a session is closed when its budget runs out, a budget spent by other traffic closes an idle session on the flush timer and, with usage_flush_secs = 0, on the one-second budget tick, usage is charged while the session continues, rpm applies per key rather than per process, and turn tokens fill the key’s tpm window. The Redis-backed ones read ROLTER_TEST_REDIS_URL and skip, or skip their Redis assertions, without it.

Shutdown has three: an in-process drain flushes every live session before it returns and refuses new ones with 503, a response in flight at shutdown is logged as unknown, and a real rolter-gateway child sent SIGTERM closes its session with 1001 and exits 0 only after the meter has flushed. With Redis, the first and the third route the gateway through a relay that holds every reply back by 200ms once shutdown starts. A flush charges a budget with INCRBYFLOAT and sends its EXPIRE only after that reply, so a budget key that carries its expiry proves the meter ran to the end of its flush instead of merely starting it. Sessions itself is unit-tested for holding the drain while a meter or an upgrade is still running.

The tracker’s frame handling, including which deltas count as a first token, is unit-tested in realtime_metering.rs itself.

Performance

Goal: beat the reference Python proxy (LiteLLM cites ~8ms P95 added latency at 1k RPS) with a much smaller per-request overhead in Rust.

Hot-path principles

• Lock-free config reads — the routing table is an ArcSwap<Snapshot>; readers never block, even during a hot reload.
• Minimal-copy streaming — upstream responses are piped to the client as a Body::from_stream over the reqwest byte stream; rolter does not buffer whole responses.
• Connection reuse — pooled reqwest clients with HTTP/2 keep-alive and tcp_nodelay; one client per egress-proxy target, cached.
• Cheap auth — virtual-key lookup is an O(1) hash-map hit on the in-memory snapshot.
• Avoid full deserialization — only the fields needed for routing (model, stream) are read; the body is forwarded as raw bytes.
• Logging off the hot path — usage/cost rows are batched and written to ClickHouse asynchronously.
• Release profile — lto = "thin", codegen-units = 1, strip = true.

Things to watch

• The approximate cache-aware trie is per-route in-memory state; bound its size with eviction before it grows large.
• Per-request JSON parse for model is small but measurable; consider a fast path / partial parse for very high RPS.
• Prefer bytes::Bytes (ref-counted) over Vec<u8> copies when rewriting the model field.

Benchmarking

Three layers. Only the third is on the per-PR gate:

• Micro-benches (just bench) — criterion, covering balancer pick across every strategy and the prefix trie. Compiled on every PR via clippy so they cannot bit-rot, but not run (timings are noisy on shared runners).
• End-to-end load (just bench-sim / bench-vllm / bench-sglang) — integration/engines/bench.py against a real engine, measuring rolter’s added latency directly and its behaviour under sustained concurrency.
• Allocation counting (cargo test -p rolter-gateway --test hot_path_allocations) — asserts that the per-attempt admission checks make zero allocations. This one is a merge gate, because it measures a property rather than a duration.

Why allocation counting is a test and not a benchmark

“Keep the hot path allocation-light” is a rule this document states and AGENTS.md repeats, but timings alone cannot enforce it: a per-request allocation costs tens of nanoseconds, which is well inside the noise of a shared CI runner. That is why the criterion benches are compiled and not run — and it means a regression like the (String, usize) key that Breaker::allows and Cooldowns::is_parked once built could come back without anything failing.

crates/rolter-gateway/tests/hot_path_allocations.rs installs a counting #[global_allocator] and asserts an exact zero on the steady-state paths. Zero is deliberate: an allocation budget would be a portability trap across allocators and toolchains, whereas “this path does not allocate at all” is both stable and the property actually wanted. A count is exact and reproducible in a way a duration is not, so this can gate where the benches cannot.

Two details worth knowing before extending it:

• The counter is per-thread, not global. Cargo runs a binary’s tests on parallel threads, so a process-wide counter attributes a sibling test’s ordinary allocations to whichever measurement happens to be open, and fails it for something it never did. Arming and counting are both thread-local, and a test asserts that property directly.
• It is verified against an injected regression, not just observed passing. A guard that cannot fail is worse than none, since it reads as evidence. Put a model.to_string() back into Breaker::allows and exactly the two breaker tests fail, with one allocation per candidate target; the cooldown tests, which touch none of the changed code, keep passing.

To extend it to another hot-path component, add a test that warms any lazily built internals first — otherwise you measure first-call setup — then wraps the steady-state operation in counting(...) and asserts zero.

Tool decision (#847, closing #455)

The load harness extends bench.py rather than adopting GuideLLM or driving load with oha/k6. Recorded here because #847 required settling it before writing code.

• Air-gapped operation is a hard requirement for rolter. bench.py is stdlib-only — no install step, no wheels or binaries to vendor. GuideLLM brings a substantial Python dependency tree; oha/k6 are external binaries. A benchmark you cannot run on the disconnected machine you are tuning is not much of a benchmark.
• The direct-vs-rolter delta is the entire point of this harness, and no general-purpose load generator produces it. They measure one endpoint; added_latency_p50_ms needs both driven identically in the same run under the same conditions. Adopting one of them means keeping bench.py anyway for the delta — two harnesses instead of one.
• ITL needs SSE token boundaries. oha and k6 measure bytes and requests, not tokens, so they cannot produce inter-token latency at all. GuideLLM does understand tokens, but the two points above still stand.
• Reproducible without credentials — the default sim engine (llm-d-inference-sim) needs no API key, so anyone can reproduce a number.

The cost is carrying ~250 lines of stdlib Python for concurrency and knee detection. That was judged cheaper than a dependency tree plus a second harness. Revisit if the harness starts needing tokenizer-aware workload shaping, which is where GuideLLM genuinely earns its footprint.

What it measures

bench.py drives the engine directly and through rolter in the same run:

Profiles: --concurrency holds closed-loop steady state after a warmup, and --sweep 1,2,4,8,16 walks concurrency upward — a ramp/burst profile — reporting max sustainable RPS: the highest achieved throughput whose p99 stayed inside --knee-factor (default 2×) of the lowest-concurrency baseline and which was not erroring.

That bound matters. Throughput usually keeps climbing well past the point where latency collapses, so a peak RPS with no latency condition attached describes a system nobody would actually run. The report also names the concurrency level where the knee was crossed, so a reader can see where it went rather than only that it did.

just bench-sim     # sequential: per-request added latency (fast, unchanged)
just load-sim      # concurrency sweep: max sustainable RPS, ITL, error rate
just test-bench    # harness unit tests — stdlib only, no engine or network

load-* is deliberately a separate recipe: a sweep across five concurrency levels takes minutes, and silently making the added-latency run that much slower would be a bad trade. Tune with LOAD_LEVELS, LOAD_REQUESTS, LOAD_WARMUP and LOAD_MAX_TOKENS.

Two measurement rules worth knowing when reading the JSON:

• Failed requests are counted, never timed. A connection refused in 0.1 ms is not a fast request; letting it into the percentiles would make a saturated system look quicker than a healthy one. They appear in error_rate instead.
• ITL keys are absent, not zero, when nothing streamed. A reported ITL of 0 would read as “instant tokens” rather than “not measured”.

Baseline

No baseline snapshot is recorded yet. It needs a run on fixed, documented hardware, and a number taken on a laptop under thermal throttling would be worse than none — an unreproducible baseline invites false regressions. Record hardware, engine version and rolter version alongside the numbers when one is taken.

Inference engines

The ROL-238 suite checks rolter against OpenAI-compatible engine servers. Output is intentionally meaningless; this validates the HTTP, OpenAI JSON, and SSE contracts rather than model quality.

The default engine, sim, is llm-d-inference-sim: a ~30MB multi-arch vLLM API simulator that needs no model downloads and boots in milliseconds, which makes the suite cheap enough to run as a regular PR check. The real CPU vLLM and SGLang profiles remain for on-demand runs; they use trl-internal-testing/tiny-random-LlamaForCausalLM only for its configuration/tokenizer and initialize random weights with --load-format dummy (with a head_dim=64 override, since the CPU attention kernels reject the model’s native head_dim=4). The CPU vLLM profile uses eager execution to avoid expensive compilation warm-up during CI smoke runs.

It runs on CPU in Docker and therefore works on GitHub-hosted runners. Each engine profile starts two independent dummy upstreams so the gateway exercises a real target pool. Run one engine locally:

just integration-sim
just integration-vllm
just integration-sglang

Each command boots routes for every balancing strategy (round-robin, random, power-of-two, consistent-hash, cache-aware, weighted, pipeline, cheapest, and fastest). It verifies /v1/models, non-streaming chat, and SSE both directly and through rolter, and explicitly confirms round-robin reaches both targets. Logs are kept in artifacts/engines/<engine>/.

Local end-to-end run

just integration-vllm and just integration-sglang start two CPU dummy-weight servers, render the gateway configuration, and run the OpenAI JSON and SSE assertions. The runner cleans up all containers and child processes on exit and preserves the combined engine log plus the gateway log under artifacts/engines/<engine>/.

For manual inspection, start the selected two-server pool and leave it running:

docker compose -f docker/docker-compose.engines.yml --profile vllm up -d
# use profile sglang for ports 30000 and 30001

Render the gateway configuration in another terminal and start rolter:

config=$(mktemp)
sed \
  -e 's/__ROLTER_PORT__/4010/g' \
  -e 's/__ENGINE_1_PORT__/8000/g' \
  -e 's/__ENGINE_2_PORT__/8001/g' \
  integration/engines/rolter-dummy.toml.in >"$config"
cargo run -p rolter-gateway -- --config "$config"

Verify non-streaming JSON and streaming SSE through the gateway:

curl -i http://127.0.0.1:4010/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"dummy-round-robin","messages":[{"role":"user","content":"Reply with one token."}],"max_tokens":1,"temperature":0}'

curl -N http://127.0.0.1:4010/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"dummy-round-robin","stream":true,"messages":[{"role":"user","content":"Reply with one token."}],"max_tokens":1,"temperature":0}'

Exercise all configured strategies; each request must return a non-empty OpenAI choices array:

for model in dummy-round-robin dummy-random dummy-power-of-two \
  dummy-consistent-hash dummy-cache-aware dummy-weighted dummy-pipeline \
  dummy-cheapest dummy-fastest; do
  curl -fsS http://127.0.0.1:4010/v1/chat/completions \
    -H 'content-type: application/json' \
    -d "{\"model\":\"$model\",\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}],\"max_tokens\":1}" \
    | jq -e '.choices | length > 0' >/dev/null
done

Clean up the pool and temporary gateway configuration:

docker compose -f docker/docker-compose.engines.yml --profile vllm down --volumes
rm -f "$config"

For non-gating direct-versus-gateway samples, run just bench-vllm or just bench-sglang. They record non-streaming and streaming p50/p95/p99 latency and streaming first-byte time in JSON. Results only compare runs on the same host, CPU image, engine versions, and host configuration; throughput thresholds are deliberately not merge gates.

The engine integration workflow runs the sim smoke as a regular check on pull requests that touch engine paths. A newer push to the same pull request cancels the smoke still running for the older commit; a manual dispatch is never cancelled by another run. Dispatch it manually with engine=vllm to smoke the real CPU engine (Actions tab, or gh workflow run "engine integration" -f engine=vllm). SGLang remains available through the local just integration-sglang command, but its source-built CPU image is currently too heavy for the shared CI gate. This suite is for compatibility, not a performance gate. When ROL-67 lands, add the equivalent /v1/messages assertion through the gateway.

Provider prompt caching

Rolter’s response cache and provider prompt caching solve different problems. The response cache replays a complete prior answer from Redis. Provider prompt caching lets an upstream reuse stable input context while it still generates a fresh answer.

Use the portable cache_control object only on requests routed to a direct Anthropic provider:

{
  "model": "claude-route",
  "messages": [{"role": "user", "content": "Summarize this document."}],
  "cache_control": {
    "enabled": true,
    "ttl": "5m",
    "breakpoints": ["system", "tools", "messages"]
  }
}

Rolter converts each selected breakpoint to Anthropic’s native ephemeral cache_control marker. ttl is limited to 5m or 1h; omitted breakpoints defaults to system. Existing nested provider-native controls are preserved.

bedrock and vertex providers currently use their OpenAI-compatible wire mode in Rolter. They reject this portable control with prompt_cache_unsupported instead of silently forwarding it or claiming cache use. Configure a direct Anthropic provider for the portable contract until native Bedrock/Vertex Claude transports are added.

Prompt-cache reads and writes are provider billing/usage concepts. They are distinct from the invocation log’s cache_hit, which only denotes a Rolter response-cache hit.

Architecture Decision Records

Lightweight decision log. Each entry: Status · Context · Decision · Consequences. Supersede rather than rewrite.

A standalone record opens with its title and a single metadata line — no table:

# Title of the decision

**Status:** Accepted · **Date:** 6 Aug 2026 · **Issues:** [#812](https://github.com/rolter-ai/rolter/issues/812)
**Supersedes:** ADR-0007 for exact vLLM modes
**Relates:** ADR-0022 (config-vs-DB tiering)

## Context

Status, Date and Issues are the line; Supersedes and Relates get their own line and are omitted when there is nothing to say. Nothing else belongs there — authorship and dates are what git is for, and a field whose value is either constant across every record or a — (unassigned) placeholder is noise that makes the actual decision harder to find. Records are English only.

ADR-0001 — Rust + Axum/Hyper/Tower for the data plane

Accepted. Need maximum proxy throughput with rich API semantics and SSE streaming. Chose Axum/Hyper/Tower on Tokio over Pingora/Actix for ecosystem fit and ergonomics. Consequence: idiomatic async stack; revisit Pingora only if profiling demands it.

ADR-0002 — Two-binary topology over shared crates

Accepted. Keep the hot proxy path lean and independently scalable from management. rolter-gateway (data plane) and rolter-control (management + UI host) share library crates. Consequence: clear seam; some duplicated wiring.

ADR-0003 — Vite + React + shadcn/ui SPA, Bun toolchain, served by Rust

Accepted. shadcn/ui (Radix + Tailwind) for the dashboard, built with Vite and managed with Bun; output is static assets served by rolter-control (no Node runtime in prod). Consequence: simple prod footprint; Bun used for install/dev/build.

ADR-0004 — Postgres + Redis + ClickHouse

Accepted. Postgres = source of truth (config/RBAC/keys/pricing); Redis = cache + rate-limit counters + config pub/sub; ClickHouse = high-volume request/cost logs. No SQLite. Consequence: three datastores to operate; each fits its job.

ADR-0005 — Org → Team → Project → Virtual Key tenancy

Accepted. Budgets and rate limits attach at any scope, most-restrictive-wins. Consequence: flexible multi-tenancy; enforcement must resolve a scope chain.

ADR-0006 — Local accounts + virtual keys + roles; SSO/LDAP later

Accepted. v1 ships local accounts (argon2id) and roles admin/member/viewer; OAuth2/OIDC and LDAP arrive as pluggable identity providers. Consequence: usable day one without an IdP.

ADR-0007 — Approximate cache-aware balancing behind a pluggable trait

Accepted. v1 uses an approximate per-target prefix trie (no engine coupling) behind LoadBalancer. Precise (KV-events) and lmcache-aware land later without API changes. Consequence: immediate wins; precise mode is additive.

ADR-0008 — Reload-free config: Postgres truth + Redis pub/sub + ArcSwap

Accepted. Control plane writes Postgres, bumps a version, publishes on Redis; gateways fetch and atomically swap an in-memory snapshot, reconciling by version. Consequence: instant fan-out, self-healing, lock-free reads.

ADR-0009 — Envelope encryption for provider secrets

Accepted. Upstream keys are AES-256-GCM envelope-encrypted with a master key from env/file; Vault/KMS backends later. Consequence: no plaintext secrets at rest; master key is the critical secret.

ADR-0010 — Packaging: maturin (uv) + cargo + Docker

Accepted. Ship a maturin-built PyPI wheel bundling the unified rolter launcher (uv tool install rolter), cargo install rolter, and a multi-stage Docker image. The rolter binary dispatches to gateway/control subcommands so one wheel/crate ships the whole system. Consequence: three distribution paths from a single named artifact.

ADR-0011 — API surface v1

Accepted. OpenAI /v1/chat/completions, /v1/completions, /v1/models and Anthropic /v1/messages. Embeddings, images, audio and other modalities follow. Consequence: drop-in for the two dominant client SDKs first.

ADR-0012 — Conventional Commits + CI PR-title lint

Accepted. Commit messages and PR titles follow Conventional Commits; enforced by commitlint, conventional-pre-commit, and a CI PR-title check. Consequence: consistent history, automatable changelogs/releases.

ADR-0013 — OpenTelemetry-based observability with engine propagation

Accepted. Export traces/metrics via OTLP to any compatible backend (SigNoz, Datadog, Grafana, Langfuse, …); propagate W3C trace context to vLLM/SGLang so engine spans join the same trace; federate upstream engine metrics. Consequence: vendor-neutral observability; an OTel Collector is the recommended hub.

ADR-0014 — Extensible API protocol translation

Accepted. Resolve translation by client/upstream protocol pair in rolter-proxy, including incremental SSE, while the gateway retains transport, caching and accounting ownership. Consequence: new provider dialects extend one translation boundary; non-equivalent modalities remain explicit and are never silently dropped.

ADR-0015 — OpenAI Responses API translation

Development. Add OpenAI Responses as a protocol pair for native OpenAI, Chat Completions and Anthropic Messages, while model-less lifecycle operations remain uniformly unsupported until tenant-scoped storage exists.

ADR-0016 — Routing OpenAI Responses resources through a tenant-scoped registry

Development. Pin Responses lifecycle operations to a bounded tenant-scoped process-local record, preserving the original provider credential while making unknown and cross-tenant IDs indistinguishable.

ADR-0017 — Provider/model addressing to disambiguate identical model names

Accepted. First-class provider-slug/model addressing coexists with named routes: a stable, URL-safe provider slug resolves provider-slug/model to a pinned (provider, upstream_model) target (avoiding LiteLLM’s base_url ambiguity). Pinning bypasses cross-provider fan-out but still balances within the provider. Slugs are org-scoped and collision-safe: deterministic reported migration backfill with -N de-dup, 409+suggestion on runtime create-conflict (never silent suffixing), soft-delete-preferred reclaim. Addendum: provider groups unify fleets (e.g. ten vLLM instances) under one group slug in the same namespace — vllm-cluster/qwen3 fans out across members with a chosen balancing strategy and passthrough model names, so no route-per-model. Consequence: new immutable slug column + proxy parsing + /v1/models + UI work + provider-groups entity (see follow-up issues).

ADR-0018 — Config mutated via granular CRUD, not whole-config replace

Accepted. The normalized Postgres store (ADR-0004, source of truth) is mutated only through the scoped CRUD API — providers, routes, targets, virtual keys — each write bumping the config version and hot-reloading gateways via the snapshot poll (ADR-0008). PostgresConfigStore::save is deliberately read-only: there is no whole-config “apply” endpoint, because a full replace would fight the normalized model and clobber concurrent edits. The dashboard Config page is a read-only effective-config viewer; the /gw reverse-proxy that fronts the gateway for the Playground is unrelated to config writes. Consequence: one live-edit path (CRUD) with instant fan-out; a raw whole-config editor would need a deliberate transactional diff/apply (guarding config-owned entries, preserving key hashes) and is out of scope (closed #494).

ADR-0019 — Per-provider egress proxy pools

Accepted. Rotate across a provider-local proxy pool, fail over only connection/tunnel failures, and quarantine repeatedly failing members. Authenticated URLs are resolved exclusively from whole-value environment references. Consequence: resilient egress without leaking credentials; health state remains process-local.

ADR-0020 — Bounded semantic response caching in Redis

Accepted. Run semantic lookup only after an exact miss, embed through an explicitly configured provider, and scan a bounded recent Redis window. Consequence: similarity reuse without another datastore; embedding and cache failures fail open and candidate search remains deliberately bounded.

ADR-0021 — External cache telemetry for routing

Accepted. Extend ADR-0007 with opt-in vLLM KV-event and LMCache occupancy scorers whose network I/O runs in background tasks. Consequence: exact/capacity-aware routing stays allocation-light and fail-open; stale or untrusted telemetry becomes neutral and least-load selection remains available.

ADR-0022 — Uniform config-vs-DB tiering for models, providers, and provider groups

Accepted. Give models, providers, and provider groups the same two-tier config wrapper over the DB (CRUD) tier: readonly entries are immutable and config-owned; default entries are seeded into the default project once at startup and then owned/editable via API/UI; pure DB rows come from CRUD. readonly wins resolution; a default colliding with a readonly key is a load-time error. Top-level [[routes]]/[[providers]]/[[provider_groups]] stay as deprecated readonly aliases for back-compat. Consequence: providers gain a seed-then-edit story and provider groups gain a full DB lifecycle (new tables/repos/seed/CRUD, tracked as follow-up PRs); the model tiering pattern is reused rather than reinvented per entity.

ADR-0023 — Propagating access-profile model policy to the data plane

Accepted. Resolve each virtual key owner’s merged access-profile policy when the snapshot is built, carry it on the key record, and enforce it on the gateway’s model and route selection; the key’s own allow-list and the owner’s policy must both permit a model. Rejects key-mint-time resolution because a later policy edit would never reach issued keys, so a revocation would silently fail to revoke. Adds bump_config_version() triggers to exactly the four tables that feed the resolution, changing migration 0058’s premise rather than its rule. Consequence: the policy /rbac/effective reports is the policy enforced, and a deployment with no access profiles is unaffected.

ADR-0024 — Dashboard UX telemetry as a structural-only event stream

Accepted. Collect dashboard usability events into a ui_events ClickHouse table beside request_logs rather than adding a product-analytics vendor, and make the schema itself the privacy guarantee: every column is a key, an enum, a duration or an id, so a form value or prompt has nowhere to land. Attribution is server-side; the ingest endpoint is guarded by CurrentUser with no capability row, because a create capability granted to any authenticated caller would break the RBAC invariant that a viewer writes nothing. Consequence: usability data with no new vendor, egress or retention policy, joinable to gateway traffic on trace_id; the cost is that any question needing a value rather than a key is unanswerable by design.

ADR-0025 — Events as logs, not span events

Accepted. OpenTelemetry is deprecating the Span Events API, so log-based events become rolter’s event model: no new explicit add_event/record_exception, and #809 (OTLP log export) sequences before #808 (GenAI conventions) because those conventions are specified as log-based events and cannot be emitted until logs are exported at all. A grep finds zero uses of the deprecated API, but tracing-opentelemetry turns every tracing event fired inside a span into a span event, so rolter is in practice an exclusive implicit user of it — which makes the migration a dependency upgrade in one bridge rather than a sweep of 57 call sites. Consequence: tracing stays; #809’s trace_id/span_id correlation becomes a blocking acceptance criterion rather than a nicety; new event types have no correct home until it lands, and should wait rather than add a span event that must later be removed.

ADR-0026 — Client control over telemetry: a kill switch now, collector-routed tenant destinations later

Accepted. Splits #812 in two. ROLTER_TELEMETRY_ENABLED ships now as one explicit off for every signal — traces, metrics, the dashboard’s browser tracing, and logs once #809 lands — that can only subtract, defaults to enabled so no existing deployment changes, and leaves export on for an unrecognized value so a typo cannot silently blind a deployment. It is environment-only despite the issue asking for a config key: telemetry::init installs the subscriber before any config file is read, so a config-file switch could not gate trace export at all. Per-tenant destinations are deferred and, when built, belong in the OpenTelemetry Collector’s routing processor keyed on an org resource attribute rather than in-process exporters — that keeps one egress path in the gateway and keeps tenant-supplied URLs, an SSRF surface, out of the data plane entirely. Consequence: operators get a reviewable “no telemetry leaves here” today; per-tenant routing later requires running a collector, which is already the recommended topology.

ADR-0027 — End-to-end test harness: Python/uv project driving a black-box stack

Accepted. The e2e suite is a Python driver on the latest CPython managed by uv, exercising the real HTTP APIs of a docker-compose stack (postgres, redis, clickhouse, control, gateway, and N llm-d-inference-sim fake-vLLM engines) with no in-process shortcuts, so the tests see exactly what an operator or tenant sees. Numbered after ADR-0026 despite its earlier date because ADR numbers are append-only and never reused. Consequence: the suite needs a container runtime rather than cargo test alone; in exchange it covers the wiring between the two binaries and the three datastores that unit tests cannot reach.

ADR-0028 — Disaggregated prefill/decode routing belongs to the engine, not the gateway

Accepted. rolter will not implement P/D disaggregation; a disaggregated fleet is one upstream and its own coordinator owns phase selection and the KV handoff. Engines move KV tensors through connector-selected transports such as NIXL/UCX. Some can coordinate that transfer through engine-specific HTTP fields, but rolter would still have to own compatible-worker topology, version-specific metadata, two upstream calls and per-request handoff state — exactly the lifecycle coupling ADR-0014 prevents. The distinguishing test is whether rolter can consume an input through a stable, engine-independent contract without joining the engine’s request lifecycle. Consequence: disaggregated fleets work with rolter today for any engine, with no code and no mid-request-handoff failure modes; revisit if a stable phase-placement contract emerges on the OpenAI or Anthropic surface.

ADR-0029 — An explicit per-provider opt-out from a hosted kind’s host pin

Accepted. ProviderKind::Openrouter pinned api_base to the literal https://openrouter.ai/api/v1, which closes a real exfiltration path — a hosted kind names one operator, so an edited base URL would send that operator’s key elsewhere — but also ruled out an egress proxy, a vendor-compatible endpoint, and testing the dialect at all, so the #924 dogfooding fleet had to declare the endpoint openai_compatible and left the adapter unexercised. Adds allow_custom_api_base = true per provider: the safe default is unchanged, the opt-out is one reviewable line, only the host pin is relaxed (the api_key_env rule still holds), and rolter check warns on every provider that sets it while --strict fails. Rejects a loopback/private-range carve-out, which would not help a corporate proxy and would key the rule on where rolter runs rather than on intent. Consequence: the OpenRouter dialect gets local coverage; providers stored in Postgres keep the safe default until the column, CRUD surface and dashboard control land as their own change.

ADR-0030 — A served OpenAPI document for the control-plane API

Accepted. The gateway already serves a hand-built OpenAPI 3.1 document at /openapi.json with an embedded Scalar reference at /docs; the control plane serves the same for its ~160-path management API, built from one operations table in crates/rolter-control/src/openapi.rs rather than from a second OpenAPI toolchain. Rejects hand-written SDK surfaces (#421/#422/#851 would each transcribe 240 operations, three times, forever) and utoipa derivation (a proc-macro graph plus a schema describing rolter’s Rust types rather than its wire contract, which diverge wherever a handler flattens or hides a field). Completeness is a merge gate, not a hope: a test reads every route registration under src/ and fails naming any (path, method) missing from the document, with the reverse check for stale rows. Consequence: adding an endpoint costs one row and forgetting it fails the build; body schemas are typed for the stable CRUD surfaces and open objects for the dynamic ones, which the gate makes a visible, enumerable debt rather than a silent gap.

ADR-0031 — One model for turning a subsystem off

Proposed. A survey for #1073 found eight different ways a rolter subsystem can be turned off — a deployment-wide flag, a per-subsystem enabled, an Option left unset, an empty collection, a numeric sentinel, a per-object row, a cargo feature, and absent infrastructure — with no written boundary between them, and ten subsystems (MCP gateway, PII sanitizer, guardrail webhooks, semantic cache, budgets, rate limits, plugins, payload capture, realtime, analytics) having no deployment-wide switch at all. Collapses these to two layers: capability (compile-time or infrastructure, read-only, extending the existing unavailable_flags()) and enablement (one DB-backed boolean per subsystem, hot-reloaded through /internal/snapshot). Defines “off” per class rather than per subsystem — filters and optimisations bypass, gateway surfaces answer 501 naming the flag rather than a 404 indistinguishable from a typo, enforcement bypasses audibly, recorders drop at ingest, UI-only surfaces hide — under one rule: a disabled subsystem never changes an answer the client already had a right to. Per-object enabled rows stay and are explicitly outside the model. Rejects a bare per-subsystem enabled (cannot express capability), a generic HashMap<String, bool> (loses the compile-time flag-name guarantee and the default-on test), and a second SPA bootstrap payload. Consequence: default-on is guaranteed by construction and by a test over FeatureFlagsConfig::default(), so an upgrade cannot change behaviour; the cost is ten subsystems each needing a flag, a bump_config_version() migration, snapshot plumbing and a dashboard switch, filed as sized follow-ups.

ADR-0032 — What 1.0.0 guarantees, surface by surface

Accepted. 1.0.0 is a compatibility claim semver assigns a meaning to whether or not one is written, so #922 writes one per surface. The Rust crates are internal: no crate offers a stable Rust API at 1.0 or during 1.x, said in each published crate’s description, its //! docs and the README, because a shared version number cannot carry the caveat — and both mechanisms that would otherwise enforce the unmade promise are made to agree, with semver_check = false for release-plz (a Rust API diff must never major-bump the product) and the semver-checks job permanently advisory rather than promoted at 1.0 as api-stability.md had planned. /v1/* guarantees the dialect, not a schema: the v1 is OpenAI’s and there is no /v2/ escape hatch, so rolter promises fidelity and tracking — following an upstream dialect change is not a rolter breaking change — while the rolter-owned parts (virtual-key auth, provider-slug/model addressing, the error envelope, /v1/models) are promised like the control API. /api/v1/* is additive within v1, and removing anything published in the OpenAPI document takes two minor releases and 90 days, whichever ends later, announced in the release notes, deprecated: true in the document and served with Deprecation/Sunset headers; both bounds because a release count is worthless at an unfixed cadence and a time period can pass with no release. A 1.0 rolter.toml is read by every 1.x on the same terms; the schema moves forward only, with restore-from-backup as the sole rollback. After 1.0 BREAKING CHANGE: means exactly one thing — a change to a promised surface shipping without a completed deprecation window — never a Rust API change, an upstream dialect change, or a subsystem carrying the experimental marker, which stays the single documented exemption route. Consequence: internal refactors such as #1041/#1042 stay unblocked, the deprecation machinery (Deprecation/Sunset, deprecated: true, a release-notes section) becomes prerequisite work for the first deprecation rather than for 1.0, and removing a stability marker becomes the expensive act.

ADR-0033 — The merge queue is how master stays green

Accepted. ci-ok is evaluated per pull request against whatever base each branch was cut from, and branch protection does not require a branch to be up to date, so two individually-green PRs can combine into a red master — which has now happened twice (#1150’s ProviderConfig field, and the OpenAPI completeness gate landing against two newly-mounted routes in #1310/#1311/#1299). The exposure is structural: this repo deliberately carries several tests that assert a global property of the tree, and every one of them is a guard two independent changes can each satisfy alone and jointly violate. merge-protection.md had set the condition for revisiting at two more incidents; it is met. Enables GitHub’s merge queue on master and keeps strict = false: ci.yml gains a merge_group: trigger so the same ci-ok runs against the synthetic tree the queue will actually produce, batched rather than per PR. Rejects the up-to-date requirement, which is strictly worse here — one gate run per PR instead of one per batch, a hand rebase per author, and last-merge-wins thrash that can be unsatisfiable when master moves faster than a gate run takes. The title-edit fast path is fenced to pull_request so a merge-group tree can never inherit a verdict from a run that never saw it, and both agent-session-url checks follow the queue because a squash merge writes the PR body into history. Consequence: merging becomes asynchronous (gh pr merge enqueues), latency per merge rises by one gate run while throughput does not fall, stacked PRs are unaffected because a child targets an unprotected parent branch, and the master push run stays as the backstop for anything that bypasses the queue.

ADR-0034 — CI is sized to the 20-slot runner pool

Accepted. The organisation’s Free plan gives every workflow run one pool of 20 concurrent jobs, yet one PR push started 36 and an isolated merge to master up to about 98 (about 57 on the measured average), because release-plz re-ran all of quality.yml as verify and then dispatched a 35-job run on the release PR. From 09-26 to 09-28 the 26 master push gates lost a median 21.5 min each to queueing, 17 jobs did under 20 s of work each, the helm job built the maturin wheel by accident, and the Actions cache sat over its 10 GB limit. Collapses the gate into fewer, larger jobs (static checks, rust lint, rust build, nextest, image smoke, one bun job, coverage on PRs only, two codeql legs with an in-job Rust skip on PRs that touch no Rust), moves macOS, compose smoke, msrv and master coverage to a nightly extended.yml, replaces verify with a release-gate bound to the ci.yml push run’s ci-ok job, runs pr-title and the session-url checks inside ci-ok against the live PR body, dispatches release-PR CI only under release:ready, and saves Rust caches from master only: 11 jobs per PR push and 13 per isolated merge (23 while the release PR carries release:ready). ci-ok keeps its name and id, gate-ok keeps its name for the fast path, every former check stays a named step under !cancelled() with a report step that fails on a skipped must-run step, quality.yml stays secret-free, semver-checks stays advisory per ADR-0032, and no blocking check is skipped by diff. Rejects paths: filters (a skipped workflow leaves ci-ok pending) and diff-based skips (a skipped job reads as success, #1562), a check-run name lookup for the release gate, self-hosted runners on a public repository, and ARM or slim runners that draw on the same pool. Consequence: on 09-28 the measured master push gate median was 16.9 min and the simulation puts it at 9.1 after the change, and in the 09-26/27 burst 27.8 against a simulated 11.6; a failing check is a step inside a job, so the report step carries the per-check signal; msrv checks 1.82 for the first time and will go red (#2026); and the merge queue of ADR-0033, still off (#2029), should run with build concurrency 1-2 once enabled.

OpenAI Responses API translation

Status: Development · Date: 13 Jul 2026 · Issues: ROL-252

Context

ROL-252 adds the public POST /v1/responses. The gateway already routes on model, and rolter-proxy already holds an isolated registry of OpenAI Chat Completions and Anthropic Messages translations with incremental SSE handling. Responses carries text, multimodal input, tools, tool results and its own streaming event model. Lifecycle operations use a response identifier without a model, so they cannot safely select a route without storing the tenant/key → upstream response association.

Options considered

Option 1 — Native passthrough only

Forward /v1/responses to every provider without translation.

flowchart LR
    C[Responses client] --> G[rolter gateway]
    G --> P[upstream /v1/responses]

Option 2 — Protocol registry with Responses

Add Responses as a protocol: native OpenAI gets passthrough, while Chat Completions and Anthropic Messages get adapted requests and return Responses objects or SSE events.

flowchart LR
    C[Responses client] --> G[rolter gateway]
    G -->|OpenAI| R["/v1/responses"]
    G -->|Chat-compatible| CC["/v1/chat/completions"]
    G -->|Anthropic| AM["/v1/messages"]
    R --> G
    CC --> G
    AM --> G

Comparison

Decision

Option 2 was chosen. Protocol::OpenAiResponses is added to the rolter-proxy registry. ProviderKind::Openai uses the native /v1/responses; Chat-compatible providers use /v1/chat/completions; Anthropic uses /v1/messages. Common fields are translated, and responses and streaming events are returned in the Responses representation.

Rationale

This preserves the existing architectural boundary: the gateway owns authentication, tenant scope, target selection, tracing, metrics and accounting, while rolter-proxy owns only the wire protocol. Native Responses loses no provider-specific fields, and the existing Chat and Anthropic routes stay reachable for clients on the new OpenAI SDK.

Consequences

Benefits:

• one shared Responses endpoint works across three families of upstream APIs;
• SSE is translated incrementally, without buffering the live stream;
• input_tokens/output_tokens are counted by the existing cost and rate-limit accounting.

Drawbacks and risks:

• background, store, previous_response_id and provider-specific reasoning have no safe Chat/Anthropic equivalent and are not forwarded there;
• lifecycle for native OpenAI is implemented by a separate tenant-scoped registry in ADR-0016; translated Chat/Anthropic resources still return 501 response_lifecycle_unsupported;
• supporting new Responses event types requires extending the converter and its tests.

System impact:

rolter-gateway, rolter-proxy, the OpenAPI spec and the API documentation change; the storage schema does not.

Related records

• ADR-0014 — Extensible API protocol translation
• ROL-252 — OpenAI Responses API passthrough and streaming

Open questions

• Consider a distributed backend registry for multi-replica deployments without sticky routing.
• Extend streaming translation of function-call and reasoning event types as supported upstream contracts are confirmed.

Routing OpenAI Responses resources through a tenant-scoped registry

Status: Development · Date: 13 Jul 2026 · Issues: ROL-252, ROL-264

Context

ROL-252 added OpenAI Responses creation, routed on the model field. ROL-264 adds GET and DELETE /v1/responses/{id}, POST /v1/responses/{id}/cancel and GET /v1/responses/{id}/input_items. These requests carry no model, tenant or provider, and the upstream identifier cannot safely be used to re-balance: the request could go to a different provider, target or provider key and expose another tenant’s resource.

The gateway already authenticates the virtual key, atomically refreshes the routing snapshot and selects a concrete provider credential. A safe lifecycle requires remembering that decision after a successful POST /v1/responses, streaming responses included. Translated Chat Completions and Anthropic Messages do not create a persistent upstream Responses resource.

Options considered

Option 1 — Re-select the route from the response ID

Hash response_id and use the current balancer or iterate over providers.

flowchart LR
    C[Lifecycle client] --> G[rolter gateway]
    G --> B{Current balancer}
    B --> P1[Provider A]
    B --> P2[Provider B]

Option 2 — Bounded process-local tenant registry

After a successful creation, store the composite key virtual-key digest + response_id and pin it to the provider, target, model and provider-credential fingerprint. The entry has a TTL and is removed after a successful DELETE.

sequenceDiagram
    participant C as Client
    participant G as Gateway
    participant R as Tenant registry
    participant P as Native OpenAI provider
    C->>G: POST /v1/responses with model
    G->>P: create via selected target and credential
    P-->>G: response ID or SSE response event
    G->>R: store tenant plus pinned route and capabilities
    C->>G: GET, DELETE, cancel or input_items by ID
    G->>R: lookup by authenticated tenant plus ID
    R-->>G: pinned native route
    G->>P: lifecycle request with same credential

Option 3 — Distributed registry in Redis

Store the same entries in shared Redis with a TTL, so that any gateway replica can serve a lifecycle request.

flowchart LR
    C[Client] --> G1[Gateway replica 1]
    C --> G2[Gateway replica 2]
    G1 --> R[(Redis registry)]
    G2 --> R
    G1 --> P[OpenAI provider]
    G2 --> P

Comparison

Decision

Option 2 was chosen. rolter-gateway keeps a bounded process-local registry. The key is composed of a peppered digest of the virtual key and the public response_id; the plaintext key is not stored. The value records the provider, target, public model, provider-native ID, the fingerprint of the selected provider credential and capability flags.

By default an entry lives for 24 hours and a single process holds at most 100,000 entries. Both are configured through [responses] registry_ttl_secs and registry_max_entries; a zero value disables registration. Expiry and overflow are cleaned up lazily on lookup/insert. A successful DELETE removes the entry immediately.

Rationale

A process-local registry closes the main security invariant without requiring Redis on the inference path. The composite key makes unknown, expired and cross-tenant lookups indistinguishable. Pinning to the provider credential prevents reaching the resource through a different upstream account. The fingerprint makes it possible to find the same credential after the key pool is reordered, without storing the secret a second time.

Translated Chat Completions and Anthropic Messages get an entry with empty lifecycle capabilities. That allows returning a precise 501 response_lifecycle_unsupported to the owner while keeping a uniform 404 response_not_found for any other tenant.

Consequences

Benefits:

• a lifecycle request is never re-balanced and always uses the original provider account;
• unknown, expired, deleted, cross-tenant and post-config-change unreachable resources all return the same 404 with no route metadata;
• the native upstream status and body are forwarded without translation;
• non-streaming JSON and completed SSE are registered from a single completion-observer point.

Drawbacks and risks:

• multi-replica deployments require sticky routing by client or response ID;
• a gateway restart drops the registry earlier than upstream retention;
• the full response is temporarily buffered by the existing accounting stream; the registry reuses that buffer, but the memory footprint is not reduced;
• deleting a provider, changing its kind or rotating the original credential makes the entry unreachable and returns 404.

System impact:

rolter-core (retention configuration), rolter-gateway (registry, lifecycle handlers and response observer), rolter-proxy (model-less forwarding), the OpenAPI spec, the API documentation and the engine smoke suite change. The Postgres/Redis schemas do not.

Related records

• ADR-0005 — Org → Team → Project → Virtual Key tenancy
• ADR-0014 — Extensible API protocol translation
• ADR-0015 — OpenAI Responses API translation
• ROL-252 — OpenAI Responses API passthrough and streaming
• ROL-264 — Responses API lifecycle resources

Open questions

• Add an optional Redis registry backend if lifecycle must work without sticky routing across replicas.
• Extend capability discovery once OpenAI-compatible providers reliably implement the native Responses lifecycle.

Provider/model addressing to disambiguate identical model names

Status: Accepted · Date: 14 Jul 2026 · Issues: ROL-266

Context

When several providers serve the same model name, a client has no first-class way to say which one it wants. This is common with self-hosted OpenAI-compatible backends: two vLLM/SGLang instances at different base URLs both serving qwen3, or an openai-kind provider whose base_url points at a non-OpenAI upstream.

How rolter addresses models today

• A client sends model = a route name (routes.model, unique per project).
• A route fans out to one or more route_targets, each {provider_id, upstream_model?, weight}.
• The proxy rewrites the outgoing model to upstream_model when set (crates/rolter-proxy/src/lib.rs:333, maybe_rewrite_model); target selection and failover happen in crates/rolter-gateway/src/handlers.rs (weighted balancer over entry.route.targets).
• Providers are unique(org_id, name); there is no URL-safe slug.

flowchart LR
    C[client: model = route name] --> R[route lookup]
    R --> B[balancer over targets]
    B --> T1["target A · provider vllm-msk · upstream qwen3"]
    B --> T2["target B · provider vllm-spb · upstream qwen3"]

So disambiguation today means the operator invents distinct route names (qwen3-msk, qwen3-spb). There is no client-facing provider/model addressing and no convention linking a route name to a concrete provider.

Prior art

• LiteLLM — a two-tier scheme: clients send a public model_name alias; each deployment carries litellm_params.model with a provider prefix (openai/gpt-4o) plus its own api_base/credentials. Multiple deployments sharing one model_name are load-balanced. The prefix is a routing hint tied to a base_url, so you can point openai/... at a Qwen endpoint by swapping base_url — the prefix is semantically muddy and is not a stable identity.
• Envoy AI Gateway — extracts the model field from the request body into an x-ai-eg-model header before routing, then applies ordinary header-match rules in an AIGatewayRoute to pick a backend. Selection is out-of-band (header/route config), not encoded in the model string the client sends.

The lesson: keep the address segments stable identities, not free-form hints (avoid LiteLLM’s base_url ambiguity), while still letting a route fan out.

Options considered

Option A — Status quo (distinct route names)

Operators keep inventing unique route names per provider (qwen3-msk).

• Pro: zero code; already works; route name stays the single routing key.
• Con: no convention; clients must know deployment-specific names; poor discovery; the provider is invisible in the address.

Option B — First-class provider-slug/model addressing (coexisting)

Introduce a stable, URL-safe slug on providers. A client may send either a bare route name (unchanged) or provider-slug/model. The gateway resolves provider-slug/model to the concrete (provider, upstream_model) pair, pinning the provider and using model as the upstream model.

flowchart LR
    C["client: model = 'vllm-spb/qwen3'"] --> P{"contains '/' and<br/>left = known provider slug?"}
    P -- no --> RN[resolve as route name today]
    P -- yes --> PR["pin provider vllm-spb"]
    PR --> UM["upstream model = 'qwen3'"]
    UM --> FW[forward to that provider only]

• Pro: unambiguous, self-describing addressing; slug is a stable identity (fixed kind + base_url), avoiding LiteLLM’s base_url muddiness; coexists with named routes, so it is additive and backward-compatible; maps cleanly to “pick a provider from the list, or add one inline, then reference its models” in the UI.
• Con: needs a new slug column (migration + uniqueness/charset rules + CRUD/UI); parsing precedence and slash-collision rules to define; a pinned provider bypasses multi-provider fan-out (see open questions — it can still fan out across that provider’s own key pool / targets).

Option C — Auto-derived provider/model aliases (read-only convenience)

No new column: derive an alias by slugifying the existing provider name and pairing it with each target’s upstream_model, exposed only through /v1/models and accepted on input.

• Pro: no schema change; quick.
• Con: provider name is mutable and only unique(org_id, name) — renames silently break addresses; slugifying a display name yields collisions and unstable ids. Same fragility LiteLLM has. Rejected as the primary path.

Option D — Out-of-band provider selector header

Keep model as today; add an optional x-rolter-provider: <slug> header (Envoy-style) to pin the provider.

• Pro: no change to the model string; no slash-parsing.
• Con: not expressible in stock OpenAI/Anthropic SDK model fields, so clients that can only set model (the common case) cannot use it; discovery is worse. Useful as a complement to B, not a replacement.

Comparison

Recommendation

Adopt Option B — first-class provider-slug/model addressing that coexists with today’s named routes — and optionally add Option D later as a complementary header for clients that need to pin a provider without touching model.

Rationale: B gives unambiguous, self-describing, rename-safe addressing that fits the OpenAI/Anthropic model field clients already use, and matches the desired UX (“pick an existing provider or add one inline, then reference provider-slug/model”). It is purely additive: existing bare-model routes keep working.

Proposed resolution semantics (to confirm)

1. Precedence: try the whole model string as a route name first (preserves any existing route whose name contains /). If unmatched and the string contains /, split on the first /: if the left segment is a known provider slug in the caller’s scope, treat the right segment as the upstream model and pin that provider; otherwise fall through to the normal not-found path.
2. Slug: ^[a-z0-9][a-z0-9-]{0,62}$, unique(org_id, slug), immutable-by-default (renaming display name never changes the slug). Backfill existing providers from a slugified name with numeric de-dup on migration.
3. Balancing: a provider-slug/model request pins the provider and bypasses cross-provider fan-out, but still uses that provider’s key pool, cooldowns, and (if the provider has multiple same-model targets) intra-provider selection. Bare route names keep full multi-provider balancing.
4. /v1/models: list both — existing route ids and provider-slug/model ids (grouped by provider in a follow-up UI), so either address is discoverable.

Slug collision handling

Slugs are unique(org_id, slug), so the DB is the final arbiter — but “just add a unique index” leaves the behaviour around collisions unspecified. This section pins it down for the three moments a collision can happen. Note first that slugs are org-scoped: there is no cross-org collision, and provider-slug/model always resolves within the caller’s org (enforced in the query where org_id = $caller), so the blast radius of any collision is a single org.

1. Migration backfill (deterministic). Shipped in crates/rolter-store/migrations/0014_provider_slug.sql:

• Providers are processed per org in a stable order (id ascending via row_number() over (partition by org_id, base order by id)), so the result is reproducible.
• Slugify name: lowercase, map every run outside [a-z0-9] to a single -, trim leading/trailing -; an empty result falls back to provider.
• The first claimant of a base slug keeps it bare (truncated to 63); each subsequent collision gets a -N suffix from its row number (vllm, vllm-2, vllm-3, …), the base truncated to 58 chars so base-N still fits. The migration never fails on a collision — it always converges.
• Follow-up: emit a migration report (one line per adjusted provider: org_id, provider_id, name, base_slug, final_slug) so operators can reconcile any externally-published address.

2. Runtime creation (validate, never silently mangle). Shipped in crates/rolter-control/src/crud.rs:

• An explicit slug is validated against ^[a-z0-9][a-z0-9-]{0,62}$; an omitted one is derived from name (a name with no ascii alphanumerics requires an explicit slug). A conflict surfaces as the DB unique-violation error — the API never auto-appends a suffix behind the operator’s back, because an address is a contract the human picks explicitly.
• Follow-up: map the unique violation to 409 with the next free suggestion ({"error": "slug taken", "suggestion": "vllm-2"}), and have the UI pre-check availability before submit.
• Slug is immutable by default (it is the stable identity). An in-place change requires the explicit allow_slug_change=true opt-in on update, which warns that the provider-slug/model address changes with it.

3. Deletion & reclaim (explicit, not automatic). A hard-deleted provider frees its slug immediately; the next create may reuse it. This is intentional but load-bearing: reusing openai after deleting the old openai repoints that address to a new upstream. Therefore:

• Prefer soft-delete for providers that ever served traffic, so a freed slug is not silently re-pointed; a reused slug is then an explicit operator action on a tombstoned name.
• Reclaim is a new identity, never a restore — document that provider-slug/model after reclaim may resolve to different weights/base_url than before.

Decision (14 Jul 2026)

• Adopt Option B, coexisting with named routes. provider-slug/model resolves to a pinned (provider, upstream_model); bare route names keep full multi-provider balancing. Named routes are not replaced.
• Balancing when pinned: a provider-slug/model request pins the provider and bypasses cross-provider fan-out, but still fans out within that provider — its key pool, cooldowns, and any same-model targets stay in rotation.
• Precedence & slug rules as proposed above (route-name-first, then first-/ split; slug ^[a-z0-9][a-z0-9-]{0,62}$, unique(org_id, slug), immutable by default).
• Collision policy as in Slug collision handling above: slugs are org-scoped (no cross-org collision); migration backfill is deterministic with -N de-dup (shipped in migration 0014); runtime creation validates and rejects conflicts rather than silently suffixing (409 + suggestion as follow-up); slug changes need the explicit allow_slug_change opt-in; hard-delete frees a slug immediately, but soft-delete is preferred so a reclaimed slug is an explicit operator action, not a silent re-point.
• Option D (header selector): deferred, not part of the initial implementation.

Addendum (20 Jul 2026) — provider groups: unifying slugs under one address

Problem

Per-provider slugs alone push fleet operators the wrong way. Run ten vLLM instances (distinct base_url + credentials each) and you get ten providers — vllm-1 … vllm-10 — where pinning any single one is the opposite of what the operator wants. Named routes can unify them, but a route binds one public model name to targets, so a cluster serving 30 models needs 30 hand-made routes. LiteLLM’s “model group / alias” answers the per-model case; nothing answers “this whole fleet, all of its models, one address”.

Decision

Introduce provider groups — an org-scoped, named set of providers that shares the slug namespace with providers (unique(org_id, slug) enforced across both), so the left segment of X/model resolves through a single unified lookup and is never ambiguous.

• Membership: groups are fully operator-defined — any slug, any member set — and membership is many-to-many: one provider may belong to several groups. Overlapping groups are the intended way to express scopes, e.g. vllm-cluster-msk (five Moscow instances), vllm-cluster-nsk (five Novosibirsk instances), and vllm-cluster-all (all ten) — the client picks the scope by address. No group nesting (groups of groups) initially: flat membership covers the scoping cases, nesting adds cycle-detection and resolution complexity for little gain; deferred.
• Resolution: precedence unchanged — whole string as route name first, then split on first /. The left segment is looked up in the unified slug namespace: a provider slug pins that provider (as decided above); a group slug fans out across the group’s members, balanced with the group’s configured strategy (any rolter_balancer::LoadBalancer), honoring per-member weight, each member’s key pool, and cooldowns.
• Model handling: default is passthrough — the right segment is forwarded unchanged as the upstream model (the homogeneous-cluster case: every member serves the same model set). An optional per-member upstream_model rewrite covers heterogeneous groups, mirroring route_targets.
• Credentials: unchanged. Clients authenticate with a rolter virtual key; member credentials stay per provider. A group gives one address, one client key, N upstreams with N distinct upstream creds — which is the unification the fleet operator actually needs.
• Relation to routes: a group is effectively a wildcard route family (vllm-cluster/*) without creating a route per model. Named routes remain the curated-alias layer for cherry-picked public names; groups cover fleets.
• /v1/models: group addresses are listed as the deduplicated union of member-served models (vllm-cluster/qwen3, vllm-cluster/llama4, …) alongside provider-slug and route ids.
• Collision interplay: groups also soften the collision pressure from the section above — individual instances can carry mundane slugs (vllm-msk-1), while the group owns the meaningful address (vllm-cluster, or even vllm).

Addendum (26 Sep 2026) — precedence within the caller’s tenancy (#1844, #1845)

Resolution runs within the calling key’s tenancy (Snapshot::resolve_for in crates/rolter-gateway/src/state.rs). A named route of another org, or one its admin narrowed to another project, counts as a miss rather than a refusal. It shadows neither the caller’s own slug/model address nor the builtin fake-llm, and the caller gets the same 404 model_not_found as for a model nobody configured. A pinned address of another org’s provider or group answers the same way. Route names may still contain / (Qwen/Qwen2.5-7B), and a route of the caller’s own org still wins over the caller’s own address.

Until per-org namespaces land (#1857), one slug namespace spans the whole deployment: providers and groups of every org plus the bootstrap file’s readonly rows. The control plane refuses a provider or group slug that any of them holds, and a route name that is the slug/… address of another org’s or a readonly provider or group, or the builtin. Keys from a config file carry no org, so for them every row is in scope, and the same holds for anonymous callers when require_auth is off. The refusal runs in one direction only: it keeps a new route off an address that already exists, but a provider or group created later with the slug of another org’s existing slug/… route is accepted. Callers in the new row’s org get their own address, since the other org’s route is out of their tenancy; org-less callers resolve the named route first and reach the other org’s route and its upstream. Refusing that slug instead would let any org reserve slugs by naming routes after them. The gateway indexes provider slugs first-wins in file order, bootstrap rows before database rows, and then group slugs, skipping any a provider already holds. A slug two rows still share from before the guard therefore resolves the same way on every build, and a legacy database provider keeps a readonly group’s slug.

Proposed follow-up implementation issues

1. store: add immutable slug to providers — migration, unique(org_id, slug), deterministic reported backfill with -N de-dup, 409+suggestion on create-conflict, soft-delete-preferred reclaim, CRUD wiring. (relates to ROL-81, see Slug collision handling)
2. proxy/gateway: provider-slug/model parsing + resolution with the precedence rule and provider pinning; interaction with maybe_rewrite_model/upstream_model; tests.
3. gateway: extend /v1/models to surface provider-slug/model ids.
4. ui: model-management surfaces the provider-slug/model address; inline add-provider flow. (relates to ROL-222)
5. (optional) proxy: x-rolter-provider header selector (Option D).
6. store/gateway/ui: provider groups — provider_groups entity (slug in the unified namespace, strategy, members with weights), group-slug/model resolution with passthrough + per-member rewrite, /v1/models union listing, group CRUD/UI. (see Addendum: provider groups)

Sources

• LiteLLM proxy config — model_name / litellm_params / load balancing / wildcard
• Envoy AI Gateway — model-name-based routing via x-ai-eg-model

Per-provider egress proxy pools

Status: Accepted · Date: 18 Jul 2026 · Issues: #305

Context

A provider may need several outbound proxies for regional reachability, IP-based rate limits, or resilience. The existing singular egress_proxy setting cannot spread traffic or recover when its proxy becomes unavailable. Proxy credentials must not enter database rows, config snapshots, API responses, or metrics labels.

Options considered

1. Keep one proxy and rely on an external proxy load balancer.
2. Select a random proxy independently for every attempt.
3. Maintain a per-provider round-robin pool with local failure quarantine.

Decision

Adopt option 3. egress_proxies is the canonical ordered pool, while the legacy egress_proxy remains a backward-compatible one-element pool. Each member owns a cached reqwest client and may use HTTP, HTTPS, SOCKS5, or SOCKS5H. Requests start at the next round-robin member and retry connect or tunnel failures through the remaining eligible members. Three consecutive failures quarantine a member for 30 seconds; a successful request clears its failure streak.

Authenticated URLs are accepted only as whole-value ${ENV_VAR} references. Metrics identify members with a redacted stable label and never expose userinfo.

Consequences

• Providers can spread traffic and survive an individual proxy failure without a separate egress load balancer.
• Existing singular configuration continues to work unchanged.
• Retry is deliberately limited to connection/tunnel failures; retrying arbitrary HTTP responses could duplicate a request whose upstream already processed it.
• Quarantine state and round-robin position are process-local, so gateway replicas may temporarily make different choices.
• Operators must inject authenticated proxy URLs into every gateway replica.

Bounded semantic response caching in Redis

Status: Accepted · Date: 18 Jul 2026 · Issues: #261

Context

The exact response cache only reuses byte-equivalent normalized requests. Similar prompts still reach an upstream model even when an existing response would be acceptable. Semantic reuse adds an embedding request and a nearest-neighbour search, but it must not make gateway availability depend on the embedding provider or Redis and must keep request-path work bounded.

Options considered

1. Keep exact caching only.
2. Introduce a dedicated vector database and approximate-nearest-neighbour index.
3. Store embeddings alongside Redis cache entries and scan a bounded recent window.

Decision

Adopt option 3 as an opt-in route capability. Exact lookup always runs first. On an exact miss, rolter embeds the normalized prompt using an explicitly configured provider/model and compares cosine similarity against at most max_candidates recent entries in the route/key-isolated Redis index. A response is reused only when its score reaches threshold. Semantic metadata and the exact response share the route cache TTL.

Embedding, Redis, decoding, dimension, missing-entry, and stale-entry failures all fail open to normal upstream routing. Streaming responses enter both indexes only after the stream completes successfully.

Consequences

• Similar prompts can avoid model cost and latency without a new datastore.
• Exact-cache behavior and its faster lookup remain unchanged.
• Search cost is predictable but linear in the configured candidate bound; this is not intended as a large corpus vector-search engine.
• The embedding provider adds cost and latency on semantic misses and becomes part of cache-result quality, though never gateway availability.
• Operators own the similarity threshold and must account for unsafe reuse in domains where small wording changes alter the correct answer.

External cache telemetry for routing

Status: Accepted · Date: 18 Jul 2026 · Issues: #258, #259 Supersedes: ADR-0007 for exact vLLM and LMCache-aware modes

Context

ADR-0007 deliberately introduced approximate prompt-prefix affinity without coupling the gateway to an inference engine. vLLM can now publish exact KV block residency events, while LMCache deployments can expose availability and occupancy. Using those signals can improve reuse, but blocking network calls or trusting stale telemetry on the request path would weaken gateway latency and availability.

Options considered

1. Keep only the engine-independent approximate prefix trie.
2. Query cache state synchronously during every routing decision.
3. Consume telemetry in background tasks and expose bounded in-memory scorers.

Decision

Adopt option 3 as two opt-in strategies behind the existing balancer interface:

• precise_cache_aware subscribes to the supported vLLM V1 three-frame ZMQ msgpack KV-event protocol, derives stable token-block identities, and scores the resident leading fraction of tokenizer-aligned request blocks.
• lmcache_aware polls a rolter-defined HTTP JSON signal containing occupancy and cache_available, then scores available targets as 1 - occupancy.

Both sources update bounded process-local state outside the request path. Missing request token IDs, malformed input, disconnects, stale data, and vLLM sequence gaps produce a neutral score and preserve least-load fallback. A sequence gap clears the precise index until an explicit all-blocks-cleared boundary re-establishes trust.

Consequences

• Exact vLLM residency and LMCache capacity can influence routing without request- path network I/O.
• Approximate cache_aware remains available and engine-independent.
• vLLM exact scoring requires token IDs from the same tokenizer and a compatible event protocol; version assumptions are part of the operator documentation.
• LMCache integration depends only on the small documented controller contract, not an unstable internal LMCache API.
• Telemetry state is local to each gateway replica and intentionally disposable; restart, staleness, or desynchronization temporarily reduces routing quality but does not prevent requests.

Uniform config-vs-DB tiering for models, providers, and provider groups

Status: Accepted · Date: 20 Jul 2026 · Issues: #306 Relates: ADR-0017 (provider/model addressing), ADR-0018 (granular CRUD)

Context

rolter bootstraps from a TOML config and (optionally) a Postgres store, merged at startup via MergedConfigStore. Today the three routable entity types treat the config/DB boundary inconsistently:

Three problems:

1. Providers cannot be seeded-then-edited. A [[providers]] entry is permanently config-owned and rejected by CRUD (require_not_config_owned), so an operator who wants to bootstrap a provider from config but later edit its credentials/base_url through the UI has no path.
2. Provider groups have no DB tier at all — they exist only as read-only config (ADR-0017 addendum / #571), so they cannot be created via the API or UI.
3. The model shape is idiosyncratic — top-level [[routes]] and a parallel [models.readonly] both mean “immutable config route”, which is redundant and does not generalize to providers/groups.

The desired behaviour is uniform across all three: an operator can declare an entry as readonly (immutable, served straight from config), as a default (seeded into the DB once at startup, then owned and editable via API/UI), or create it purely through the API/UI (a DB row with no config presence).

Decision

Adopt one uniform two-tier config wrapper for every routable entity type — models, providers, and provider groups — layered over the existing DB (CRUD) tier:

[models]
readonly = [ /* immutable, config-owned, rejected by CRUD */ ]
default  = [ /* seeded into the default project once, then DB-owned */ ]

[providers]
readonly = [ ... ]
default  = [ ... ]

[provider_groups]
readonly = [ ... ]
default  = [ ... ]

Semantics, identical for each entity type:

• readonly — merged into the effective running config and into the gateway snapshot directly; tracked in ConfigOwned; CRUD create/update/delete against a name/slug in this set is rejected. Immutable for the process lifetime.
• default — seeded into the bootstrap default/default/default tenancy exactly once (idempotent: an existing row by natural key is never overwritten on restart), then it is an ordinary DB row — editable and deletable via API/UI, and not config-owned. Mirrors today’s seed_default_models.
• DB (API/UI) — pure store rows with no config presence, already the CRUD path.

Resolution precedence when the same natural key (routes.model, providers.slug, group slug) appears in more than one tier: readonly wins (it is immutable and must never be shadowed by a DB row), then the DB row, then a default that has not yet been claimed as a DB row. Seeding a default whose key already exists as readonly is a config error, surfaced at load time (as models.default already does against readonly routes).

Migration & back-compat

• The current top-level [[routes]] and [[providers]] arrays are kept as deprecated aliases for their readonly tier, so existing configs and the bootstrap rolter.example.toml keep working unchanged. New docs steer operators to [models.readonly] / [providers.readonly].
• [[provider_groups]] (introduced in the ADR-0017 addendum / #572, before this ADR) becomes the provider_groups.readonly alias; the config-only groups slice is forward-compatible with this decision.
• Slugs remain the stable identity across tiers (ADR-0017): a default provider or group keeps its slug when it becomes a DB row, so its provider-slug/model / group-slug/model address is stable across the config→DB transition.

Consequences

• One mental model and one code path shape (readonly merge + default seed + ConfigOwned guard) for all three entity types; the provider-groups and provider work reuses the model tiering rather than inventing per-entity rules.
• Providers gain a seed-then-edit story; provider groups gain a full DB lifecycle.
• New store tables/repos are required for provider groups (provider_groups + membership), plus seed functions for providers.default and provider_groups.default, plus group CRUD — tracked as separate PRs (see below).
• The deprecated top-level arrays add a small amount of parse-time aliasing to maintain until a future breaking release removes them.

Follow-up implementation issues

1. core: uniform tier wrapper for providers + provider groups (readonly/default), with top-level [[routes]]/[[providers]]/[[provider_groups]] kept as deprecated readonly aliases; load-time validation that a default never collides with a readonly key.
2. store/control: providers.default seed tier — seed function mirroring seed_default_models; providers seeded this way are DB-owned, not config-owned.
3. store: provider_groups + membership tables, repo, and MergedConfigStore wiring so DB-defined groups reach the gateway snapshot.
4. control: provider-group CRUD (create/update/delete, slug validation shared with providers, 409+suggestion on slug conflict) + provider_groups.default seed.
5. ui: provider-group management screens (create/edit/membership/strategy), surfaced alongside providers and models. (separate issue — not in the initial backend PRs)

Propagating access-profile model policy to the data plane

Status: Accepted · Date: 4 Aug 2026 · Issues: #791, #534 Relates: ADR-0022 (config-vs-DB tiering)

Context

#534 added org-scoped access profiles that can carry a model and route allow/deny policy. The control plane resolves the profiles a caller holds, merges their policies, and reports the result on GET /api/v1/rbac/effective — but nothing enforced it on the gateway, so the policy described access it did not actually control.

Enforcement was deferred because the obvious mechanism does not work. The RBAC tables deliberately carry no bump_config_version() trigger: control-plane authorization is evaluated per request against the live database, so there is nothing to propagate, and bumping the version on every role edit would wake the whole gateway fleet for a change it could not observe. Migration 0058 records that reasoning and states the tables must never grow such a trigger.

The gateway, meanwhile, never sees a user. It authenticates a virtual key and knows only what the snapshot told it about that key. A policy is a property of a person; a request carries a credential. Something has to bridge the two.

Decision

Resolve the merged policy per key owner when the snapshot is built, carry it on the virtual-key record, and enforce it on the gateway’s model and route selection.

Concretely:

• rolter_core::ModelPolicy becomes the single definition of the policy shape, its merge rule, and its allow/deny matching. The control plane’s MergedPolicy is now an alias for it.
• load_virtual_keys resolves each key’s created_by user to a merged policy in one batched query for the whole key set, and publishes it on VirtualKeyRecord::access_policy. None means unrestricted.
• KeyMeta::model_permitted and KeyMeta::route_permitted gate the request path. The key’s own model allow-list and the owner’s policy must both permit a model: they are separate grants and neither may widen the other.
• Migration 0060 adds bump_config_version() triggers to exactly the four tables that feed this resolution: access_profile_policies, access_profile_assignments, access_profiles, and memberships.

Why not resolve at key-mint time

#791 floated stamping the policy onto the key when it is minted. That is unsound: a profile edited after the mint would never reach keys already issued, so revoking a model would not revoke it. A policy change that fails to restrict is a security bug, and the failure is silent — the control plane would report the new policy on /rbac/effective while the gateway kept honouring the old one. Snapshot resolution has the opposite failure mode: the worst case is a bounded staleness window that the existing config-version machinery already closes.

Why this does not contradict migration 0058

0058 says these tables must never bump the config version because the data plane does not consume them. This ADR changes that premise rather than overruling the rule: the gateway now consumes them, and the repository’s standing convention is that any table the data plane reads must bump the version inside the write transaction. 0060 therefore applies the rule to the tables whose premise changed, and only those.

custom_roles and custom_role_grants stay untriggered. They decide control-plane authorization, which is still evaluated live per request, so 0058’s reasoning holds for them unchanged.

memberships is included because it is the second path by which a profile reaches a user: a profile assigned to a team applies to everyone in it, so adding a member changes that user’s effective policy without any row in the profile tables changing. Membership is not a hot table, the trigger is statement-level, and the gateway coalesces on the version number — so even a bulk SCIM sync costs one snapshot poll rather than one per member.

Consequences

• The policy /rbac/effective reports is now the policy that is enforced. One implementation of “deny beats allow” serves the control plane, the store and the gateway, so the reported and enforced answers cannot drift.
• A deployment with no access profiles is bit-for-bit unaffected: every key carries None, and model_permitted reduces to the pre-existing model_allowed check.
• Editing a profile or a team membership now bumps the config version, so those edits wake the gateway fleet. This is the intended cost of making the policy observable; it is bounded to four tables, none of them on a request path.
• Enforcement is keyed on created_by. A key with no owner — admin-created and config-defined keys — carries no policy, because there is no person whose profiles could apply. Restricting those remains the key’s own model list.
• The staleness window is the snapshot poll interval, as it is for every other DB-backed config. A revocation is not instantaneous; it is as fast as a provider or route change.

Dashboard UX telemetry as a structural-only event stream

Status: Accepted · Date: 6 Aug 2026 · Issues: #805 Relates: ADR-0023 (access-policy propagation)

Context

Browser tracing (#807) tells us how long a dashboard screen took and whether it threw. It does not tell us whether the screen was usable: which screens are slow to become interactive, where people back out, which forms get abandoned, which error and empty states are actually reached. Those are the questions that decide what to build next, and spans cannot answer them.

Collecting that is ordinarily where a product-analytics vendor is added. For a self-hosted gateway that sits in the request path of its operators’ LLM traffic, a second data processor is a hard sell: a different retention policy, a different redaction story, a different jurisdiction, and an outbound connection operators did not ask for. Several rolter deployments are air-gapped, where it simply does not work at all.

There is also a real hazard specific to this kind of telemetry. UX event streams are where free text leaks. An “abandoned form” event is one careless field away from carrying what was typed into the form, and the dashboard’s forms hold provider API keys, virtual keys and prompts.

Decision

Collect UX events in-house, into a ui_events ClickHouse table beside request_logs, and make the schema the privacy guarantee.

Structural only. Every column is a key, an enum, a duration or an id. There is deliberately no column a form value, prompt or free-text body could be written into. target names the form, control or validation rule; it never carries what failed the rule. A future column that could hold free text is a change to this ADR, not a routine migration.

This is chosen over the usual alternative — accept arbitrary payloads and redact on write — because redaction is a policy that has to keep being correct as the dashboard grows, while a schema with nowhere to put a value is correct by construction. logging.payload_capture already exists for the case where someone genuinely wants raw bodies, and it is off by default; this stream is deliberately not that.

Client-stamped time, server-bounded. The interaction’s instant can only come from the browser — the batch that carries it may be seconds or hours younger — so each event is stamped as it is queued and the endpoint accepts that ts. Trusting it outright would let one laptop with a wrong clock sit at the top of every recent-events query forever, so a supplied instant is used only within a window around ingest time and otherwise replaced by it (#1224). The two instants are therefore not interchangeable, and a row is only ever as accurate as the clock that produced it.

Server-side attribution. user_id is taken from the authenticated session and a client-supplied one is ignored, so a caller cannot file events against another user.

Authentication, not authorization. The ingest endpoint is guarded by CurrentUser alone, following me.rs, and has no row in the capability table. The RBAC model holds that a viewer writes nothing — rbac_matrix.rs asserts exactly that — and a ui_event:create capability granted to any authenticated caller breaks the invariant. A viewer who could not file their own screen views would be absent from every funnel, which makes the data quietly wrong rather than absent. Keeping this off the capability table states plainly that it is not an operator surface.

On by default, with two off switches. Absent clickhouse_url the stream is inert. logging.ui_events = false is a deployment-level opt-out. It defaults on — unlike payload_capture — precisely because the schema cannot carry sensitive content, so the usual reason to make telemetry opt-in does not apply.

Consequences

Operators get usability data with no new vendor, no new egress and no new retention policy: same deployment, same TTL, same 90-day partitioning as request logs. Because events carry trace_id, a slow screen and the gateway request behind it are one join rather than two systems.

The cost is expressiveness. Any question that needs a value rather than a key is unanswerable by design, and some genuinely useful analyses are foreclosed. That is the intended trade: the class of incident this prevents — dashboard secrets in an analytics table — is worse than the analyses it forbids.

Adding a screen means adding a stable screen key rather than letting a URL through. screen, target and from_screen are LowCardinality columns capped at 96 characters, which a real URL does not fit; sending one would degrade the table, so the bound fails the write instead.

Per-tenant routing of this stream is out of scope and tracked separately (#812).

Events as logs, not span events

Status: Accepted · Date: 6 Aug 2026 · Issues: #814, #808, #809, #805 Relates: ADR-0024 (dashboard UX telemetry)

Context

OpenTelemetry is deprecating the Span Events API — Span.AddEvent, Span.RecordException and friends. Maintaining two overlapping ways to emit a correlated event (span events via the Tracing API, log events via the Logs API) produced split guidance for instrumentation authors and a slower-evolving event model. The replacement is events as logs: emitted through the Logs API, named, and correlated to the active span through context rather than attached to it. Semantic-convention authors are now explicitly told to document events as log-based events.

Nothing is being removed soon. The deprecation is phased, existing data stays valid in the OTLP trace model, and compatibility layers are a stated priority. This is direction-setting, not an outage.

What rolter emits today

A grep for the deprecated API finds nothing:

$ grep -rn "add_event\|record_exception" --include='*.rs' crates/
(no matches)

That result is misleading, and reading the bridge is what the audit actually required. rolter instruments through tracing + tracing-opentelemetry, and that layer’s on_event turns every tracing event fired inside an active span into an OTel span event (tracing-opentelemetry-0.33.0/src/layer.rs:1468, span.add_event(...); the pre-export path pushes onto builder.events identically). An ERROR-level event additionally sets the span status to error.

So rolter has zero explicit uses of the deprecated API and is, in practice, an exclusive user of it: the ~57 tracing event macros across rolter-gateway, rolter-proxy and rolter-core become span events whenever they fire inside one of the pipeline spans. The migration surface is the bridge, not our call sites.

Why the ordering between #808 and #809 changes

The GenAI semantic conventions (#808) are precisely the area moving to log-based events. Implementing them against span events would be building onto a deprecated API on the day it is written.

Emitting log-based events requires exporting logs at all, and rolter does not: there is no OTLP logs exporter in the workspace and signoz_logs is empty by construction. #809 therefore stops being an independent nice-to-have and becomes a prerequisite.

Decision

Log-based events are rolter’s event model. New instrumentation targets the Logs API; span events are legacy, produced only incidentally by the tracing bridge.

Concretely:

1. No new explicit span events. add_event and record_exception are not to be called directly. Nothing calls them today and nothing should start.
2. #809 (OTLP log export) sequences before #808 (GenAI conventions). The conventions #808 implements are specified as log-based events, so the log pipeline has to exist before they can be emitted correctly. #808’s acceptance criteria adopt log-based events as the target.
3. tracing stays. The bridge is idiomatic Rust and the ecosystem’s own integration, not a bespoke house wrapper — the same distinction drawn in the #815 wrapping audit. It is where the span-event mapping lives, so it is also the single place a future migration happens: when tracing-opentelemetry routes events through the Logs API, rolter follows by upgrading, without touching 57 call sites.
4. Correlation is the acceptance bar. A log-based event is only a replacement for a span event if it carries trace_id and span_id, which is already called out as most of the value in #809. An exported log that cannot be joined back to its span is a regression against what the bridge does today.
5. Content stays off. The GenAI opt-in content attributes (gen_ai.input.messages / gen_ai.output.messages) remain disabled whether they are carried as span events or as log records. Moving event model does not move the privacy line.

Consequences

• The migration is a dependency upgrade rather than a code sweep, because rolter never adopted the deprecated API directly. This is the payoff of having gone through tracing instead of calling OTel by hand.
• #809 gains a second justification beyond “logs are missing”: it unblocks #808. Its trace_id/span_id correlation requirement is now load-bearing rather than a nicety, and should be treated as a blocking acceptance criterion.
• Until #809 lands, error detail continues to reach backends as span events via the bridge. That is fine — the deprecation is phased and the data stays valid — but it means the GenAI work should not start early and improvise its own event channel.
• Anything that wants a new event type before #809 has no correct home. The answer is to wait rather than to add a span event that will have to be removed.

Alternatives considered

Implement #808 now against span events, migrate later. Rejected. The conventions are specified as log-based events, so this builds a second migration into work that has not shipped yet, on the exact attributes most likely to be re-specified.

Emit both span events and log events during a transition. Rejected. Duplicate correlated events on both signals is the split-guidance problem OpenTelemetry is deprecating span events to escape, and it doubles egress for data that already joins on trace_id.

Drop tracing and call the OTel Logs API directly. Rejected. It is a large rewrite of every call site to avoid a bridge that will itself be updated upstream, and it contradicts the #815 finding that the ecosystem bridge is not the wrapping the “don’t wrap” guidance targets.

Client control over telemetry: a kill switch now, collector-routed tenant destinations later

Status: Accepted · Date: 6 Aug 2026 · Issues: #812, #805, #809 Relates: ADR-0019 (egress proxy pools), ADR-0025 (events as logs)

Context

Telemetry is currently an operator decision baked into the deployment: OTEL_EXPORTER_OTLP_ENDPOINT for the backend exporter and ROLTER_UI_OTEL_ENDPOINT for the dashboard. A client of the gateway has no say in either — they cannot turn tracing off for their own traffic, and they cannot have their spans delivered to a backend they own.

#812 raises two things of very different size, and filed them together so the relationship is on the record. This ADR splits them.

The switch. “Off” is implicit today: you achieve it by leaving an endpoint unset. That works but is not discoverable, does not survive somebody setting the endpoint for one signal, and gives an operator nothing to point at in a security review.

Per-tenant destinations. rolter is multi-tenant (org / team / project, virtual keys, RBAC) while telemetry is single-destination, which does not match. A team running rolter as shared infrastructure wants its spans in its backend; some tenants want no export at all for data-residency reasons; a customer debugging their own integration should not have to ask the operator to fetch traces. This half raises questions — scope, fan-out, egress safety, cardinality — that are design work rather than implementation.

Decision

Ship the kill switch now as an environment variable. Do not build per-tenant fan-out in rolter; route it in the collector when it is built.

1. ROLTER_TELEMETRY_ENABLED

One switch covering traces, metrics, the dashboard’s browser tracing, and logs once #809 lands.

• It can only subtract. When false, no exporter is built regardless of which OTEL_* endpoints are set. It is never a second way to turn export on.
• Unset means enabled, so no existing deployment changes behaviour. That is not a weaker default than it sounds: with no endpoint configured nothing is exported anyway, so the effective default remains “exports nothing”.
• An unrecognized value leaves export on. A typo must not silently blind a deployment; only the explicit falsy spellings (0, false, no, off) disable it.
• Environment-only, with no config-file key, despite #812 asking for “the matching config key”. telemetry::init installs the subscriber in main before any config file is read — the gateway has to be able to log a config parse failure — so a config-file switch could not gate trace export at all. Honouring one only for the signals initialized later would mean the same key meant different things depending on which signal you asked about, which is worse than not having it. The OTEL_* contract this composes with is environment-based for the same reason.

2. Per-tenant destinations belong in the collector

When the second half is built, it is configuration of an OpenTelemetry Collector that sits in front of the tenants’ backends, not a fan-out inside rolter.

• Scope: org. It is the easiest to administer and the coarsest unit anyone actually asks about (“my team’s spans in my team’s Honeycomb”). Virtual key is finer and matches how traffic is attributed, but a destination per key is a cardinality problem with no matching demand. rolter’s job is to stamp the attribute; the routing key can be refined later without changing where fan-out happens.
• Fan-out: the collector’s routing processor. Multiple SDK exporters in-process does not scale with tenant count — each is a connection, a queue and a retry buffer, all inside the request-serving process. rolter emits one OTLP stream to one collector with a resource attribute naming the tenant, and the collector’s routing processor sends each tenant’s data onward. This keeps exactly one egress path in the gateway and makes per-tenant destinations an operator’s configuration change rather than a rolter deploy.
• Egress safety follows from that. A tenant-supplied endpoint is an arbitrary URL, and the deciding argument against in-process fan-out is that it would make the gateway POST to one. Terminating tenant endpoints in the collector keeps that SSRF surface out of the data plane entirely. If a tenant-supplied endpoint is ever accepted through the API, it must go through the same EgressPolicy that provider api_base values do (ADR-0019), and its credentials through the encrypted-secret path — never plaintext config.
• Opt-out is the switch, one level down. A tenant wanting no export at all is a routing rule that drops, not a second mechanism.

Consequences

• An operator gets a single, documentable, reviewable “no telemetry leaves this deployment” today, without waiting on the design work.
• The gateway keeps exactly one exporter and one egress path no matter how many tenants exist. Cardinality and buffer growth become the collector’s problem, where they are a sizing question rather than a data-plane risk.
• The cost is that per-tenant destinations require running a collector. That is already the recommended topology (docs/dev-docs/architecture/observability.md), so it is not a new dependency for anyone following it — but a deployment exporting straight to a vendor endpoint would have to add one.
• Nothing here is blocked on #809, but when OTLP log export lands it inherits both decisions automatically: the switch covers it, and its records carry the same resource attributes the routing processor keys on. That is why the switch is worded per-deployment rather than per-signal.
• ROLTER_TELEMETRY_ENABLED is now part of the public configuration surface and cannot quietly change meaning. In particular it must stay “can only subtract” — making it required-true would break every existing deployment on upgrade.

Alternatives considered

A config-file [telemetry] enabled key. Rejected on initialization order, as above. Worth restating because it is the obvious request and the reason it does not work is not obvious.

Per-signal switches (ROLTER_TRACES_ENABLED, …). Rejected. The problem #812 names is that “off” is currently spread across several variables; adding more variables reproduces it. Per-signal control already exists through the signal-specific OTEL_*_ENDPOINT variables.

In-process per-tenant exporters. Rejected on all three of scaling, egress safety and cardinality, as above.

Reflecting a tenant-supplied endpoint straight from a virtual key. Rejected outright: it turns every key holder into someone who can aim the gateway’s egress, which is the SSRF primitive ADR-0019 exists to prevent.

End-to-end test harness: Python/uv project driving a black-box stack

Status: Accepted · Date: 21 Jul 2026 · Issues: #613, #614

Context

Per-crate unit tests and the two narrow suites we already have (#414 in-process integration, #449 compose smoke) never exercise rolter as a whole: a freshly booted stack, multiple tenants, live RBAC changes, a fleet of upstream engines, and balancing across clusters. #613 needs a broad governance/routing E2E suite, and #614 is the harness every other scenario (RBAC matrix, balancing, key lifecycle, security) builds on.

Two decisions had to be made: what drives the stack, and how the driver code is packaged.

Options considered

1. Rust in-process integration test — spin control+gateway in-process against testcontainers. Fast and CI-native, but not a real deployment: it bypasses the HTTP edge, the compose network, and the container boundaries where isolation and auth actually matter.
2. Bash script driving docker-compose — closest to “run fresh rolter”, but bash grows unmaintainable for a table-driven RBAC matrix with JSON assertions.
3. Python driver over docker-compose — real HTTP against a real composed stack, with a proper test framework for the matrix.

For packaging the Python driver:

• PEP 723 inline script metadata (uv add --script) — deps live in a single file header, uv run file.py auto-installs. Ideal for one-file scripts.
• A uv project (pyproject.toml + uv.lock) — deps and a lockfile for a multi-file package.

Decision

Adopt option 3: a Python driver, latest CPython managed by uv, hitting the real HTTP APIs of a stack brought up by docker-compose (postgres, redis, clickhouse, control, gateway, and N llm-d-inference-sim fake-vLLM engines). No in-process shortcuts — the tests see exactly what an operator/tenant sees.

Package it as a uv project under integration/e2e/, not inline PEP 723 script metadata. The harness is a pytest package: a shared helper/client library plus multiple scenario modules (test_rbac.py, test_balancing.py, test_keys.py, test_security.py) sharing fixtures. Inline metadata is per-single-file and cannot express cross-module fixtures or a lockfile, so it is the wrong grain here. Inline PEP 723 remains fine for any standalone one-off helper run by hand.

The harness project is kept separate from the root maturin pyproject.toml: that one packages the shipping wheel at requires-python >=3.9; the test harness has no such floor and pins the latest stable CPython via uv python pin.

RBAC only enforces when ROLTER_ADMIN_TOKEN is set (otherwise the control plane runs in open/superadmin mode, per #454/ROL-250), so the harness compose sets the admin token and shares ROLTER_KEY_PEPPER/ROLTER_SESSION_PEPPER between control and gateway. Bootstrap uses the admin token as superadmin; per-role scenarios create local accounts and drive them via session tokens.

The suite is heavy, so it is gated (manual dispatch + nightly), never on the default per-PR gate.

Consequences

• One harness backs every #613 sub-issue; scenarios are Python modules, not new bespoke rigs.
• Tests run against the real container edge, so cross-tenant isolation, auth, and balancing are exercised where they actually live.
• Fake engines only (llm-d-inference-sim) — offline/air-gapped safe, no secrets, no model downloads, deterministic.
• A lockfile makes the harness reproducible; the cost is a second Python project in the tree, deliberately isolated from the wheel’s packaging metadata.
• Full stack-boot verification requires Docker and building the rolter images, so it lives in CI (dispatch/nightly), not the fast PR path.

Disaggregated prefill/decode routing belongs to the engine, not the gateway

Status: Accepted · Date: 11 Aug 2026 · Issues: #853 Relates: ADR-0014 (protocol translation boundary), ADR-0007 (cache-aware balancing)

Context

#853 carried three unclaimed ideas from llm-d. The second — routing by inference phase to specialised pools — was flagged as “the largest of the three and the one most coupled to how the fleet is deployed”, with the note that it may not fit rolter’s position in the stack and that the useful output in that case is a written “no, because”. This is that note.

Disaggregated serving splits inference in two. Prefill processes the prompt, is compute-bound, and produces a KV cache. Decode generates tokens, is memory-bandwidth-bound, and consumes that KV cache. Running them on separately sized pools improves utilisation, because the two phases stress different resources and otherwise contend on the same accelerator.

Making that work requires, per request: selecting a prefill worker, running prefill there, transferring the resulting KV cache to a decode worker, and then streaming generated tokens from the decode worker.

What rolter is

ADR-0014 fixed the boundary: rolter-gateway owns authentication, tenancy, target selection, tracing, metrics and accounting; rolter-proxy owns the wire protocol. rolter forwards one client HTTP request to one upstream and streams the response back. Its routing inputs come from gateway-owned configuration, the client request, and bounded telemetry adapters; it does not participate in an engine’s multi-stage request lifecycle.

Options considered

Option 1 — Implement P/D disaggregation in rolter

The gateway selects a prefill target and a decode target, drives both, and coordinates the KV handoff between them.

Option 2 — Treat a disaggregated fleet as one upstream

The engine-side coordinator owns phase routing and the KV transfer. rolter balances across the fleet’s entry points exactly as it does for any other pool, and does not know a request was split.

Option 3 — Hybrid: rolter routes phases, engine transfers KV

rolter picks both workers and issues two calls, but the KV cache moves over the engine’s own sidechannel rather than through rolter.

Comparison

Decision

Option 2. rolter will not implement disaggregated prefill/decode routing. A P/D-disaggregated deployment is a single upstream fleet from rolter’s point of view, and its own coordinator owns phase selection and the KV handoff.

Rationale

The KV handoff is the central mechanism. The tensors do not travel in an OpenAI or Anthropic response; engines move them through connector-selected transports such as NIXL/UCX. A gateway that carried the tensors itself would therefore have to join that engine-specific data path.

Coordination can still be initiated over HTTP. For example, vLLM’s experimental disaggregated-prefill API accepts kv_transfer_params on separate prefill and decode requests while its connector moves the KV data out of band. That makes option 3 technically possible, but does not create a stable, engine-independent contract: rolter would still own engine-specific fields, compatible-worker topology, two-call orchestration and state across the handoff. That is precisely the version coupling and lifecycle ownership ADR-0014 was written to prevent.

It would also be redundant. vLLM ships disaggregated serving with a connector API, and llm-d’s router is co-designed with its engine and scheduler — it can assume things about worker topology and KV placement that a general-purpose gateway in front of heterogeneous providers cannot. rolter’s value is that it sits in front of many engines and hosted APIs; a feature that only works for one engine, at one version, with one transport configured, is not a good trade against that.

The distinguishing question is: can rolter consume the input through a stable, engine-independent contract without joining the engine’s request lifecycle? The answer is yes for prefix-cache affinity, queue depth, KV-event residency, LMCache occupancy and LoRA-adapter affinity, which fit the bounded scorer and telemetry interfaces behind ADR-0007 and ADR-0021. P/D placement does not: the placement decision and transfer metadata are part of one engine-specific, multi-call lifecycle.

Consequences

Benefits:

• the ADR-0014 boundary is preserved: no engine-specific KV connector or phase-control extension in the data plane, and no per-request state held across an upstream handoff;
• disaggregated fleets are usable with rolter today, with no code, for any engine that implements disaggregation — including engines that do not exist yet;
• rolter does not inherit the failure modes of a mid-request handoff, which has no clean recovery once prefill has completed and decode is unreachable.

Drawbacks and risks:

• rolter cannot improve on the engine’s phase placement, and cannot report on it beyond whatever the fleet exposes over its metrics endpoint;
• if a stable, engine-independent phase-placement contract emerges on the OpenAI or Anthropic surface, this decision should be revisited, because the portability and lifecycle objections would no longer hold.

System impact:

None. This is a decision not to build something; no code changes.

Related records

• ADR-0007 — Approximate cache-aware balancing behind a pluggable trait
• ADR-0014 — Extensible API protocol translation
• ADR-0021 — External cache telemetry for routing

Open questions

• Should rolter document a recommended topology for putting a disaggregated fleet behind it (which entry point to target, and how to configure health checks against a coordinator rather than a worker)? That is documentation rather than a routing feature, and would be useful independently of this decision.

An explicit per-provider opt-out from a hosted kind’s host pin

Status: Accepted · Date: 2 Sep 2026 · Issues: #925, #924 Relates: ADR-0022 (config-vs-DB entity tiering)

Context

ProviderKind::Openrouter pinned api_base to one literal string:

#![allow(unused)]
fn main() {
if provider.api_base.trim_end_matches('/') != "https://openrouter.ai/api/v1" {
}

Paired with the rule that an openrouter provider must source its key from api_key_env, the pin closes a real exfiltration path. A hosted kind names one operator, and the credential belongs to that operator. Without the pin, a single edited line in a config file — or a row in the providers table — would send a live OpenRouter key to whatever host the edit named, over TLS, with no error and no signal that anything changed.

The pin also rules out three things operators legitimately do:

• put a corporate egress proxy or internal gateway in front of the vendor
• point at an endpoint that re-implements the vendor’s surface
• test the dialect at all — which is how this surfaced. The #924 dogfooding fleet could not declare kind = "openrouter" against a local fake, so it declared the endpoint openai_compatible instead and the OpenRouter adapter went unexercised end to end.

So this is a decision about how an operator opts out, not a bug fix. The question the options have to answer is what stops the opt-out from quietly becoming the shape everyone copies off a blog post.

Options considered

1. Allow loopback and private-range hosts unconditionally. Enough to test with, no help for a corporate proxy, and it makes the rule depend on where rolter happens to run rather than on what the operator intended.
2. An explicit allow_custom_api_base = true on the provider. The safe default is unchanged; the unsafe case is one reviewable line in the configuration, in the same diff as the base URL it enables.
3. Drop the pin and rely on the api_key_env rule alone. That rule keeps the secret out of config files and database rows, but it does nothing about where the secret is sent — which is the exfiltration path the pin closes.

Decision

Option 2. ProviderConfig gains allow_custom_api_base: bool, defaulting to false:

• Unset, every hosted kind keeps the exact validation it has today.
• Set, the host pin is skipped for that provider and nothing else is relaxed. In particular the api_key_env requirement still applies, so the credential cannot be inlined next to the redirected base URL.
• The rejection message names the opt-out, so the operator who hits the pin learns the supported way past it instead of inventing one.
• rolter check reports every provider that sets it as a warning, and rolter check --strict fails on it. The opt-out is legitimate; being invisible is what would make it dangerous.

Only file-configured providers can set it today. Providers stored in Postgres keep the safe default: the column, the CRUD surface and the dashboard control are a separate change, tracked as its own issue.

Consequences

• The dogfooding fleet declares kind = "openrouter" against 127.0.0.1:18002, so the OpenRouter dialect is exercised by a running fleet for the first time.
• An operator behind an egress proxy can use the hosted kind rather than downgrading to openai_compatible and losing the dialect’s behaviour.
• A deployment that does not set the flag is bit-for-bit unaffected.
• The flag is per provider, not global, so enabling it for a test endpoint cannot silently unpin a production one beside it.
• A stored provider cannot opt out until the DB surface lands, which is a visible asymmetry between the two configuration tiers for as long as it stands.

A served OpenAPI document for the control-plane API

Status: Accepted · Date: 8 Sep 2026 · Issues: #1040, #421, #422, #851 Relates: ADR-0027 (end-to-end test harness)

Context

The gateway describes itself. crates/rolter-gateway/src/openapi.rs hand-builds an OpenAPI 3.1 document, serves it at GET /openapi.json, and renders it with a Scalar bundle embedded in the binary at GET /docs. A client can point a generator at a running gateway and get the OpenAI/Anthropic-compatible surface.

The control plane does not. Its management API — roughly 160 paths and 240 operations across tenancy, users, RBAC, providers, routes, virtual keys, budgets, guardrails, MCP, SSO, SCIM, alerting and the deployment settings singletons — exists only as an Axum router assembled in crates/rolter-control/src/lib.rs from about thirty router() functions. The only machine-readable description of it is the dashboard’s own ui/src/lib/api.ts, which is a client, not a contract.

Three SDKs are planned: Rust (#421), Python (#422) and JS/TS (#851). Without a schema each of them hand-writes the same 240 operations, and each of them re-hand-writes them every time an endpoint is added. Nothing tells an SDK author that a new endpoint exists, so the three clients drift from the server and from each other, independently and silently.

There is a second, sharper problem: even with a document, a hand-maintained one goes stale the first time somebody adds an endpoint and forgets. A schema that is 95% right is worse than none, because a generated SDK reports the missing 5% as “not part of the API” rather than as “not documented yet”.

Options considered

1. Accept hand-written SDK surfaces. Cheapest today. It moves the drift problem into three repositories instead of solving it once, and puts the cost on whoever adds the 241st endpoint, three times over.
2. Derive the document from the handlers with utoipa. Attractive because the annotation lives next to the code it describes. It means adding a proc-macro dependency and its whole derive graph to a crate that currently has neither, annotating every handler and every request/response type in one change, and accepting that the derived schema describes rolter’s Rust types rather than its wire contract — the two diverge wherever a handler flattens, renames or hides a field, which several of them do (VirtualKey.key_hash, User.password_hash, CreatedVirtualKey’s flattened plaintext).
3. Hand-build the document, and make a test enforce completeness. The gateway’s existing pattern, plus the missing half: a coverage gate that fails the build when a mounted route has no row in the document.

Decision

Serve it, option 3.

crates/rolter-control/src/openapi.rs hand-builds an OpenAPI 3.1 document from one table of operations and serves it at GET /openapi.json, with the same embedded-Scalar reference at GET /docs the gateway already ships. This matches the gateway’s approach rather than introducing a second OpenAPI toolchain into the workspace, and it keeps the document describing the wire contract rather than the Rust types behind it.

Completeness is enforced, not hoped for. every_registered_route_is_documented walks every route registration under crates/rolter-control/src/ — reading the source, so it cannot drift from what is actually mounted — and fails naming any (path, method) with no row in the table. every_documented_route_is_registered runs the check in reverse, so a deleted endpoint cannot linger in the schema. This is the same shape of drift guard as the_matrix_lists_every_capability_exactly_once in rbac_matrix.rs and the_gateway_never_uses_a_poisonable_mutex in the gateway’s lock_discipline test: a list that must not silently fall behind the code gets a test that compares it to the code.

Body schemas are deliberately uneven, and the gate is what makes that safe. Endpoints whose handlers already have a stable serde shape — tenancy, users and memberships, providers and provider groups, routes and route targets, virtual keys, budgets, rate limits and model prices — are typed against 47 entries in components/schemas. Surfaces whose payload is genuinely dynamic — settings singletons, guardrail configs, SCIM envelopes, analytics rows — are documented as open objects. Every one of them still appears with its path, method, tag, path parameters and error response, so a generated SDK has the call even where it does not yet have the struct, and tightening one later is a local change rather than a discovery problem.

Consequences

• The three SDK issues can generate from a schema instead of transcribing a router, and a fourth client gets the same starting point for free.
• Adding a control-plane endpoint now costs one row in operations(). Forgetting it is a test failure that names the route, not a silent gap — which is the whole point, and the reason a hand-built document is defensible here.
• rolter-control gains the scalar_api_reference dependency the gateway already carries. The bundle is embedded, so /docs still works air-gapped.
• The document describes the surface the binary can serve, not the subset a given process happens to have mounted: the CRUD routes only mount with a postgres pool, but the schema lists them unconditionally. A schema that changed shape with deployment configuration would be useless to generate from.
• The coverage gate reads source text rather than introspecting the router, because Axum exposes no way to enumerate registered routes. It skips comment lines and #[cfg(test)] items, and a route registered through anything other than a literal path would escape it. That is a known limit, and the crate registers every route as a literal today.
• Loose bodies are a debt that is now visible and enumerable rather than invisible: every operation carrying an open object is one grep away.

One model for turning a subsystem off

Status: Proposed · Date: 8 Sep 2026 · Issues: #1073, #535 Relates: ADR-0030 (served OpenAPI document)

Context

rolter ships a growing set of optional subsystems, and an operator who wants to turn one off has to find out, per subsystem, which of several unrelated mechanisms applies. #1073 describes this as three mechanisms coexisting without written boundaries. The survey below found eight.

What exists today

Mechanisms 1 and 2 are not alternatives — 1 drives 2. apply_feature_flags() writes the deployment flag into the subsystem’s own enabled field, and for two flags it does something else entirely: cache_aware_routing off rewrites every cache-aware route’s strategy to PowerOfTwo, and complexity_routing off removes a route param. So “off” already means three different things in one function: clear a boolean, substitute a fallback behaviour, and delete configuration.

The three gaps

Coverage. Ten subsystems have no deployment-wide off switch: MCP gateway, PII sanitizer, guardrail webhooks, semantic cache, budgets, rate limits, plugins, payload capture, realtime, analytics. Several have a per-object enabled (mechanism 6), which is not the same thing — an operator disabling a subsystem should not have to edit every row.

The dashboard does not react. ui/src/lib/nav.tsx:167 and ui/src/App.tsx:131 reference feature flags exactly once each, and only to register the FeatureFlags screen itself. No other nav entry or route consults flag state, so disabling a subsystem leaves its nav entry and screen in place, reachable, backed by an API that now returns nothing useful.

No shared definition of “off”. Nothing says whether a disabled subsystem should reject, bypass, 404, or merely hide. Each subsystem answers independently, which is why mechanisms 3, 4 and 5 exist at all: each was a locally reasonable choice made without a rule to follow.

Decision

A. Two layers, and only two

• Capability (compile-time or infrastructure): can this deployment run the subsystem at all? Sources are cargo features (7) and infrastructure presence (8). Not operator-settable at runtime; surfaced read-only.
• Enablement (deployment-wide, DB-backed, hot-reloadable): should this deployment run it? One boolean per subsystem in FeatureFlagsConfig.

Per-object enabled rows (6) stay, and are explicitly not part of this model: they answer “should this provider be used”, not “does this deployment run provider health checks”. The ADR’s rule is that a subsystem must never be disabled only by emptying its collection or unsetting an Option — mechanisms 3, 4 and 5 are demoted from enablement mechanisms to configuration details, and each gains a real flag.

A cargo feature can never be a dashboard switch. That is the whole reason the capability layer is separate, and the reason unavailable_flags() already exists: it is the precedent for “the switch exists, but this deployment cannot honour it”, and it should be extended rather than duplicated.

B. Off semantics by class

The rule behind the table: a disabled subsystem never changes an answer the client already had a right to. Filters and optimisations bypass; surfaces that would otherwise lie say 501.

C. Propagation

Flags the data plane reads travel the existing path: control plane → /internal/snapshot → ArcSwap in the gateway. Per AGENTS.md any new flag column needs a bump_config_version() trigger migration, or the gateway serves stale config forever.

The dashboard learns state from the existing GET /api/v1/feature-flags, extended with the capability layer, rather than a second bootstrap payload — the response already carries unavailable, which is the same shape.

D. Config-file reconciliation

rolter.toml’s [feature_flags] is the boot default; the DB row wins once the control plane has one, matching how every other config table already behaves. A store-less gateway uses the file alone. rolter check warns when the file disagrees with the DB, because silently ignoring a file an operator edited is how a deployment ends up with a flag nobody can explain.

E. Naming

<area>.<subsystem>, grouped for the dashboard by area: routing, safety, caching, observability, integrations. Flags are permanent operational toggles, not release flags — so no flag is ever added “temporarily”, and removing a subsystem removes its flag in the same change.

Consequences

• Every optional subsystem gets one switch in one place, and the dashboard can finally hide what is off.
• Default-on is guaranteed by construction: every new flag defaults true via #[serde(default = "default_true")], so an upgrade that adds flags cannot change behaviour. This wants a test asserting FeatureFlagsConfig::default() is all-true, so the guarantee cannot rot as flags are added.
• Ten subsystems need a flag, a migration with a bump_config_version() trigger, snapshot plumbing, an off-semantic implementation and a dashboard switch. That is the follow-up work, sized per subsystem rather than as one change.
• Mechanisms 3, 4 and 5 do not disappear; they stop being the way to disable something. Existing behaviour is unchanged — an empty plugin list is still inert — but it is no longer the documented control.
• The 501 choice for gateway surfaces is a wire-visible decision. It is the one part of this ADR a client can observe, so it belongs in the OpenAPI document (ADR-0030) when implemented.