rolter documentation

Start here. rolter is a high-performance OpenAI/Anthropic-compatible AI gateway and load balancer (Rust data plane + control plane, shadcn/ui dashboard).

Architecture

• Overview — system shape, crates, data/control plane split
• Load balancing — strategies and the cache-aware design
• Caching — response cache and KV-cache affinity
• Provider prompt caching — portable Anthropic prompt-cache controls
• Multi-region deployment — active-active/passive state and failover topology
• Config & hot reload — reload-free updates
• Data model — tenancy, keys, pricing, budgets
• RBAC & auth — roles, virtual keys, membership sources
• Invitations — one-time onboarding links for deployments with no IdP
• Single sign-on (OIDC) — optional OIDC login and group→role mapping
• MCP OAuth — MCP servers, consent grants, sealed token sessions
• SCIM provisioning — IdP-driven user lifecycle and provisioning tokens
• Security — secret handling, threat model
• Observability — metrics, tracing, logs
• Performance — hot-path principles and targets

Decisions

• ADRs / decision log

API

• OpenAI & Anthropic surface

Development

• Setup
• Testing
• Contributing
• Commit conventions
• Packaging (uv / cargo / docker)

Deployment

• Zero-config quickstart — start with no keys, add providers at runtime
• Configuration reference
• Custom CA bundles — trust private upstream PKI without disabling TLS verification
• Docker

Planning

• Roadmap
• TODO

Architecture overview

rolter is split into a data plane (the hot proxy path) and a control plane (management, RBAC, config writes, UI host), sharing a set of Rust library crates. This separation lets the proxy stay lean and fast while the control plane handles CRUD, auth, and persistence.

Components

flowchart LR
  subgraph clients[Clients]
    C([OpenAI / Anthropic SDKs])
    A([Dashboard user])
  end
  C -->|/v1/*| GW["rolter-gateway<br/>data plane (Axum)"]
  A --> CTL["rolter-control<br/>control plane + UI host"]
  GW -->|forward + stream| UP["Upstreams<br/>OpenAI · Anthropic · vLLM pool"]
  CTL -->|read/write| PG[("PostgreSQL<br/>source of truth")]
  CTL -->|publish change events| RDS[("Redis / Valkey")]
  RDS -->|notify| GW
  GW -->|atomic snapshot swap| GW
  GW -->|async batched writes| CH[("ClickHouse<br/>logs + cost")]

Crates

• rolter-core — config model, domain types, error type, telemetry init. No I/O.
• rolter-balancer — LoadBalancer trait + strategies; pure and unit-tested.
• rolter-proxy — Forwarder over pooled reqwest clients; header injection, model rewrite, streaming, per-provider egress proxy.
• rolter-store — repository traits + in-memory impl; Postgres/Redis/ClickHouse backends land behind features.
• rolter-auth — virtual-key verification, roles, access checks.
• rolter-gateway (bin) — the data plane.
• rolter-control (bin) — the control plane + static UI host.

Data plane (rolter-gateway)

Built on Axum/Hyper/Tower. Holds an ArcSwap<Snapshot> routing table; reads are lock-free. Per request it:

1. parses the JSON body enough to read model and stream
2. authenticates the virtual key (when keys are configured) and checks model access
3. resolves the route, asks the route’s balancer to pick a target, then observes it (cache-aware learning)
4. forwards to the upstream provider with the right auth header, rewriting the model id
5. streams the response body straight back with minimal copying

See data plane details in performance.md.

Control plane (rolter-control)

Hosts the management REST API consumed by the dashboard and serves the built SPA as static assets (ServeDir). It owns all writes to Postgres, enforces RBAC, and publishes a config-change event to Redis so gateways hot-swap their snapshot without a restart (see config-and-hot-reload.md).

Datastores

• PostgreSQL — source of truth for tenancy, RBAC, providers, routes, virtual keys, pricing, budgets.
• Redis / Valkey — response cache, rate-limit counters, cooldown state, and pub/sub for config propagation.
• ClickHouse — high-volume request and cost logs, queried by the dashboard for usage analytics.

Request lifecycle (sequence)

sequenceDiagram
  participant Client
  participant Gateway
  participant Balancer
  participant Upstream
  participant ClickHouse
  Client->>Gateway: POST /v1/chat/completions
  Gateway->>Gateway: auth virtual key + model access
  Gateway->>Balancer: pick(route, ctx)
  Balancer-->>Gateway: target index
  Gateway->>Upstream: forward (Bearer/x-api-key, rewritten model)
  Upstream-->>Gateway: response stream
  Gateway-->>Client: streamed body
  Gateway--)ClickHouse: async usage + cost log

Extensibility

• New balancing strategy: implement rolter_balancer::LoadBalancer, wire into build().
• New provider protocol: extend ProviderKind and rolter-proxy translation.
• New storage backend: implement rolter-store traits behind a cargo feature.
• New modality (audio/image/video): add endpoints + provider adapters; the balancer/auth/logging layers are modality-agnostic.

Load balancing

Each route maps a public model name to one or more upstream targets and a strategy. Strategies implement rolter_balancer::LoadBalancer:

#![allow(unused)]
fn main() {
pub trait LoadBalancer: Send + Sync {
    fn name(&self) -> &'static str;
    fn pick(&self, ctx: &RouteContext, loads: &[u64]) -> Option<usize>;
    fn observe(&self, target: usize, ctx: &RouteContext) {}
}
}

pick returns an index into the route’s targets; observe lets learning strategies (cache-aware) record what a target served. RouteContext carries an optional session_key (from x-session-id) and the request prompt used for affinity scoring.

Strategies (v1)

• round_robin — sequential rotation; predictable, zero state.
• random — uniform random; good for simple homogeneous pools.
• power_of_two — pick the less loaded of two random targets; needs a load snapshot.
• consistent_hash — hash-ring keyed by session_key (falls back to prompt hash); pins a session/user to a target for KV reuse, survives target changes with minimal reshuffle (160 vnodes).
• cache_aware — approximate prefix affinity; see caching.md.
• weighted — smooth weighted round-robin honouring each target’s weight.
• pipeline — composable filter → weighted-score → argmax selection: eligibility filtering drops ineligible targets, then a stack of Scorers (session affinity + static weight + in-flight load + prefix-cache affinity) is combined as a weighted sum and the argmax wins (ties broken randomly). Session affinity pins repeat requests from the same x-session-id to their last-served target (TTL-bounded) for warm-cache reuse. The extension point every future cost/latency/KV-cache scorer plugs into.
• precise_cache_aware — consumes each target’s vLLM ZMQ KV-event stream and scores the exact leading fraction of caller-supplied token blocks resident on that target. Missing token ids and stale, malformed, disconnected, or sequence-gapped streams stay neutral; least-load routing remains the fallback.
• lmcache_aware — polls each target’s configured LMCache controller signal and prefers available caches with free capacity (1 - occupancy). Empty, saturated, failed, and stale controllers stay neutral and fall back to least load.
• adaptive — a weighted blend of observed latency, catalog cost and in-flight load, governed by the deployment-wide [adaptive_routing] policy. See below.
• predicted_latency — rank targets by what this request is modelled to cost on each of them, from the queue it would join and its own prompt size, rather than by a per-target average. See below.
• lora_aware — LoRA-adapter affinity for a fleet serving many adapters over shared base weights: prefer a target that already holds the requested adapter resident, with prefix affinity and in-flight load behind it. See below.

Adaptive routing

adaptive is the only strategy whose behavior is owned by a global policy rather than the route:

[adaptive_routing]
enabled = false           # kill switch; the whole feature is off by default
latency_weight = 1.0
cost_weight = 0.5
load_weight = 0.25
exploration_ratio = 0.05  # clamped to [0, 0.5]
min_samples = 50

Three conditions must all hold before the blend routes a single request: the kill switch is on, at least one weight is non-zero, and the route has both served min_samples requests and gathered latency samples for at least two targets. Until then — and immediately again if the policy is switched off — every pick goes to the same pipeline stack the route would have used otherwise, so moving a route to adaptive shifts no traffic on its own. The fallback stack keeps learning while the blend is engaged, so disengaging lands on a warm session/prefix cache.

The policy is also owned by the control plane: GET/PUT /api/v1/adaptive-routing-policy (superadmin only) persists it, audits the change as adaptive_routing_policy.update, and returns the affected_routes the change reaches. It travels to the data plane in the normal snapshot, so a change applies without a restart. The API refuses an all-zero blend outright — stopping adaptive routing is what enabled = false is for.

Once engaged, an exploration_ratio share of picks is made uniformly at random so a target the blend has learned to avoid keeps producing fresh latency samples instead of going dark. Operator input is clamped on the way in: negative weights become zero and exploration never exceeds half the traffic.

Telemetry

The scores the blend ranks on live in the balancer, which lives in the gateway process, so the control plane cannot read them directly. Each gateway therefore pushes a sample every 15s to POST /internal/adaptive-telemetry on the control plane — the same internal token and the same x-rolter-node-id identity as the snapshot poll, so one node is one row here and in the cluster inventory (#543). Nothing is written on the request path: the sample is taken by a background task, and a pick costs two relaxed atomics for per-target attribution.

Per route and target the sample carries the blended score, its latency/cost/load components, the raw signals behind them (smoothed latency in ms, catalog price, in-flight count), how many picks that target has served and how long ago the last one was — plus the decision split, and the sanitized policy that node actually runs, which can lag the stored policy until the node converges.

The control plane keeps the newest sample per (node, model) in adaptive_routing_telemetry and serves GET /api/v1/adaptive-routing-telemetry (superadmin only), grouped by route with one entry per reporting node. Samples older than 60s are excluded as no longer current, and rows are pruned after an hour so a scaled-down node leaves the scoreboard.

This is deliberately current state, not history: one row per node and route, overwritten on every report, so the table is the size of the fleet rather than of the traffic. A time series belongs in the request log, and would be a separate endpoint rather than a change to this one. Prometheus deployments get the same per-target scores without the dashboard from rolter_adaptive_routing_target_score{model,target}.

LoRA-aware routing

For a fleet serving many LoRA adapters off shared base weights, routing a request to a node that already holds the adapter is the same class of win as prefix-cache affinity (#853, borrowed from llm-d). lora_aware composes with the existing scorers rather than replacing them:

Adapter residency outranks prefix affinity because the costs are not comparable: missing a warm prefix recomputes some tokens, while missing a resident adapter can force the engine to load adapter weights before it decodes anything at all. Load stays in the stack so a fleet where every target holds the adapter still balances instead of pinning.

Residency is learned from traffic, not declared. rolter cannot see which adapters an engine currently holds, and a static declaration would go stale the moment the engine evicted one. Each target keeps a bounded LRU set of the adapters it has recently served, sized to mirror vLLM’s --max-loras, so it tracks what the engine plausibly still holds rather than everything it has ever served. An unbounded “has ever served” set would be worse than no scoring at all, because it steers confidently to a target that went cold long ago.

Adapter identity is the requested model, which is how vLLM addresses adapters over shared base weights. The gateway only sets it when the request addresses something other than the route’s own model — that is, a passthrough provider-group route (ADR-0017). On a single-model route the two are equal, no adapter is set, and the scorer is inert. This matters: adapter affinity deliberately pins rather than spreads, so treating a route’s one model as an adapter would pin the entire route to whichever target happened to serve first.

When no candidate holds the adapter — a cold adapter, or a request with none — every candidate scores 0.0. That is neutral rather than a penalty: an equal contribution cannot move the argmax, so the rest of the pipeline decides and this scorer stays silent instead of guessing.

Predicted-latency scheduling

fastest ranks on a per-target latency EWMA. That is the right signal when every request is the same size, and the wrong one otherwise: a target whose average is high because it happened to serve the long prompts looks slow even when it is the emptiest box in the fleet, and a target with a deep queue looks fast right up until the queue is what you join.

predicted_latency models the cost instead of averaging it (#853, borrowed from llm-d). Per target:

latency_ms ≈ w0 + w1 · queue_depth + w2 · prompt_ktokens

w0 is fixed overhead, w1 is what one queued request ahead of you costs, and w2 is prefill cost per thousand prompt tokens. The three coefficients are learned online from completed requests with normalized least mean squares — one multiply-add per feature per sample, no matrix, no allocation, no periodic refit.

Load stays in the stack for two reasons: it carries the route while the models are cold, and once they are warm it is the tiebreaker between targets the model rates equally — the common case on a homogeneous fleet, where the honest answer is “either, take the emptier”.

Three features, not more. A gateway sees queue depth and prompt size. It does not see batch composition, KV-cache pressure, or where the engine is in its scheduling loop, and a model with parameters it cannot observe fits noise. The interesting error is between “the queue is deep” and “the prompt is long”, and a linear model separates those. That is the honest ceiling for this vantage point.

A cold target predicts nothing. Below 8 completed requests a target returns no prediction, and the scorer reads that as unknown, not as slow. A route that switches to predicted_latency therefore behaves exactly like the least-load pipeline until the models have evidence, so the switch itself moves no traffic.

Only successful requests teach. The same gate the latency EWMA uses: a fast failure cannot train a target into looking cheap. Coefficients are clamped, so a client that held a stream open for a week produces one bad sample rather than a permanently poisoned model.

Queue depth is read before the increment, so the model learns the queue a request joined, not the one it created.

Models live in the load tracker, not the routing snapshot, for the same reason the latency EWMA does: a config reload must not throw away what they learned, or every reload would send the route back to cold behaviour. A route that gains a target keeps its existing models and the new target reads as unpredicted until the next process start — the conservative direction, since a target with no evidence should not be ranked.

Choosing a strategy

Both external strategies perform network I/O only in background tasks. The request hot path reads bounded in-process state and atomics.

Selecting a strategy from the dashboard

The route and provider-group editors offer every strategy in this table except adaptive, which is governed by the deployment-wide [adaptive_routing] policy and has its own screen — a per-route dropdown would misrepresent how it is controlled. precise_cache_aware and lmcache_aware are offered with a hint that they need a telemetry source on the target providers, since without one they fall back to least-load silently rather than failing.

A picker always renders the value the route or group already holds, even one it would not otherwise offer — including a strategy set from rolter.toml or the API, or one added to the backend allowlist ahead of the dashboard. A native <select> whose value matches no option displays the first option instead, so before #897 a group balanced by adaptive read as round_robin. Whatever the menu chooses to offer, editing must never rewrite a strategy the operator did not touch.

Caching

rolter deals with two distinct kinds of caching.

1. KV-cache affinity (load balancing)

The big win for self-hosted fleets: vLLM/SGLang reuse the attention KV cache for shared prompt prefixes (system prompts, few-shot examples, conversation history). But that only helps if the next matching request lands on the same replica. Naive round-robin scatters related requests and destroys cache locality.

The cache_aware strategy keeps, per target, a byte trie of prompts it has served. For an incoming prompt it computes the fraction of leading bytes already present on each target and:

• if the best match ≥ threshold (default 0.5), pins the request to that target (cache hit)
• otherwise spreads to the least-warmed target (or least loaded once load is wired)

This is approximate (no coupling to the engine). The per-target trie is capped at a node ceiling (default 1M nodes) with LRU eviction: inserting past the cap drops the least-recently-inserted prompt, pruning only the nodes that become unreferenced (shared prefixes survive). Each trie exposes an eviction counter for observability.

flowchart TD
  R[incoming prompt] --> S{best prefix match >= threshold?}
  S -- yes --> P[pin to best target<br/>cache hit]
  S -- no --> L[least-warmed / least-loaded target]
  P --> O[observe: insert prompt into target trie]
  L --> O

Precise vLLM mode

Set a provider’s [providers.kv_events] block and choose strategy = "precise_cache_aware". rolter supports the vLLM V1 msgpack KVEventBatch protocol documented for vLLM 0.22: a three-frame ZMQ publication (topic, big-endian sequence, payload) containing tagged BlockStored, BlockRemoved, and AllBlocksCleared events. BlockStored.token_ids and block_size derive stable local prefix identities; external block hashes allow exact removals. State is capped by max_blocks per provider.

Exact scoring needs the token ids produced by the same tokenizer as vLLM. Send them in x-rolter-vllm-token-ids as comma-separated unsigned integers. Without the header, with stale/malformed events, or after a sequence gap, the scorer is neutral and least-load routing takes over. A sequence gap clears the local index and precise scoring remains disabled until an AllBlocksCleared event establishes a clean boundary.

[[providers]]
name = "vllm-a"
kind = "openai_compatible"
api_base = "http://vllm-a:8000"

[providers.kv_events]
endpoint = "tcp://vllm-a:5557"
topic = "kv-events"
max_blocks = 1000000
stale_secs = 30

vLLM must enable KV events with the ZMQ publisher and matching topic. Metrics expose consumed/malformed events, stream failures, decision count, and per-provider freshness.

LMCache-aware mode

Set [providers.lmcache] and use strategy = "lmcache_aware". The supported controller signal is an HTTP 200 JSON object:

{"occupancy": 0.42, "cache_available": true}

occupancy is clamped to [0,1]; available targets score 1 - occupancy, unavailable targets score zero. Polling happens in the background. Failed, malformed, or stale signals are neutral, so existing least-load routing continues.

[providers.lmcache]
endpoint = "http://lmcache-a:9000/v1/occupancy"
refresh_secs = 2
stale_secs = 10

2. Response cache

Optional caching of full responses to cut cost/latency for repeated requests:

• exact: hash of the normalized request → cached response (Redis), short TTL, opt-in per route/key.
• semantic: after an exact miss, embed the normalized prompt through a configured provider and compare cosine similarity against a bounded recent-entry window in Redis. The route controls the threshold and candidate cap. Embedding, Redis, and decode failures fail open to normal routing.

Streaming responses are cached on completion and replayed as a synthetic stream. Cache status is surfaced via response headers (e.g. x-rolter-cache: hit|miss).

[cache]
enabled = true

[routes.cache]
enabled = true

[routes.cache.semantic]
provider = "openai"
model = "text-embedding-3-small"
threshold = 0.92
max_candidates = 256

Configuration & hot reload

A core requirement: operators change routes, providers, keys, limits and pricing from the UI and have them take effect without restarting the gateway.

Sources of config

1. Bootstrap file (rolter.toml) — used for first run, local dev, and IaC. Maps to rolter_core::GatewayConfig.
2. Database (Postgres) — the runtime source of truth once the control plane is running. The control plane composes a GatewayConfig-equivalent snapshot from normalized tables.

Propagation

sequenceDiagram
  participant UI
  participant Control as rolter-control
  participant PG as PostgreSQL
  participant Redis
  participant GW as rolter-gateway
  UI->>Control: PUT /api/v1/routes/... (RBAC checked)
  Control->>PG: write change in a transaction
  Control->>PG: bump config_version.version
  Control->>Redis: PUBLISH rolter.config {version}
  Redis-->>GW: message {version}
  GW->>Control: GET /internal/snapshot?version=N
  Control-->>GW: full snapshot (JSON)
  GW->>GW: build Snapshot, ArcSwap::store (atomic)

• The gateway keeps the routing table in an ArcSwap<Snapshot>. Swapping is atomic and wait-free for readers — in-flight requests keep using the old snapshot; new requests see the new one.
• Versioning: config_version in Postgres is the monotonic source of truth. The gateway also reconciles on an interval (and at startup) so a missed pub/sub message self-heals.
• Validation: the control plane validates a snapshot (every route target references a known provider, etc.) before bumping the version, so gateways never load a broken config.
• Per-row resilience: validation used to be all-or-nothing, so a single half-built row 500’d /internal/snapshot and froze config propagation for every tenant. Rows that are unservable only by themselves are now omitted from the snapshot instead. That covers both a route with no usable target (created before its targets are added, or pointing at a deleted provider) and a provider whose own definition is invalid — a malformed api_base, a hosted adapter missing its api_key_env. Dropping a provider drops the routes that depended on it, since they are then left with no resolvable target. Structural problems that span rows (duplicate names, a bad metrics_path, unreadable CA bundles) still fail the whole snapshot, since silently dropping one of two colliding rows would be worse than refusing.
• Saying what was dropped (#926): omitting an entry quietly is its own failure mode — every gateway keeps serving its last good config, so nothing looks broken until a gateway restarts cold or someone wonders why a change never took effect. So the reasons travel with the config:
  • /internal/snapshot carries a problems: [...] array alongside version and config, present only when something was dropped. It is absent from a healthy snapshot, which is what every gateway in the fleet transfers on every poll.
  • the gateway logs them once per change, at the point it applies a new version, rather than once per poll — a fleet polling every 5s would otherwise repeat the same complaint thousands of times a day. An older control plane sends no such field, so it defaults rather than failing the decode during a rollout.
  • the dashboard reads GET /api/v1/config/problems and shows them on the Providers screen. That endpoint runs the same computation the snapshot does, so the two cannot disagree; it also reports structural problems, which never reach a gateway at all.

Why this design

• Redis pub/sub gives near-instant fan-out to many gateway replicas.
• Postgres versioning makes the system correct even if Redis drops a message.
• ArcSwap keeps the hot path lock-free; no read ever blocks on a config write.

Alternatives considered: Postgres LISTEN/NOTIFY (avoids a Redis dependency but Redis is already needed for cache/rate limits), and pure polling (simplest, higher latency). See ADR.

Data model

PostgreSQL is the source of truth. The initial schema lives in migrations/0001_init.sql; ClickHouse log schema in clickhouse/001_logs.sql.

Tenancy

erDiagram
  orgs ||--o{ teams : has
  teams ||--o{ projects : has
  projects ||--o{ routes : has
  routes ||--o{ route_targets : has
  projects ||--o{ virtual_keys : has
  orgs ||--o{ providers : has
  providers ||--o{ provider_keys : has
  users ||--o{ memberships : has
  orgs ||--o{ mcp_servers : registers
  users ||--o{ mcp_oauth_grants : consents
  mcp_servers ||--o{ mcp_oauth_grants : authorizes
  orgs ||--o{ mcp_tool_groups : governs
  orgs ||--o| mcp_gateway_settings : configures
  mcp_oauth_grants ||--o{ mcp_oauth_sessions : issues

• Org → Team → Project → Virtual Key is the hierarchy. Budgets and rate limits attach at any scope and combine most-restrictive-wins.
• Providers are owned at the org level and referenced by route targets. Upstream credentials live in provider_keys, envelope-encrypted (see security.md).
• Routes belong to a project and map a public model to route_targets with a strategy.
• Virtual keys belong to a project, store only a hash of the key plus a display prefix, and carry an optional model allow-list.
• MCP servers belong to an org and declare required OAuth scopes and exposed tool names. Only enabled servers reach gateway snapshots. Grants bind a user to a server; sealed token sessions belong to a grant. MCP tool groups persist named server/tool manifests, while MCP gateway settings hold organization defaults for registration and MCP-aware clients; neither is currently a request-path authorization boundary.

Cost & limits

• model_prices — USD per million tokens for input/output (+ cached input). Used to compute cost_usd per request, written to ClickHouse.
• budgets — spend caps per scope and period; enforced before forwarding and refreshed from spend aggregates.
• rate_limits — RPM/TPM per scope; counters live in Redis for multi-instance correctness.

Config versioning

config_version holds a single monotonic counter the gateways watch for reload-free updates (config-and-hot-reload.md). audit_log records who changed what.

The RBAC tables are split on exactly this question. access_profile_policies, access_profile_assignments, access_profiles and memberships all carry a bump_config_version() trigger, because the gateway resolves the model/route policy of each virtual key’s owner from them and enforces it on the request path (#791, ADR-0023). custom_roles and custom_role_grants do not and must not: they decide control-plane authorization, which is evaluated live per request, so a bump would wake the fleet for a change it cannot observe.

plugin_instances is the control-plane registry for desired request/response middleware configuration. A row belongs to an org and may narrow to a project; slugs are unique within that scope. Endpoint credentials are environment-variable references and config is always a JSON object. The table deliberately has no bump_config_version() trigger until the allocation-light gateway dispatcher in #509 consumes it—an enabled registry row is configuration, not a false claim that middleware is running.

The guardrail registry is deployment-wide:

• guardrail_rules stores the ordered built-in and bounded-regex policy managed by the dashboard. Exactly one of builtin or pattern is present on every row.
• guardrail_providers stores external webhook endpoints and environment-variable credential references. A partial unique index permits at most one enabled provider because the gateway exposes one vendor-neutral webhook contract.

Both tables bump config_version in the write transaction. File-owned rules remain immutable and win name collisions; database rules extend that policy. An enabled file-owned webhook remains authoritative, otherwise the active registry provider supplies the snapshot webhook.

Several deployment-wide settings are stored as singleton tables: one row keyed by id boolean primary key default true check (id), seeded by their own migration, so a read never has to handle “not configured yet”. runtime_policy, compatibility_policy, security_settings, logging_settings, client_settings and model_defaults all follow this shape and all bump config_version in the write transaction.

Two of them landed with the Settings screens (#564):

• client_settings — the base URL the dashboard advertises (advisory; the gateway never reads it), the allowlist of inbound client headers forwarded to the upstream provider, the static headers the gateway injects on every upstream request, and the request-id header. Trace-context headers propagate independently of the allowlist, and injected header values are treated as credential material: they reach the gateway through the snapshot but never the audit log, which records only the names.
• model_defaults — an enabled kill switch plus optional default_model, default_temperature, default_top_p and default_max_tokens. Defaults only ever fill a key the request omitted, so the table can be populated without changing the meaning of any request that was already explicit. default_temperature and default_top_p are double precision, not real: an f32 default serializes into JSON as 0.800000011920929 and that is what would reach the provider.

Data written by the data plane

Most tables flow control plane → gateway. Two flow the other way, written from the channel the gateway already holds and never read back by it:

• cluster_nodes — one row per node, upserted from the snapshot poll (#543).
• adaptive_routing_telemetry — one row per (node, model) holding the newest adaptive-routing sample that node pushed (#751): the decision split, the sanitized policy it runs, and a JSON document of per-target scores and signals. A scoreboard, not history — every report overwrites the row, so the table is the size of the fleet rather than of the traffic.

Neither carries a bump_config_version() trigger, and neither may grow one: the data plane does not consume them, and a version bump on every heartbeat or sample would make routine bookkeeping look like a config change and wake the whole fleet.

ClickHouse tables

PostgreSQL holds configuration; ClickHouse holds the high-volume append-only streams, all partitioned by day with a 90-day TTL and written in batches off the hot path. Schema lives in clickhouse/, applied by the container’s init directory:

• request_logs — one row per proxied request, with cost and token counts.
• provider_health_events — per-target health observations from every signal.
• mcp_tool_call_logs — MCP tool invocations.
• ui_events — dashboard UX events (#805): screen views and time-to-interactive, navigation and back-outs, form submit/abandon and which validation rules fire, empty- and error-state impressions, save-to-confirmation latency. Carries trace_id, so a UX event and the gateway request it caused are one join apart.

ui_events is structurally incapable of holding content: every column is a key, an enum, a duration or an id, so “no form values, no prompt text” is a property of the schema rather than a policy applied on write. None of these tables is read by the data plane, so none carries a bump_config_version() trigger.

Migrations are append-only

Migrations are embedded with sqlx::migrate! and sqlx stores a checksum of every migration it applies. If a shipped file’s bytes change — even a stripped trailing newline, as a formatting hook once did in #710 — every database that already ran it refuses to start on its next upgrade:

Error: store error: migration 18 was previously applied but has been modified

CI databases always start empty, so nothing else catches this: the change merges green and breaks deployments later. Never edit, delete or rename a file under crates/rolter-store/migrations/; add a new NNNN_*.sql instead. Numbers are append-only and never reused, even where the sequence has a gap.

scripts/check-migrations-immutable.sh enforces this. It runs as the migrations append-only job in quality.yml (inside the ci-ok gate) and as a prek hook locally, rejecting any modified, deleted or renamed migration relative to the branch’s fork point from master.

Mapping to the gateway

The control plane composes the normalized tables into the same shape as rolter_core::GatewayConfig (providers, routes, virtual keys and authorized MCP sessions), which the gateway turns into an immutable Snapshot.

RBAC & authentication

Two distinct auth surfaces:

1. Gateway (data plane) — virtual keys

Clients call /v1/* with a virtual key (Authorization: Bearer <key> or x-api-key). The gateway:

• looks the key up in the current snapshot
• checks the key’s model allow-list (empty = all)
• (roadmap) enforces budgets and RPM/TPM limits for the key’s scope chain

Keys are stored as hashes; the presented key is compared in constant time (rolter_auth::verify_key).

Empty key sets

An empty effective key set does not mean “auth disabled” on a managed deployment. A gateway started with --snapshot-url (its config comes from the control plane) fails closed: with no virtual keys in the snapshot every /v1/* request gets 401. This keeps revoking the last key a lock-down rather than an accidental opening of the whole data plane.

A gateway running from a static bootstrap config with no keys stays keyless, so local fake-llm development needs no setup.

Override either default with server.require_auth:

2. Control plane (dashboard) — users + roles

Human users authenticate to the control plane. Two providers ship today: local accounts (argon2id password hashes, rolter_control::auth) and OAuth2/OIDC SSO (rolter_control::sso, authorization-code flow with PKCE, JWKS-verified id tokens, IdP group → role mapping). RBAC roles:

• admin — full control within scope (manage providers, routes, keys, members, budgets)
• member — create/edit routes and keys within scope
• viewer — read-only (dashboards, logs)

Roles are granted via memberships at an org / team / project scope. Permission checks resolve the most specific membership for the target resource.

flowchart LR
  U[User] -->|member of| Scope[org / team / project]
  Scope -->|role| Caps[admin / member / viewer]
  Caps --> Action{allowed?}

The capability table is the only source of truth

CAPABILITIES in crates/rolter-control/src/rbac_matrix.rs records, for every resource, what each of read / create / update / delete takes — a minimum scoped role, superadmin-only, or that the resource has no such action at all. Both the published matrix and the guard read it, so they cannot drift.

A guarded handler names a (resource, action) pair instead of a role:

#![allow(unused)]
fn main() {
authorize(&state, &principal, ScopeChain::org(org_id), cap!("provider", Create)).await?;
// deployment-wide, no scope to hold a role in:
authorize_superadmin(&principal, superadmin_cap!("feature_flags", Update))?;
}

cap! resolves the requirement through a const fn, so naming a resource the table does not define — or an action it marks as unsupported — is a compile error, and superadmin_cap! additionally fails to compile unless the table says the pair is superadmin-only. No handler names a Role; unit tests in rbac_matrix.rs scan every control-plane module to keep it that way, check the module list against src/ so a new file cannot slip past, and assert that every row in the table is claimed by at least one guard.

Two read-only endpoints publish that table, so a dashboard never assembles a permission matrix of its own:

• GET /api/v1/rbac/matrix — every role and, per resource, the minimum role each action takes (or that the action is superadmin-only, or unsupported entirely). Any authenticated caller may read it; it describes rules, not anyone’s access.
• GET /api/v1/rbac/effective?org_id=&team_id=&project_id= — the calling principal’s resolved role at that scope chain and the concrete resource:action pairs they may perform, evaluated from their memberships.

effective is advisory to the client and authoritative only on the server: a caller that ignores it and issues the request anyway gets the same 403. Scope precedence is unchanged — a project-scoped grant authorizes that project, not the whole org.

Read access is a viewer’s and mutations are an admin’s, with three deliberate exceptions:

• deployment-wide policy (feature flags, runtime/compatibility/adaptive policy, logging settings, cluster nodes, security settings, alerting, MCP tool-call logs) has no tenancy scope to be a member of, so it is superadmin-only;
• global account lifecycle — creating an org, editing or deleting a user account, and the model/pricing catalog — reaches across orgs, so it is superadmin-only too, while inviting a user into an org stays an org admin’s;
• a user’s own things — minting a virtual key for yourself takes member (a viewer cannot), and revoking your own MCP OAuth grant or session takes only a viewer membership plus ownership, which the handler checks after the guard.

Listing the pricing catalog (GET /api/v1/model-prices) and the effective model list (GET /api/v1/models) is every authenticated caller’s, with no membership anywhere. That is a third authority alongside a scoped role and superadmin, and the table names it rather than implying a role floor: both are deployment-wide catalogs of upstream capability and list price that carry no tenant’s data, and deployment is not a scope a membership can be held at, so a viewer floor there would have described a bar nobody could clear. GET /api/v1/rbac/matrix reports those cells as authenticated_only. The effective model list is still filtered per caller by the access-profile model policy (#534), so what a caller sees remains theirs alone.

Every cell in the table is now backed by the guard.

Custom roles and access profiles

The three built-in roles are a floor, not the whole rule set. An org may define custom roles: a base role plus a set of explicit (resource, action) grants drawn from the same CAPABILITIES table the guard reads. A grant can only widen — a custom role never takes away what its base role already allows, so the built-in roles keep behaving exactly as before and nothing has to be migrated.

Custom roles are not assigned to people directly. They are composed into an access profile, which names each role together with the org, team or project it applies at, and the profile is then assigned to users or teams. One profile can therefore say “auditor at the org, deploy admin on this one project” and be reused across an organization instead of re-granted per person.

flowchart LR
  P[Access profile] -->|role @ scope| CR[Custom role]
  CR -->|base| Built[admin / member / viewer]
  CR -->|grants| Pairs["resource:action pairs"]
  P -->|assigned to| Who[user or team]
  P -->|optional| Pol[model / route policy]

Evaluation order inside authorize is unchanged for the common case: memberships resolve first, and the configurable half is consulted only if the built-in answer was “no”. That keeps a plain deployment on exactly the old code path, and makes a custom grant strictly additive.

A profile may also carry a model and route policy — allow and deny lists over the models and routes its holders may reach. Deny wins over allow. Where a user holds several profiles the lists are unioned rather than intersected, since a second profile must never reduce access; one consequence is load-bearing: a profile with no model restriction at all makes the merged allow-list unrestricted, because that profile already permitted everything on its own.

That policy is published on GET /api/v1/rbac/effective as model_policy, and since #791 it is also enforced by the data plane. The bridge is the virtual key: a policy belongs to a person, but a request carries a credential, so the control plane resolves each key owner’s merged policy when it builds /internal/snapshot and publishes it on the key record. The gateway then applies it in KeyMeta::model_permitted and KeyMeta::route_permitted, alongside the key’s own model allow-list.

Both must permit a model, deliberately. The key list is what the key’s creator scoped that credential to; the policy is what an operator decided the person may reach at all. Neither can widen the other, so a key naming a model its owner is denied stays denied.

The shape, the merge rule and the allow/deny matching all live in rolter_core::ModelPolicy, which the control plane, the store and the gateway share — two implementations of “deny wins” free to drift apart would be a security bug.

Enforcement keys on the virtual key’s created_by. A key with no owner — admin-created and config-defined keys — carries no policy, because there is no person whose profiles could apply; restricting those is still the key’s own model list.

Because the gateway now reads them, four of these tables do carry a bump_config_version() trigger: access_profile_policies, access_profile_assignments, access_profiles and memberships (a profile assigned to a team reaches every member, so a membership change alters someone’s effective policy with no profile row changing). custom_roles and custom_role_grants still do not and still must not: they decide control-plane authorization, which is evaluated per request against the live database, so there remains nothing to propagate and a trigger would only wake the fleet for a change it cannot observe. See ADR-0023 for why the policy is resolved at snapshot time rather than when a key is minted.

Changing one is safe by construction:

• deleting a custom role that a profile still references returns 409 rather than silently emptying the profile’s composition; detach it first;
• deleting a profile does cascade its own assignments, since the assignment has no meaning without it;
• every create, update and delete on a role, a profile, an assignment or a policy is written to audit_log with the before/after, because all of them change what real people can do.

GET /api/v1/rbac/matrix?org_id= returns the org’s custom roles alongside the built-in ones, so the dashboard’s matrix is API-backed and updates after any change instead of holding state of its own.

Endpoints:

• GET/POST /api/v1/orgs/{org_id}/custom-roles, GET/PUT/DELETE /api/v1/custom-roles/{id}
• GET/POST /api/v1/orgs/{org_id}/access-profiles, GET/PUT/DELETE /api/v1/access-profiles/{id}
• GET/POST /api/v1/access-profiles/{id}/assignments, DELETE /api/v1/access-profile-assignments/{id}
• PUT /api/v1/access-profiles/{id}/policy

Identity providers (ROL-35)

Local login and SSO both end at the same place: a verified identity that gets turned into a session and reconciled memberships. That shared shape is rolter_auth::IdentityProvider — an async trait with one method, resolve(Credential) -> Result<Identity, IdentityError> — so a new provider only has to prove who someone is; everything downstream (session issuance, group → role reconciliation, audit logging) is unchanged.

• rolter_control::auth::LocalIdentityProvider verifies a Credential::Password, preserving the original handler’s timing-safety property: every rejection path (unknown account, deactivated, sso-only, password login disabled by org policy, wrong password) still runs exactly one argon2 verification, so response time reveals nothing about which case applied.
• rolter_control::sso::OidcIdentityProvider verifies a Credential::AuthorizationCode, wrapping the existing code-exchange and JWKS id-token verification.
• Concrete providers live in rolter-control (next to the sqlx/reqwest they need), not in rolter-auth, which only defines the trait and stays free of those dependencies.

Roadmap

• LDAP — bind + group mapping for enterprise directories (#241), the next provider to implement IdentityProvider.
• JWT service auth and short-lived tokens.
• Audit log surfaced in the UI.
• Optional constant-time map / pepper for virtual-key lookup hardening.

Invitations

Invitations are how a rolter deployment onboards people without an identity provider. An admin mints a one-time link, the invitee opens it and chooses their own password, and the role the invitation carries is granted on acceptance.

This is deliberately not the same as POST /api/v1/orgs/{org_id}/users, which creates an account with a password the admin picked. That is right for seeding and for service accounts, and wrong for onboarding a colleague: it leaves someone else holding their credential — usually in a chat log.

The flow

1. POST /api/v1/orgs/{org_id}/invitations (admin at the target scope) returns the invitation, the token, and a ready-made accept_url. The token is shown exactly once; only its peppered SHA-256 digest is stored, the same treatment sessions and virtual keys get, so a database dump alone yields no usable link.
2. GET /api/v1/invitations/accept/{token} — unauthenticated preview returning the org name, the invited email, the role and the expiry. Nothing else: a link that leaked should not also leak a directory.
3. POST /api/v1/invitations/accept/{token}/accept with the invitee’s chosen password creates the account, grants the membership, and returns a live session so they land signed in.
4. DELETE /api/v1/invitations/{id} revokes a pending invitation.

Links expire after seven days.

Failing closed

Expired, revoked, already-accepted and simply wrong tokens all return the same 401. The caller learns whether their link works, not which of those it is.

Acceptance is single-use even under a race: the claim is an UPDATE … WHERE accepted_at IS NULL, which either affects one row or tells the loser they lost — before any account or membership is created.

One live invitation exists per email per org, enforced by a partial unique index. Re-inviting replaces rather than accumulates.

Existing accounts

Acceptance adopts an account that already exists under the invited email rather than forking a second row for the same person. Someone may already hold a login in another org, or have arrived through SSO first.

• An account with a password keeps it. An invite link is not a password reset.
• An SSO-only account (no password) gains the password it was invited to set.
• A deactivated account cannot be revived by an invitation (403).

Relationship to single sign-on

Invitations and single sign-on co-exist; neither requires the other.

The membership an acceptance grants carries source = 'manual', and an SSO login only ever reconciles source = 'sso' rows. So an invited role survives every later IdP login, while roles that came from IdP groups are recomputed each time. A deployment can run invitations only, SSO only, or both at once.

Configuration

rolter does not send the email itself: it returns the link and lets the operator deliver it however their organization already delivers things. That keeps the control plane free of an SMTP dependency, which matters for air-gapped deployments.

Related

• Single sign-on (OIDC) — the IdP path, and how membership sources keep the two from fighting
• RBAC & auth — roles and scopes
• SCIM provisioning — IdP-driven lifecycle for deployments that want accounts created without anyone clicking a link

Single sign-on (OIDC)

rolter can authenticate operators against an OIDC identity provider — Keycloak, Okta, Entra ID, Auth0, Google Workspace — and turn the groups that provider reports into rolter roles.

Single sign-on is optional and additive. A deployment that never registers a provider behaves exactly as it did before: local accounts, passwords, and operator-granted roles. Nothing about SSO is reachable, and the login screen never mentions it.

The three supported deployments

The dashboard asks GET /api/v1/auth/methods — the one unauthenticated endpoint in this area — and renders whichever of the three it is told. That endpoint returns provider names, slugs and start URLs only; all of which are already visible in the login URL, and none of which are secret.

The flow

Authorization code with PKCE, no implicit grant, no client-side tokens:

1. GET /auth/sso/{slug}/start mints a state, a nonce and a PKCE verifier, stores them in sso_login_states, and redirects to the provider’s authorization_endpoint.
2. The provider redirects back to GET /auth/sso/{slug}/callback.
3. The callback consumes the state row (DELETE … RETURNING), so a replayed code + state pair finds nothing and is refused. States older than ten minutes are treated as absent and swept.
4. The code is exchanged at the token_endpoint with the PKCE verifier and the sealed client secret.
5. The id token is verified against the provider’s JWKS: signature by kid, issuer, audience (client_id), expiry, and the nonce from step 1.

Rules that hold on every path:

• The redirect URI comes from configuration (ROLTER_PUBLIC_URL), never from the request, so an attacker cannot point the callback elsewhere.
• Only asymmetric algorithms are accepted (RS256/384/512, ES256/384, PS256). HS256 and none are rejected — a symmetric id token signed with a value the attacker may know is not evidence of anything.
• The discovery document’s issuer must equal the configured issuer.
• A failed token exchange reports the HTTP status only. The provider’s error body can echo the client secret back, and that must not reach a log.

Groups become memberships

An operator maps an IdP group to a role at an org, team or project scope:

POST /api/v1/sso-providers/{id}/group-mappings
{"group_name": "platform", "role": "admin", "team_id": "…"}

The groups claim is read in every shape providers actually send it — a JSON array, a single string, or a space-separated list — and Keycloak’s leading / is stripped, so operators map the group name they see in the IdP’s own UI. The claim name is configurable per provider (group_claim, default groups).

If no mapping matches, the provider’s default_role applies. If there is no default_role either, the login is refused: SSO authenticates, it does not implicitly authorize.

Manual and SSO grants co-exist

Every membership records where it came from:

• source = 'manual' — an invitation, the admin API, the seed command.
• source = 'sso' — an IdP group mapping.

Each login reconciles only the sso rows inside that provider’s org. So:

• A role an operator granted by hand survives every SSO login, forever.
• Dropping a user from a mapped group revokes the role that group granted, on their next login — including the login that is then refused for having no grants left. That is how deprovisioning through the IdP works without SCIM.
• Another org’s SSO grants are untouched; they belong to that org’s provider.

An account is adopted by verified email: someone invited last month who now arrives through the IdP keeps the same user row, the same virtual keys and the same manual roles. SSO-created accounts get no password — an SSO identity must not silently gain a second, weaker credential.

Org login policy

PUT /api/v1/orgs/{org_id}/auth-policy (org admin) sets two flags:

• allow_password_login — when false, members of this org cannot use the password form.
• allow_sso — when false, callbacks for this org’s providers are refused without deleting the provider rows, so an IdP can be cut off in one request.

Two guard rails, both returning 409:

• Both flags off is not a policy, it is an outage.
• Password login cannot be disabled before an enabled provider exists.

And one exemption: a superadmin can always log in with a password, whatever the policy says. A mistyped issuer or an IdP outage would otherwise lock the deployment out with no way back in. That exemption is the reason the flag is safe to turn on at all; keep the superadmin’s password strong and stored somewhere the IdP does not gate.

Configuration

Register the redirect URI "$ROLTER_PUBLIC_URL/auth/sso/{slug}/callback" with the identity provider.

Testing

Unit and Postgres-gated integration tests drive a stub IdP in-process, which covers rolter’s own logic. Interoperability is a separate question, so the e2e harness runs the same flows against a real Keycloak — genuine discovery document, real JWKS, real login form, real /-prefixed realm groups:

cd integration/e2e && uv run pytest tests/test_sso.py --idp

That suite is nightly and on-demand (.github/workflows/sso-e2e.yml), not part of the per-PR gate.

Related

• RBAC & auth — roles, scopes and how they are enforced
• SCIM provisioning — IdP-driven account lifecycle, which pairs with SSO but is independent of it
• Security — secret handling and the threat model

MCP servers, OAuth grants and sessions

rolter stores the OAuth state behind Model Context Protocol access: which MCP servers an org has registered, which users consented to what, and the token sessions held against those consents. Transport and the authorization-code / on-behalf-of exchange belong to the MCP proxy; this layer owns persistence, listing, revocation and audit, so it is useful to a stdio, SSE, streamable-HTTP or WebSocket implementation alike.

Model

mcp_servers (org-scoped)
   ├── mcp_oauth_grants   one live grant per (server, user)
   │      └── mcp_oauth_sessions   token material, sealed at rest
   ├── mcp_tool_groups    named server/tool policy manifests
   └── mcp_gateway_settings   organization defaults

• Server — name, slug (unique per org), URL, transport and the OAuth scopes every proxied call requires. Deleting one cascades to its grants and sessions: withdrawing the server withdraws access to it.
• Grant — a user’s consent against a server, with the scope set they agreed to. A user holds at most one live grant per server (a partial unique index on revoked_at is null); revoked grants are kept so the audit trail survives. Re-consenting updates the scopes in place rather than accumulating rows.
• Session — the tokens issued under a grant, with expires_at, an optional refresh token and refresh_expires_at.

Token handling

Access and refresh tokens are sealed with AES-256-GCM under the deployment KEK (ROLTER_KEK), the same mechanism as upstream provider credentials — there is deliberately no plaintext column for either. Ciphertext and nonce sit side by side; the KEK never reaches the database.

The store exposes tokens through two credential-only paths. McpOAuthRepo::open_session opens one explicitly selected live session for lifecycle code. The Postgres config store opens only the newest live session per (server, user) whose scopes remain within its live grant and cover the server’s required scopes; those records travel through the token-guarded /internal/snapshot channel already used for decrypted provider credentials. Public API DTOs carry only metadata and a has_refresh_token boolean.

The gateway indexes servers by (org, slug) and sessions by (server, user). A request to /mcp/{server} must authenticate with a database-backed virtual key whose created_by user owns the selected session. The gateway repeats the required-scope check before connecting and replaces the caller’s virtual key with the downstream bearer token. Revocation, expiry and policy writes bump config_version, so snapshot polling removes authorization without a restart.

Who sees what

Every listing is joined through mcp_servers.org_id, so a cross-tenant read is not expressible, not merely filtered out.

Endpoints

• GET/POST /api/v1/orgs/{org_id}/mcp-servers, PATCH/DELETE /api/v1/mcp-servers/{id} — viewer reads, admin writes. A server URL must be http(s); the transport must be one of stdio, sse, streamable_http, websocket. PATCH updates registry metadata, enabled state, declared tools and required scopes without destroying grants or sessions.
• GET /api/v1/orgs/{org_id}/mcp/library — curated definitions annotated with whether the slug is installed. Installing one uses the ordinary server create endpoint, so the registry remains the source of truth.
• GET/POST /api/v1/orgs/{org_id}/mcp/tool-groups, PUT/DELETE /api/v1/mcp/tool-groups/{id} — exact server/tool policy manifests. These definitions are not yet enforced by the proxy.
• GET/PUT /api/v1/orgs/{org_id}/mcp/settings — organization transport and request defaults. The current HTTP proxy still uses deployment-level transport timeouts.
• GET /api/v1/orgs/{org_id}/mcp/grants, DELETE /api/v1/mcp/grants/{id}
• GET /api/v1/orgs/{org_id}/mcp/sessions, DELETE /api/v1/mcp/sessions/{id}
• GET/PUT /api/v1/mcp-servers/{id}/oauth-client — the OAuth client rolter presents to the server’s authorization server. Admin-only in both directions: the row names a third party the tenant has chosen to trust. The client secret is sealed with the deployment KEK on write and never read back; PUT with an empty secret downgrades a confidential client to a public one.
• POST /api/v1/mcp-servers/{id}/oauth/authorize — begin consent. Returns the authorization URL rather than a 302, because the caller is the dashboard over fetch and cannot usefully follow a cross-origin redirect.
• GET /auth/mcp/callback — where the browser returns. Authenticated by the one-shot login state, not by a session bearer token.
• POST /api/v1/mcp/sessions/{id}/refresh — renew a session from its stored refresh token.
• POST /api/v1/mcp/sessions/{id}/exchange — RFC 8693 token exchange for a narrower, downstream session.
• GET/POST/DELETE /mcp/{server_slug}/{path...} — Streamable HTTP/SSE proxy on the gateway, authorized by virtual-key owner, server and required scopes

Revoking a grant revokes every session under it in the same transaction, so consent and tokens can never disagree. Server creation/deletion and both revocations are written to audit_log.

The session lifecycle

Consent runs as an ordinary authorization-code flow with PKCE. POST .../oauth/authorize mints a verifier, seals it into a one-shot mcp_oauth_login_states row keyed by state, and returns the authorization URL. The callback consumes that row — a replayed state finds nothing and fails — verifies the code against the sealed verifier, then writes the grant and its first session in one transaction.

A background refresher sweeps every 60 seconds and renews up to 100 sessions per pass, 5 minutes before expiry, so the skew between rolter’s clock and the authorization server’s plus one round trip is always covered. It handles refresh-token rotation by replacing the stored refresh material whenever the response carries a new one. A permanently refused refresh revokes the session rather than retrying: a 4xx carrying invalid_grant or invalid_scope is a final answer about this grant — consent was withdrawn or the token was rotated away — and a retry loop would only hammer the upstream. Transient failures (network errors, 5xx) leave the session alone for the next sweep.

Token exchange (urn:ietf:params:oauth:grant-type:token-exchange) is the server-to-server half. A service acting for a user gets its own session row descending from the same grant, so it can be revoked independently without taking the user’s interactive session with it, and its scopes are intersected against the grant’s — an exchange can never widen consent.

SCIM 2.0 provisioning

An identity provider can create, update, deactivate and reconcile rolter accounts and groups over SCIM 2.0 instead of an operator doing it by hand. Users answer who exists; groups, through an operator-written mapping, answer what they may do.

Tokens carry the tenant

There is no org id anywhere in the SCIM path. An operator mints an org-scoped provisioning token, and that token is what resolves the tenant:

POST   /api/v1/orgs/{org_id}/scim-tokens   { "name": "okta" }   # org admin
GET    /api/v1/orgs/{org_id}/scim-tokens
DELETE /api/v1/scim-tokens/{id}

The plaintext token (rolter_scim_…) is returned once, at creation. Only its peppered SHA-256 digest is stored — the same treatment as session tokens — so a database dump alone cannot be replayed, and a lost token is rotated rather than looked up. Revoking takes effect on the very next request; there is no cache to invalidate.

Because the token resolves to exactly one org, an IdP cannot address another tenant’s users: a foreign resource id is a 404, not a redacted row.

Users

GET    /scim/v2/Users?filter=userName eq "ada@example.com"
POST   /scim/v2/Users
GET    /scim/v2/Users/{id}
PUT    /scim/v2/Users/{id}
PATCH  /scim/v2/Users/{id}
DELETE /scim/v2/Users/{id}

• Filters. Only userName eq "value" is supported — the one shape reconciliation needs. Any other filter is a 400 with scimType: invalidFilter rather than being ignored: silently returning the whole directory reads to an IdP as “no such user”, and it then re-creates the account.
• Idempotence. IdPs retry. Creating an existing userName returns 409 with scimType: uniqueness instead of a second account. If the email already belongs to a local account (invited by an admin, or provisioned elsewhere), that account is adopted rather than failing on the unique email, so provisioning converges.
• PATCH. The active toggle is implemented, in both the path: "active" and bare {"active": false} forms, and with the string "true"/"false" some IdPs send. Any other operation is a 400 — an IdP that gets a success for an operation nothing applied would believe the change landed.
• Deactivation logs the user out. Setting active: false stamps deactivated_at and deletes the account’s live sessions, because an IdP disabling a leaver expects them out now, not merely unable to log in again.
• DELETE deprovisions. The account is deactivated, its sessions dropped, and the org’s SCIM identity mapping removed. The users row itself stays: its memberships and audit trail must outlive any one IdP, and a later POST adopts it again.

No password path

A password attribute in a SCIM body is ignored, not honoured. Provisioned accounts are SSO-shaped: password_hash stays null, and there is no code path through which SCIM can set or read a local credential.

Groups

GET    /scim/v2/Groups?filter=displayName eq "platform"
POST   /scim/v2/Groups
GET    /scim/v2/Groups/{id}
PUT    /scim/v2/Groups/{id}
PATCH  /scim/v2/Groups/{id}
DELETE /scim/v2/Groups/{id}

Same token, same tenancy rule, same ScimError envelope as Users.

• Filters. Only displayName eq "value" is supported, for the same reason userName eq is the only Users filter: answering an unsupported filter with the whole directory reads to an IdP as “no such group”, and it then creates a duplicate.
• Members must already be provisioned here. A members entry has to be a user with a SCIM identity in the token’s org. Anything else — an unparsable id, a local account the IdP never created, another tenant’s user — is a 400 with scimType: invalidValue. A group can therefore never be a side door into an account the Users surface did not make.
• PATCH. The operations IdPs actually send are implemented: add/replace on members (array of {"value": id}, array of bare ids, or a single entry), remove on members both with a value list and in the filtered members[value eq "…"] path form Okta uses, remove on the whole attribute to empty the group, replace on displayName/externalId, and the pathless {"op": "replace", "value": {…}} shape. Anything else is a 400.
• Idempotence. A replayed create is a 409 with scimType: uniqueness. A replayed add of an existing member, or a PUT re-sending the same member list, changes nothing.
• DELETE. The group row goes and its members are reconciled in the same request, so the roles it granted disappear immediately rather than at the next sync. A second DELETE is a 404.

Group→team mapping

A mapping is written by an operator, not by the IdP — the IdP may not decide what its groups are worth:

POST   /api/v1/orgs/{org_id}/scim-group-mappings   # org admin
       { "group_name": "platform", "role": "member", "team_id": "…" }
GET    /api/v1/orgs/{org_id}/scim-group-mappings
DELETE /api/v1/scim-group-mappings/{id}

This is deliberately the SSO group-mapping model, down to the vocabulary: a group_name grants one of admin/member/viewer at the most specific non-null scope id (team_id, project_id, or the org itself when neither is named), and a mapping may only grant inside its own org — a team or project belonging to another tenant is a 400. Two divergent group-mapping models in one control plane would be two things for an operator to learn and two places for a privilege bug to hide.

Mappings key on the group’s displayName, which is what an operator sees in the IdP UI. Renaming a group in the IdP therefore detaches it from a mapping written against the old name; rewrite the mapping, or keep the SCIM display name stable.

A mapping may be written before the IdP has ever mentioned the group. Creating or deleting one reconciles that group’s current members straight away, so a mapping change does not wait for the next sync.

Reconciliation converges

Every group write recomputes the affected users’ memberships from the database, never from the request body. The wanted set is “every mapping of every group this user is currently in”; source = 'scim' rows that are not in it are deleted, missing ones are created. Three consequences:

• A scheduled sync re-sending the same group state is a no-op.
• Dropping a user from a group revokes exactly what that group granted, in the same request.
• A grant an operator made by hand carries source = 'manual' and is never touched — the same rule SSO logins follow, so the enrolment paths can be used side by side. An equivalent manual grant also suppresses creating a duplicate scim row.

DELETE /scim/v2/Users/{id} drops the account from every group in the org before removing the identity, so a deprovisioned account cannot keep a group-granted role.

Roles

A provisioned account is granted a viewer membership at the org it was provisioned into, merely for existing — least privilege on purpose. Anything beyond a read comes from a group mapping an operator wrote.

Audit

scim.user.create, scim.user.update, scim.user.deprovision, scim.group.create, scim.group.update and scim.group.delete are written to audit_log, scoped to the org, with the provisioning token’s id in the detail — the actor is a token, not a human. Operator actions carry the acting operator instead: scim_token.create / scim_token.revoke, and scim_group_mapping.create / scim_group_mapping.delete.

Security

Secret handling

• Upstream provider keys are never stored in plaintext in the database. They are envelope-encrypted with AES-256-GCM: a per-record data key/nonce, wrapped by a master key (KEK) supplied via ROLTER_KEK (env/file). Pluggable backends (HashiCorp Vault, cloud KMS) are a roadmap item.
• In the bootstrap file, prefer api_key_env over inline api_key so secrets stay in the environment, not on disk.
• Virtual keys are stored as hashes with a short display prefix; the raw key is shown once at creation.
• Secrets are never logged. The gateway redacts auth headers from traces.

Transport

• Upstream calls use rustls (no OpenSSL). HTTP/2 keep-alive with connection pooling.
• Optional per-provider egress proxy (egress_proxy, HTTP/HTTPS/SOCKS5) for networks where providers aren’t directly reachable.
• Optional global or per-provider custom CA bundles add private PKI roots to outbound upstream clients while retaining public roots, certificate-chain validation, and hostname verification.
• Terminate TLS at the gateway or a fronting proxy/ingress in production.

Cross-origin policy (CORS)

security_settings.allowed_origins and allowed_headers are enforced by a middleware on the control plane (crates/rolter-control/src/cors.rs). The policy is read from an ArcSwap per request, so an edit through PUT /api/v1/security-settings applies immediately rather than at the next restart.

Only the control plane has one, because it is the only origin a browser talks to: it serves the dashboard, and the Playground reaches the data plane through the /gw/* reverse proxy rather than calling the gateway directly. A gateway CORS layer would govern requests browsers do not make, and would need SecuritySettings in the snapshot to do it.

Behaviour:

• No origins configured is the default, and the same-origin deployment everyone runs. The middleware adds no headers at all and the request path is unchanged — CORS is not needed when the control plane serves both the dashboard and the API.
• Matching is exact against the configured list, modulo case and a trailing slash. There is no wildcard and no suffix match, so evil-dash.example.com and dash.example.com.evil.test do not inherit an entry for dash.example.com. validate_origin already rejects * at write time.
• A denied origin gets a normal response with no CORS headers; the browser enforces the block. Vary: Origin is set on every answer so a shared cache cannot serve one origin’s response to another.
• Preflights are answered by the middleware with 204 and never reach the router, since they carry no credentials.
• authorization, content-type, traceparent, tracestate and x-request-id are always allowed, unioned with whatever is configured. The trace headers are what let the dashboard’s spans join the gateway’s on a split-origin deployment; leaving them to configuration means browser trace propagation silently does not work until someone remembers them.
• x-request-id is exposed via Access-Control-Expose-Headers, so a browser can read the correlation id the API already echoes.

Egress policy (SSRF)

A provider’s api_base decides where the gateway sends traffic, so an admin surface that accepts an arbitrary one turns the proxy into an SSRF primitive aimed at whatever the gateway’s network position can reach.

rolter is built to run self-hosted and air-gapped, so upstreams legitimately live on loopback, RFC1918 and container networks — blanket-denying private destinations would break the core use case. What no legitimate LLM upstream needs is the link-local range, which is where cloud instance metadata lives (169.254.169.254, fe80::/10). That is denied by default; everything else is opt-in:

[egress]
block_link_local = true   # default — cloud instance metadata
block_loopback   = false  # sidecar / single-host deployments
block_private    = false  # on-prem clusters
allow_hosts      = []     # exact-host escape hatch

Enforced in two places: the control plane rejects a denied api_base at write time (400, so it never reaches the database), and snapshot validation re-checks it, so a bootstrap toml can’t smuggle one in either.

Config-time validation classifies IP literals only. Resolving hostnames there would make validation depend on live DNS — reintroducing the “one bad row freezes every gateway” failure mode described in config-and-hot-reload.md — and would be bypassable by DNS rebinding regardless.

So the same policy is enforced a third time, at connect time, by a custom resolver on every upstream client: whatever DNS actually returns is classified immediately before the connection is made. That covers the two cases config validation cannot see — a hostname like metadata.internal, and a name that resolved to a public address when it was configured but resolves to 169.254.169.254 at request time. A denied destination surfaces as a policy error naming the host, not an opaque connect failure.

A name that resolves to several addresses keeps the permitted ones: refusing the whole name would take down a legitimate multi-homed upstream. Only a name left with nothing is refused — which is exactly what rebinding to a denied address produces. The resolver reads the policy from a live handle, so a hot reload re-tunes enforcement without discarding pooled connections.

Control-plane input validation

Every control-plane mutation body is decoded through a SafeJson extractor rather than axum’s Json. Before the body is deserialized into its typed struct, every string in it — nested objects, arrays and object keys included — is screened for control characters, and the request is rejected with a 400 naming the offending field.

The concrete failure this prevents: Postgres text columns cannot store a NUL byte, so a field carrying one failed deep inside the store and surfaced as an unhandled 500 instead of input validation, violating the “bounded error, no unwrap/expect on a request path” invariant. Screening the whole C0/C1 range (and U+007F) also closes the log-injection vector a raw escape or newline would otherwise open in operator-facing logs.

Tab, newline and carriage return stay allowed — multi-line values are legitimate, a PEM CA bundle being the obvious one. Malformed JSON now also comes back in the same OpenAI-style error envelope as every other failure instead of axum’s default rejection body.

Open mode (no admin token)

With no ROLTER_ADMIN_TOKEN set, Principal short-circuits to Superadmin for every request: the management API and /internal/snapshot have no authentication step to fail. This is the zero-credential local-dev shape, and crates/rolter-control/src/open_mode.rs is what keeps it from being anything else. Before either listener is opened, it evaluates “is a token set” against every address about to be bound:

--internal-addr counts as a listener here: an exposed credential channel is no better than an exposed API, so either one alone is enough to refuse.

The decision also rides into the dashboard through window.__ROLTER_CONFIG__ (openMode: true), which renders a persistent banner. Without it the dashboard looks identical whether the control plane is gated or wide open, which is the property that made this dangerous rather than merely permissive (#970).

This is why ROLTER_CONTROL_HOST defaults to 127.0.0.1 rather than 0.0.0.0: containers and clusters set it explicitly, and by then they have a reason to have set a token too.

Control↔data-plane trust boundary

GET /internal/snapshot returns provider api_keys decrypted. That is by necessity — the data plane needs the upstream credential to authenticate to the provider — but it makes the snapshot channel a different trust boundary from the operator-facing management API, and it should be configured as one:

ROLTER_INTERNAL_TOKEN=...            # gates /internal/*, distinct from ROLTER_ADMIN_TOKEN
ROLTER_INTERNAL_ADDR=127.0.0.1:4002  # serves /internal/* on its own socket

With ROLTER_INTERNAL_TOKEN set, the operator admin token no longer opens the snapshot — only the gateway’s own credential does. With ROLTER_INTERNAL_ADDR set, /internal/* is not mounted on the public API router at all, so it is absent from the port the dashboard and management API are served from rather than merely gated on it. Bind it to loopback or a private interface. The gateway sends ROLTER_INTERNAL_TOKEN on snapshot polls, falling back to ROLTER_ADMIN_TOKEN.

Both are optional. Unset, the historical behavior holds — /internal/* shares the public listener and accepts the admin token — and the control plane logs a warning at startup saying so. The tenant-facing CRUD surface never returns a provider key in any configuration.

Two things this deliberately does not do. There is no mTLS between the planes: a shared secret over a private interface is comparable strength when the network path is already trusted, and mTLS adds certificate lifecycle to an air-gapped deployment. And the snapshot still carries plaintext rather than sealed ciphertext: envelope-passing would require the gateway to hold the KEK, moving the master secret onto every data-plane node, which is a worse trade for most deployments. Revisit both if the planes ever cross an untrusted network.

Wire transparency

• Outbound requests to upstream providers carry no rolter-identifying marks: no User-Agent, no added X-*/Via headers, no metadata injected into the JSON body, no marks in SSE framing. The only headers sent are functionally required ones — content-type, the provider’s auth header, and anthropic-version for Anthropic.
• Responses back to clients likewise gain no rolter-added headers.
• This is a tested guarantee: golden wire tests in rolter-proxy capture the raw outbound request head and fail on any unexpected header (see openai_wire_carries_no_rolter_signature).

External PII sanitization (#848)

The gateway can hand request content — and optionally response content — to a self-hosted de-identification service before it reaches a provider. Configured under [pii_sanitizer]; see user-docs/security/pii-sanitizer.mdx for the operator-facing contract.

The design property worth stating precisely: the placeholder→plaintext mapping never enters the gateway process. The sanitizer substitutes deterministic placeholders and returns an opaque restoration token; rolter holds the token and nothing else. There is therefore no log line, metric label, span attribute or cached body from which the original values can be recovered, and no in-memory table for a crash dump to expose.

Consequences that fall out of that choice:

• RestorationTicket is not printable. Its Display renders <restoration token redacted>, and the token is reachable only through token_for(&scope), which returns None unless the caller’s org/team/project/route match the scope the ticket was minted under. A token from one project cannot restore content in another.
• Restoration is opt-out, not opt-in-by-default. RestorationPolicy::Never is the default; CallerAuthorized honours x-rolter-pii-restore only because the policy allows it, never because the caller asked.
• The response leg never requests reversibility. Restoring provider-generated content would return the very data that leg exists to remove.
• A streamed response with an active response leg is refused (pii_streaming_unsupported, HTTP 400), mirroring guardrails.streaming_post_call. Correct streaming restoration would need either a network round trip per chunk — destroying TTFT — or the mapping held in-process, destroying the property above. streaming = "passthrough" waives the response leg instead.
• A malformed sanitizer reply is a failure, unlike the guardrail webhook’s decision parsing which defaults to allow. Defaulting would forward content the gateway believes is sanitized and is not.
• Restore runs after the cache store, so what is cached is what the upstream said. A later hit on the same entry is re-evaluated against its own request’s policy and ticket rather than replaying someone else’s restored plaintext.
• fail_open is the default and is the risk to monitor. It forwards unsanitized content when the service is down. rolter_pii_sanitizer_errors_total rising under fail_open means personal data is reaching providers; deployments where the sanitizer is a compliance control should set fail_closed.

A response-leg failure never fails the request even under fail_closed: the upstream call already happened and was already billed, so refusing to deliver would spend the caller’s money and return nothing. The counter records it.

Threat model (high level)

• Tenant isolation: virtual keys are scoped to a project; model allow-lists prevent access to unconfigured models; cache keys are namespaced to avoid cross-tenant cache poisoning.
• Abuse: RPM/TPM rate limits and budgets bound spend and load (roadmap enforcement).
• AuthZ: control-plane mutations are RBAC-checked and recorded in audit_log.
• Supply chain: cargo deny/advisory scanning in CI is a roadmap item.

Operational guidance

• Always set a strong ROLTER_KEK (e.g. openssl rand -hex 32) and rotate provider keys periodically.
• Run the control plane on a private network; expose only the gateway publicly.
• Back up Postgres; treat the master key as the most sensitive secret.

LDAP authentication

The third identity provider behind rolter_auth::IdentityProvider, after local password login and OIDC SSO (#241). It is compiled in behind the ldap cargo feature and is off unless configured.

How a login is verified

A user’s distinguished name is rarely their login name, so verification is a two-phase bind:

1. Find. Bind as a read-only service account and search base_dn for the entry matching user_filter, with {login} substituted.
2. Verify. Bind a second time as the entry that was found, using the password the user submitted. That bind is the authentication.

rolter never reads, compares or stores a password hash from the directory. It also never caches the password.

A filter matching more than one entry is a hard failure rather than a login: binding as an arbitrary one of several matches would authenticate the wrong person.

Group mapping is least-privilege

group_role_map maps a directory group onto a rolter role. It is explicit and additive:

• a user in several mapped groups gets the union of their roles;
• a user in no mapped group is rejected, even with a correct password.

There is no implicit default role and no wildcard. Authenticating proves who someone is, not what they may do — the same rule SSO already follows.

Group names are matched case-insensitively and with surrounding whitespace trimmed, because directories are inconsistent about DN case and an operator who copies a group from their directory’s UI should not silently get no access.

What a client is told

Every authentication failure returns the same generic rejection:

The first four are deliberately indistinguishable. A login form that distinguishes “no such user” from “wrong password” is a user-enumeration oracle against the corporate directory, which is a worse leak than the login itself.

A directory being down is distinguishable, because that is an operational fact the operator needs and reveals nothing about any account. Distinguished names, filters, bind credentials and raw LDAP result codes are logged for the operator and never returned.

Configuration

url = "ldaps://dir.example.com"          # ldaps:// strongly preferred
bind_dn = "cn=svc-rolter,dc=example,dc=com"
bind_password = "…"                       # read-only service account
base_dn = "ou=people,dc=example,dc=com"
user_filter = "(uid={login})"             # must contain {login}
email_attr = "mail"
display_name_attr = "cn"
group_attr = "memberOf"
require_tls = true

[group_role_map]
"cn=rolter-admins,ou=groups,dc=example,dc=com" = "admin"
"cn=rolter-users,ou=groups,dc=example,dc=com" = "member"

Configuration is validated before it is used to authenticate anyone. These are rejected at startup rather than at first login:

• a URL that is neither ldap:// nor ldaps://;
• require_tls with a plain ldap:// URL — the service-account bind password would cross the network in cleartext;
• a user_filter without {login}, which would authenticate whoever the filter happens to match, for any login submitted;
• an empty base_dn;
• an empty group_role_map, under which no directory user could ever be granted a role.

Filter injection

The login is escaped per RFC 4515 before substitution. Without it a login of * matches every user in the subtree, and )(uid=admin rewrites the filter — LDAP injection, the directory analogue of SQL injection. \, *, (, ), / and NUL are escaped.

Operational limitations

• Group membership is read from the user entry via group_attr (typically memberOf). Directories that do not populate a reverse-membership attribute need a nested group search, which is not implemented.
• Nested groups are not expanded. A user in a group that is itself a member of a mapped group is not granted that role. Map the groups users are directly in.
• No connection pooling. Each login opens and closes its connections. This is correct but not fast; an interactive login is not a hot path, and pooling bound connections has its own correctness hazards.
• No referral chasing. A referral from the directory is not followed.
• ldaps:// only for transport security. StartTLS on a plain ldap:// connection is not implemented.

Health and readiness

Both rolter binaries serve two operational endpoints that answer two different questions. They are unauthenticated, cheap, and safe to probe on every pod.

The words mean the same thing on rolter-gateway and rolter-control.

Why the split exists

Collapsing both probes onto one endpoint has no correct answer, only a choice of which failure to accept:

• If the single endpoint answers as soon as Axum binds, Kubernetes marks the pod Ready before it can serve. A rolling update then routes dashboard and /internal/snapshot traffic — which every gateway in the fleet polls — at a control pod whose migrations have not run and whose pool has no connection.
• If the single endpoint instead checks Postgres, a database blip fails the liveness probe and Kubernetes kills every control pod at once. A recoverable dependency outage becomes a restart storm, and the restarts make the outage worse by reconnecting a stampede at the recovering database.

Splitting the endpoints lets a dependency failure drain traffic without touching the process.

/healthz — liveness

Returns 200 OK with the body ok whenever the HTTP server is accepting connections. It checks no dependency: not Postgres, not Redis, not ClickHouse, not an upstream provider. That is the point — nothing outside the process may cause a restart.

/readyz — readiness

Gateway

Returns 200 OK normally, and 503 Service Unavailable with the body draining once the instance has been drained from the dashboard’s cluster screen — the state arrives on the /internal/snapshot response the gateway already polls. A drained gateway therefore leaves the Service and finishes its in-flight requests, instead of being killed with them in progress.

Control plane

Returns a JSON body describing each check:

{
  "status": "ready",
  "checks": {"database": "ok", "migrations": "ok", "kek": "ok"}
}

200 when status is ready, 503 otherwise. The checks are:

• database — the Postgres pool hands out a connection within 2 seconds. The timeout is deliberately shorter than a typical probe timeoutSeconds so an exhausted pool answers a clean 503 rather than hanging until kubelet reports a timeout, which looks like a dead process.
• migrations — every migration embedded in this binary is applied to the database. A pod running ahead of its schema would serve wrong answers on the CRUD API and /internal/snapshot, so it is not ready.
• kek — ROLTER_KEK parses when it is set. A KEK set to an empty value seals nothing and decrypts nothing, so the pod cannot serve credentials. Reported as not configured (and not a failure) when the variable is unset.

Redis and ClickHouse are deliberately not checked. The control plane serves configuration without either — losing Redis costs the config-bump fan-out notification, losing ClickHouse costs the analytics screens — so their absence is degraded, not unready. Removing every control pod from the Service because analytics is down would be a strictly worse outage.

Running without --database-url (bootstrap config only), there is nothing to wait on and the answer is always ready.

Probe wiring

charts/rolter wires this for you:

readinessProbe:
  httpGet: {path: /readyz, port: http}
livenessProbe:
  httpGet: {path: /healthz, port: http}
startupProbe:               # control plane only
  httpGet: {path: /healthz, port: http}

The startupProbe on the control plane holds liveness off while first-boot migrations run against a cold database, so a slow migration is not mistaken for a wedged process. Tune the periods under control.probes in values.yaml.

What to alert on

• /readyz down for longer than a failover — the control plane cannot reach its database, or is running against a schema it has not migrated. Page.
• /healthz flapping — the process itself is crashing or wedging. Page.
• /readyz down on a single gateway — usually just a drained instance, which is expected during a rolling update. Do not page on this alone.

Observability

Metrics

• The gateway exposes Prometheus metrics at GET /metrics: counters (rolter_requests_total, rolter_upstream_errors_total, rolter_auth_failures_total, reload/log/budget/rate-limit/retry/cooldown/health/breaker/scrape counters), gauges (rolter_config_version, rolter_breaker_entries), and per-model latency histograms — rolter_request_latency_ms (total) and rolter_request_ttft_ms (time-to-first-token), each labelled {model=...} with the standard _bucket/_sum/_count series. Histograms are observed once per completed request from the log sink, off the response hot path.
• The exporter is hand-rolled (atomic counters + non-cumulative histogram buckets cumulated at render) rather than the metrics facade + global recorder, which does not fit the lock-free arc-swap design where an explicit Arc<Metrics> is threaded through the request path.
• Passive per-target SLA signal: rolter_target_requests_total{provider,target,outcome} (a counter, outcome = ok for 2xx else error) is tallied once per completed request from the log sink — free, derived from real traffic, no extra upstream calls. A per-target error rate / uptime is sum(rate(rolter_target_requests_total{outcome="error"}[5m])) / sum(rate(rolter_target_requests_total[5m])). This is the first slice of provider stability tracking (ROL-123); the ClickHouse provider_health_events table and the dashboard land in later slices. The active prober is guarded: bounded probe concurrency with per-provider jitter, consecutive-failure/-recovery thresholds gating the unhealthy flip (no single-probe flapping), and exponential probe backoff when a probe itself gets a 429.
• Multi-key providers: rolter_key_cooldowns_tripped_total counts api keys parked after a key-level failure (429/401 on a provider with several keys); the request retries in-flight on a sibling key.
• A/B attribution: rolter_variant_requests_total{model,variant} (a counter) tallies requests per chosen variant, so traffic splits are visible in Prometheus/Grafana without querying ClickHouse. Classic single-pool routes (no variant) emit nothing. Observed from the same log-sink funnel (ROL-195, part of ROL-188).
• Adaptive routing (#544): rolter_adaptive_routing_decisions_total{model,mode} splits a route’s picks between blend (the latency/cost/load blend), exploration (the bounded random share that keeps starved targets sampled) and fallback (the deterministic pipeline stack served while the kill switch is off or the evidence is too thin), and rolter_adaptive_routing_engaged{model} is 1 while the blend is actually routing. Only routes on the adaptive strategy emit these. A config reload rebuilds the balancer and so resets the counters, which lines up with a rolter_config_version bump — alert on rate(), not the absolute value. A route sitting at engaged 0 with all picks on fallback is the expected steady state before an operator enables the policy. rolter_adaptive_routing_target_score{model,target} (#751) adds the blended score each target currently carries — the same numbers the control plane serves at GET /api/v1/adaptive-routing-telemetry, computed at scrape time rather than on the request path.
• Roadmap: add per-provider/route labels on the histograms, in-flight gauges, cache-hit ratio, and circuit-breaker state gauges.
• Roadmap: scrape/federate upstream engine metrics from vLLM/SGLang/TGI /metrics and correlate them per target (queue depth, KV-cache usage, running/waiting requests) to feed load- and cache-aware routing and the dashboard.

Tracing & context propagation

• tracing + tracing-subscriber with RUST_LOG filtering; TraceLayer logs each HTTP request.
• Inbound: accept W3C traceparent/tracestate (and b3) from clients and continue the trace; honor x-request-id / x-correlation-id.
• Outbound to engines: inject the active trace context into upstream requests so vLLM/SGLang/TGI spans join the same distributed trace. vLLM and SGLang support OpenTelemetry tracing (e.g. vLLM --otlp-traces-endpoint); point them at the same OTLP collector so engine prefill/decode spans line up with rolter’s request span.
• A per-request request_id is echoed in a response header and stamped on logs, metric exemplars and spans for correlation.
• Events: rolter calls no span-event API directly, but tracing-opentelemetry turns every tracing event fired inside an active span into an OTel span event. That API is deprecated upstream in favour of log-based events; the target model, and the sequencing it forces, are recorded in ADR-0025.

How the context actually moves

rolter-core::telemetry owns both directions, and both are inert unless an OTLP endpoint is configured:

• Extract. GatewayMakeSpan (rolter-gateway::trace) is the TraceLayer’s span-maker: it builds the request span and makes the extracted inbound context its parent. A B3-only caller is normalized into an equivalent traceparent first, so one W3C propagator serves both wire formats. Without this the gateway’s spans were disconnected roots — the trace id reached the request log, but nothing joined the caller’s trace.

  It has to be the span-maker rather than a middleware layered inside the TraceLayer: DefaultMakeSpan builds the request span at DEBUG, so under the default RUST_LOG=info it is disabled, and setting a parent on a disabled span silently does nothing. With no pipeline installed it falls back to that stock DEBUG span, so the untraced path costs what it always did.

• Inject. The context handed to the provider is injected from the current span, inside the per-attempt upstream.request span, rather than copied from the caller. Copying it verbatim made the provider call a child of the caller’s span and therefore a sibling of the gateway’s own work, which silently invalidated every waterfall built from the data. The allowlisted client headers from Forwarder::forwarded_header_names are unaffected; only the trace headers changed hands.

• Log correlation. RequestLog.trace_id is read off the span context when a pipeline is installed and falls back to parsing the inbound header otherwise, so ClickHouse and the trace backend agree by construction.

Pipeline spans

One span per stage, so a slow request is attributable rather than merely slow:

queue.wait spans enqueue→dequeue only: the span travels with the queued job and the worker closes it the moment it picks the job up, so it measures the wait and not the wait plus the upstream call. The job carries the caller’s span alongside it, and the worker instruments the forward with it — the queue worker runs on its own task where nothing is in scope, so without that the forwarder’s own translate.request span becomes an orphan root in a trace of its own. Names follow the OTel GenAI semantic conventions where they fit, so a backend’s built-in GenAI views work.

Spans never carry prompt or completion content, API keys, virtual-key plaintext, or injected header values — those are credential material, and redaction stays owned by the existing logging_settings machinery rather than a second policy.

Control-plane spans

The control plane runs the same pipelines as the gateway (telemetry::init()), but until #845 emitted no spans of its own — everything it did was invisible beyond what the HTTP layer produced by default.

http.route is the matched template (/api/v1/providers/{id}), never the concrete path. control.request comes from one middleware rather than an attribute on each of ~90 handlers, so a route added tomorrow is instrumented the moment it is mounted.

snapshot.build is the one to watch. Snapshot latency is fleet-wide config-propagation delay: every gateway waits on it, so when an operator changes a route and the fleet serves stale config, this span is what says whether the delay is in generation or downstream of it. config_version on the span answers “which config was this” without putting an unbounded value on a metric.

Tenant attributes

The gateway.request span carries rolter.org.id, rolter.team.id and rolter.project.id, recorded once the virtual key resolves. The span itself is built by the tower layer, which runs before auth, so the fields start empty and are filled in rather than passed at construction.

These exist so per-tenant telemetry destinations are routable. ADR-0026 decided that fan-out to tenant-owned backends belongs in an OpenTelemetry Collector rather than in-process exporters — one egress path in the gateway no matter how many tenants, and no data-plane process POSTing to operator-supplied URLs — and that rolter’s job in that design is to stamp the attribute the collector routes on. Until this landed there was nothing to stamp: the request logs in ClickHouse always carried tenant identity, but no exported span ever did, which made the routing half unimplementable.

An unattributed request — a config-defined key, which has no org — records nothing rather than an empty string. An attribute that is present-but-blank on some spans and absent on others is harder to write a routing rule against than one that is consistently absent.

The names are deliberately rolter-local. The GenAI and HTTP conventions define nothing for tenancy, and a convention-shaped guess like tenant.id would be worse than an obviously-local name if the spec later defines it differently.

Running it locally

The observability overlay starts a collector and a trace UI alongside the normal stack:

docker compose -f docker/docker-compose.yml \
               -f docker/docker-compose.observability.yml up

Traces land at http://localhost:16686; the collector takes OTLP on 4317/4318 and re-exposes collected metrics on 8889. The overlay sets OTEL_EXPORTER_OTLP_ENDPOINT on the gateway and control services, so bringing it up is the only step — without it that variable is unset and tracing stays off.

The collector binds 0.0.0.0, not localhost: one bound to loopback inside its container is unreachable from the gateway container.

Choosing an overlay

There are two, and they are mutually exclusive — both publish OTLP on 4317/4318.

docker compose -f docker/docker-compose.yml \
               -f docker/docker-compose.signoz.yml up      # SigNoz on :8080

Jaeger is the default because it is two containers and starts in seconds, and because reading a correctly-parented waterfall is what the tracing work needed. Reach for SigNoz when “which stage is slow across all requests” matters, which Jaeger cannot answer.

Nothing in the Rust differs between them: the gateway speaks vendor-neutral OTLP and only the destination changes. Any other OTLP backend works the same way — repoint the exporter in infra/otel/collector.compose.yaml.

Querying SigNoz from an agent (MCP)

The SigNoz overlay also starts SigNoz’s MCP server on http://localhost:8000/mcp, so an agent can query traces, run ClickHouse queries, and manage dashboards against the local instance. Point an MCP client at that URL; the SigNoz agent-skills plugin ships a signoz server entry to fill in.

It needs a SigNoz API key, which is created in the UI (Settings → API Keys, admin only) and supplied to the server, not the client:

set -x SIGNOZ_API_KEY (pass show rolter/signoz-api-key)
docker compose -f docker/docker-compose.yml \
               -f docker/docker-compose.signoz.yml up -d signoz-mcp

The key is read from the environment and never written to a tracked file. With no key set the container still starts, but every call returns Authorization or SIGNOZ-API-KEY header required.

The MCP dashboard tools need SigNoz v0.135.0 or newer, which is why the signoz image is pinned ahead of the collector/ClickHouse pair.

The SigNoz overlay is a pinned equivalent of what SigNoz’s Foundry CLI generates, since SigNoz deprecated its own compose manifests in v0.130.0. Two things to know before touching it: its four images are a tested set and must be bumped together (a newer migrator emits ClickHouse settings an older server rejects), and it vendors 3.6 KB of ClickHouse config — the cluster topology and the {shard}/{replica} macros that ReplicatedMergeTree needs — rather than SigNoz’s full 56 KB config.xml, which the stock image defaults cover.

Metrics over OTLP

The counters and gauges rolter-gateway::metrics computes are exported over OTLP as well as served on /metrics. The Prometheus endpoint is unchanged — this is a second exporter over the same numbers.

Both read one list, Metrics::scalars(), so they cannot drift: a counter added there reaches Prometheus and OTLP without a second edit. Counters export as counters and gauges as gauges, since exporting a counter as a gauge would break rate() on the backend.

The instruments are observable: nothing is pushed on the request path. The SDK invokes the callback on its own schedule (OTEL_METRIC_EXPORT_INTERVAL, default 60s) and reads the same atomics the Prometheus renderer does, so the hot path still only does fetch_add. With no OTLP endpoint configured no meter provider, exporter or callback is built at all.

Per-model histograms and label-bearing counters stay Prometheus-only for now; the scalar set is what OTLP carries.

Control-plane metrics

The control plane has no Prometheus registry to mirror, so these are the one place it measures itself (#845). They are real histograms — measurements taken as they happen — not observable instruments, because a duration cannot be reconstructed from a counter after the fact.

Boundaries are deliberately not the gateway’s. A gateway request is dominated by an upstream model call and is interesting out to tens of seconds; a snapshot build and a CRUD write are database work where “is this 2 ms or 40 ms” is the whole question, so reusing the gateway’s boundaries would put nearly every observation in the first bucket.

Cardinality is bounded by construction. http.route is the matched template, so a thousand providers are one series. Status is recorded as a class, not a code: twelve statuses across ninety routes would be over a thousand series to answer a question five buckets answer. config_version is unbounded — a new value on every config write — so it lives on the snapshot.build span and never on a metric.

rolter_snapshot_payload_bytes skips the 304 case rather than recording a zero: a not-modified poll transfers no body, and folding zeroes in would drag the size distribution down and misreport what the fleet actually moves.

Payload size matters on its own. It is what every gateway transfers on every poll, and it is the first thing to look at when propagation gets slower without generation getting slower. The snapshot bench in rolter-core shows why the encode is the stage to watch: at 1000 routes it costs ~2.8 ms against ~150 µs for sanitize and ~120 µs for validate.

The control plane also registers the same process and runtime metrics as the gateway — it degrades for the same reasons and previously answered none of those questions either.

Logs over OTLP

Logs export alongside traces and metrics, gated on the same environment (OTEL_EXPORTER_OTLP_ENDPOINT, or the logs-specific OTEL_EXPORTER_OTLP_LOGS_ENDPOINT). With neither set no exporter is built and logging is stdout-only, exactly as before.

It is additive, not a replacement: the stdout fmt layer is untouched, so docker logs shows what it always did and the collector additionally receives the same records.

Records carry trace_id and span_id, which is most of the value — a log joins the trace it came from instead of being a separate pile of text. tracing-opentelemetry publishes the OpenTelemetry context when a span is entered, and the logs SDK stamps it onto each record. That correlation is asserted by a unit test rather than assumed from a dependency default, since it would fail silently if the default ever changed.

Redaction is unaffected: this exports the same tracing events the stdout layer renders, so anything already kept out of logs stays out of them.

Process metrics

Alongside the domain counters, the process reports its own vitals: resident and virtual memory, CPU time, open file descriptors, thread count and uptime. These are the numbers an operator reaches for first when a node degrades, and the gateway previously answered none of them.

They are observable instruments read from /proc, so nothing touches the request path. On a platform where /proc is unavailable the instruments are not registered at all, rather than registered and always zero — a metric that reads zero forever is worse than an absent one, because a dashboard cannot tell it from a healthy process. Names follow the OTel process.* conventions.

Runtime metrics

The process metrics describe the machine; these describe the scheduler. tokio.runtime.queue.depth is the one that matters: when latency rises, no other signal separates “the provider is slow” from “the request sat in our own run queue before we ever called the provider”, and the two have opposite fixes. It pairs directly with the queue.wait span. Alongside it are tokio.runtime.workers, tokio.runtime.tasks.alive and tokio.runtime.worker.busy.time.

Busy time is exported, not a busy ratio: a ratio computed in-process would average over whatever interval the SDK happens to use and would not re-aggregate across instances. As a monotonic counter the backend derives utilisation with rate(tokio.runtime.worker.busy.time) / tokio.runtime.workers, which does.

None of this requires --cfg tokio_unstable. #834 assumed it did, and that is true only of part of tokio’s surface: num_workers, num_alive_tasks and global_queue_depth are stable, and worker_total_busy_duration sits behind cfg_64bit_metrics!, which is #[cfg(target_has_atomic = "64")] — a property of the target, not an instability gate. What remains genuinely gated is the blocking pool and the per-worker steal/poll counters, which are therefore not exported; the workspace-wide flag decision stays unmade rather than being smuggled in with a telemetry change.

Outside an async runtime the instruments are not registered, for the same reason the process metrics are absent without /proc.

Connection-pool metrics

rolter_upstream_connections_total and rolter_upstream_connect_errors_total count connections established to providers, and attempts that never got that far.

Pool exhaustion presents as latency with healthy providers — the failure mode the other metrics cannot explain. reqwest and hyper expose no pool introspection at all, so this is instrumented rather than read: a tower layer over the connector (ClientBuilder::connector_layer) sees every connection hyper builds because it had none to reuse. The signal is the ratio rate(rolter_upstream_connections_total) / rate(rolter_requests_total) — near zero when the pool is working, approaching one when every request is paying a fresh TCP and TLS handshake.

Idle-versus-active counts stay unavailable: that needs the connection object’s drop, and the connector layer’s response type is opaque outside reqwest.

Resource attributes

Every signal carries service.name, service.version (the crate version), and where configured service.instance.id and deployment.environment.name. OTEL_RESOURCE_ATTRIBUTES is honoured by the SDK for anything else.

service.instance.id deliberately uses the same precedence as the cluster watcher — ROLTER_NODE_ID, then HOSTNAME, then nothing — so a node in cluster_nodes and a node in the trace backend are the same node by construction. When neither is set it is omitted rather than invented per restart, which would churn the identity on every deploy.

Wrapping audit (#815)

OpenTelemetry’s Don’t wrap OpenTelemetry argues that a house abstraction over the instrumentation API costs performance, maintainability and developer education. Three anti-patterns are named: wrappers that force callers to allocate an attribute collection, wrappers that look an instrument up by name per measurement, and general API abstraction that teaches a proprietary interface instead of the standard.

rolter has four things sitting between its code and the OTel API. Each was audited against that post; the verdict is keep for all four, for the reasons below. The post is guidance rather than a mandate, and it asks for a refactor only where there is a measured cost or a real maintenance burden.

Note up front that rolter instruments through tracing + tracing-opentelemetry. That is the ecosystem bridge, not a bespoke house wrapper, and it is not what the post argues against. This audit is not a proposal to remove tracing.

stage_span! expands to a direct tracing::info_span! call guarded by an is_active() check, so it is closer to the code generation the post recommends than to a runtime wrapper. It takes tracing’s own compile-time field syntax and never asks a caller to build a Vec or a slice of attributes, so the force-allocation anti-pattern does not apply. The guard is the point of the macro: with no pipeline installed it yields Span::none(), which allocates nothing.

The scalar-metrics list is the shape most at risk, since install_metrics registers one observable instrument per scalar and each instrument’s callback calls collect() and finds its own entry by name. That is a by-name lookup, but it is not on a hot path: the instruments are observable, so the SDK invokes those callbacks on its own export interval (OTEL_METRIC_EXPORT_INTERVAL, default 60s). The request path only ever does fetch_add on a named AtomicU64 field — there is no map, no lookup and no lock between a request and its counter. The post’s performance argument therefore does not bite here even though the gateway hot path is where it would bite hardest.

What the export path does cost is one Vec<ScalarMetric> allocation per instrument per cycle and a linear scan of it, so the work is quadratic in the number of scalars. At the current eight scalars, once a minute, that is immaterial. It is left as-is deliberately: the OTel Rust 0.32 API offers only per-instrument with_callback, so collecting once per cycle for all instruments is not expressible, and matching by name rather than by index keeps the callbacks independent of the order scalars() happens to return.

RequestHistograms::record is the one place a wrapper does force an allocation on the request path — model.to_string(), to build the single KeyValue both histograms take. It is unavoidable rather than incidental: OTel’s Value::String holds an owned or 'static string and model names are neither. The obvious way to avoid it is to cache a prebuilt attribute set per model, which would put a sharded-lock map lookup on the request path — precisely the lookup-based anti-pattern the post names, and something AGENTS.md forbids on the data-plane hot path. One small allocation is the cheaper of the two, and it is paid only when an OTLP endpoint is configured.

GatewayMakeSpan is a tower_http::trace::MakeSpan implementation: layer configuration, which the post explicitly separates from instrumentation and calls not wrapping. Recorded here only so the audit is complete.

Turning telemetry off explicitly

ROLTER_TELEMETRY_ENABLED=false hard-disables every export — traces, metrics and the dashboard’s browser tracing — regardless of which OTEL_* endpoints are set (#812). Unset means enabled, which changes nothing for an existing deployment: with no endpoint configured nothing is exported anyway.

The switch can only subtract. It never turns export on by itself, and an unrecognized value leaves export on rather than silently blinding a deployment; only 0, false, no and off disable it.

It exists because “off” was previously implicit — achieved by leaving an endpoint unset — which does not survive somebody setting the endpoint for one signal and gives an operator nothing to point at in a security review. It is deliberately environment-only and has no config-file equivalent; see ADR-0026, which also records why per-tenant telemetry destinations belong in the collector rather than in rolter.

Cost when tracing is off

With no OTEL_EXPORTER_OTLP_ENDPOINT (the default) behaviour and hot-path cost are unchanged: telemetry::is_active() is a single relaxed atomic load, stage spans are Span::none() (no allocation, and instrumenting a future with one is a no-op), no carrier is built, and outbound trace headers are copied verbatim exactly as before.

Exporters (OTel-compatible)

rolter emits traces and metrics via OpenTelemetry OTLP (gRPC/HTTP), so any OTel-compatible backend works without code changes — just set an endpoint and headers:

• SigNoz, Grafana Tempo/Mimir, Honeycomb, Datadog (OTLP intake or the OTel Collector datadog exporter).
• Langfuse for LLM-specific observability (prompt/response, token usage and cost as traces), ingested via its OTLP endpoint or SDK.

Recommended topology: rolter → OpenTelemetry Collector → fan-out to the chosen backends. The collector also scrapes the upstream engines’ /metrics and rolter’s /metrics, keeping vendor specifics out of rolter. Configure via env, e.g. OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME=rolter-gateway.

infra/otel/collector.yaml is a ready-to-run local collector example: it accepts OTLP/gRPC and OTLP/HTTP, scrapes the compose gateway’s Prometheus endpoint, exposes collected metrics on :8889, and prints telemetry via the debug exporter. It intentionally has no external backend configured, keeping the default example safe for air-gapped use. Replace the debug exporter with an internal OTLP-compatible destination for production.

Connector delivery: rendered collector config (#836)

The observability_connectors table (#511, docs/architecture/data-model.md) lists the sinks a deployment wants its telemetry shipped to in addition to ClickHouse, but the connector’s endpoint is never called directly by rolter. Consistent with ADR-0026 — fan-out belongs in the collector, not N in-process SDK exporters each carrying its own connection/queue/retry buffer, and terminating an operator-supplied URL in the collector keeps that SSRF surface out of the data plane — GET /api/v1/connectors/collector-config (superadmin-only) renders an OpenTelemetry Collector config document from the enabled rows instead: one otlphttp exporter and one traces/metrics/logs pipeline set per connector, receiving from the otlp receiver rolter’s own OTEL_EXPORTER_OTLP_* export targets.

Point a collector at it with the confmap HTTP provider (otelcol --config=http://control:4001/api/v1/connectors/collector-config, bearer-authenticated the same as any other control-plane endpoint), or fetch it on a schedule and reload. sampling_rate becomes a probabilistic_sampler processor scoped to that connector’s own pipeline, since sampling is now the collector’s decision, applied independently per destination rather than once for the whole deployment. A managed secret (managed_auth_secret) is decrypted server-side and rendered as a literal Authorization: Bearer header; an external reference (auth_secret_ref) instead renders as a ${env:...} placeholder the collector’s own environment must resolve, since rolter never holds that secret’s value.

ROLTER_TELEMETRY_ENABLED wins over every connector row: when it is off, the rendered document has no exporters and no pipelines regardless of what is enabled in the registry, the same way it already silences every in-process exporter.

Backpressure to a slow or dead sink is the collector’s problem now, not rolter’s: it already queues and retries per exporter, which is the reason fan-out moved here in the first place rather than something this endpoint needs to build.

The remaining sink kinds from #511 (Datadog, Prometheus remote-write, Langfuse) become an exporter-block case in this renderer, not a new in-process delivery adapter — widening KINDS in connectors.rs and the kind check constraint in migration 0062 is the only rolter-side change; everything else is collector configuration.

Request & cost logs

• Every proxied request is logged to ClickHouse (request_logs): identifiers, model, provider/target, status, token counts, cost_usd, latency, TTFT, cache flag, error.
• Retention defaults to 90 days for metadata and seven days for captured payloads, set as the TTL in the ClickHouse schema. Both are admin-managed: PUT /api/v1/logging-settings accepts retention_days (1–3650) and payload_retention_hours (1–8760) and issues the matching alter table … modify ttl against ClickHouse, which then expires parts on its own schedule. Payload retention may not exceed metadata retention, so raw prompt bodies never outlive the row they belong to. A ClickHouse failure leaves the stored policy in place and is logged rather than failing the admin write — re-saving reapplies it.
• Payload capture is disabled by default. Set [logging.payload_capture] enabled = true to write redacted request and response payloads to the separate request_payloads table, which has a seven-day TTL (versus 90 days for request metadata). max_bytes bounds each body; redact_fields adds recursively redacted JSON keys before storage. Optional models and virtual_key_ids allow-lists make the deployment-level switch route- or key-specific.
• Request id / trace continuation: every request carries an x-request-id — the caller’s when supplied, otherwise a generated UUID — which is echoed on the response and stored on the log row for end-to-end correlation. An inbound W3C traceparent or B3 (b3 / x-b3-traceid) header is parsed and its trace id stored in request_logs.trace_id, so gateway logs join the caller’s distributed trace instead of starting a disconnected one.
• Outbound propagation: when the caller sent trace context, it is forwarded verbatim to the chosen upstream (traceparent, tracestate, and the b3 / x-b3-* family) so vLLM/SGLang/TGI continue the same trace. An untraced request adds nothing to the upstream wire — this is the caller’s own context, not a rolter fingerprint, so it preserves wire transparency.
• Writes are async and batched off the hot path so logging never adds request latency.
• The dashboard queries ClickHouse for usage, spend, latency percentiles and error rates, sliced by org/team/project/key/model.

Provider health events

• Every health signal is written to ClickHouse (provider_health_events): target_id, provider, source, outcome, status_code, latency_ms, error_kind, timestamped by ClickHouse on insert.
• source distinguishes where the observation came from: passive (real traffic completing through the request funnel), probe (active liveness sweeps), and the opt-in llm_call / status_page sources.
• outcome is ok / error / timeout; error_kind gives a coarse label (rate_limited, upstream_error, connect_error, timeout).
• Writes reuse the same async, batched, off-hot-path writer and ClickHouse endpoint as request_logs; when no clickhouse_url is configured the sink is a no-op.
• Counters rolter_health_events_written_total and rolter_health_events_dropped_total track the writer, mirroring the request-log counters.
• This event stream feeds uptime %/MTTR rollups and the dashboard health panel.

Stability rollup API

Read-only, window-bounded rollups over provider_health_events, served by the control plane when --clickhouse-url is set (otherwise 503). All accept since/until (RFC3339, default last 7 days); time bounds are passed as ClickHouse query parameters, never interpolated.

• GET /api/v1/health/uptime — per provider/target: event counts, uptime, failure_rate, error_budget_burn and sla_breached against an sla target (query param, fraction in (0,1], default 0.99), and last_event.
• GET /api/v1/health/mttr — per provider/target mean time to recovery (mttr_seconds) and incident count, computed from downtime episodes (a run of non-ok events bounded by ok).
• GET /api/v1/health/timeline?bucket=hour|day|week|month — bucketed ok/error/timeout counts per provider/target for the failure timeline (default bucket hour).

Health

• GET /healthz on both binaries for liveness probes.
• GET /readyz on the gateway for readiness. It returns 503 draining once the control plane marks the node as draining (PUT /api/v1/cluster/nodes/{id}/drain), so a load balancer stops sending new traffic while in-flight requests finish; /healthz stays 200 because the process is healthy. The drain reaches the node on the snapshot poll it already makes, and the control plane refuses to drain the last live gateway.

Performance

Goal: beat the reference Python proxy (LiteLLM cites ~8ms P95 added latency at 1k RPS) with a much smaller per-request overhead in Rust.

Hot-path principles

• Lock-free config reads — the routing table is an ArcSwap<Snapshot>; readers never block, even during a hot reload.
• Minimal-copy streaming — upstream responses are piped to the client as a Body::from_stream over the reqwest byte stream; rolter does not buffer whole responses.
• Connection reuse — pooled reqwest clients with HTTP/2 keep-alive and tcp_nodelay; one client per egress-proxy target, cached.
• Cheap auth — virtual-key lookup is an O(1) hash-map hit on the in-memory snapshot.
• Avoid full deserialization — only the fields needed for routing (model, stream) are read; the body is forwarded as raw bytes.
• Logging off the hot path — usage/cost rows are batched and written to ClickHouse asynchronously.
• Release profile — lto = "thin", codegen-units = 1, strip = true.

Things to watch

• The approximate cache-aware trie is per-route in-memory state; bound its size with eviction before it grows large.
• Per-request JSON parse for model is small but measurable; consider a fast path / partial parse for very high RPS.
• Prefer bytes::Bytes (ref-counted) over Vec<u8> copies when rewriting the model field.

Benchmarking

Two layers, neither on the per-PR gate:

• Micro-benches (just bench) — criterion, covering balancer pick across every strategy and the prefix trie. Compiled on every PR via clippy so they cannot bit-rot, but not run (timings are noisy on shared runners).
• End-to-end load (just bench-sim / bench-vllm / bench-sglang) — integration/engines/bench.py against a real engine, measuring rolter’s added latency directly and its behaviour under sustained concurrency.

Tool decision (#847, closing #455)

The load harness extends bench.py rather than adopting GuideLLM or driving load with oha/k6. Recorded here because #847 required settling it before writing code.

• Air-gapped operation is a hard requirement for rolter. bench.py is stdlib-only — no install step, no wheels or binaries to vendor. GuideLLM brings a substantial Python dependency tree; oha/k6 are external binaries. A benchmark you cannot run on the disconnected machine you are tuning is not much of a benchmark.
• The direct-vs-rolter delta is the entire point of this harness, and no general-purpose load generator produces it. They measure one endpoint; added_latency_p50_ms needs both driven identically in the same run under the same conditions. Adopting one of them means keeping bench.py anyway for the delta — two harnesses instead of one.
• ITL needs SSE token boundaries. oha and k6 measure bytes and requests, not tokens, so they cannot produce inter-token latency at all. GuideLLM does understand tokens, but the two points above still stand.
• Reproducible without credentials — the default sim engine (llm-d-inference-sim) needs no API key, so anyone can reproduce a number.

The cost is carrying ~250 lines of stdlib Python for concurrency and knee detection. That was judged cheaper than a dependency tree plus a second harness. Revisit if the harness starts needing tokenizer-aware workload shaping, which is where GuideLLM genuinely earns its footprint.

What it measures

bench.py drives the engine directly and through rolter in the same run:

Profiles: --concurrency holds closed-loop steady state after a warmup, and --sweep 1,2,4,8,16 walks concurrency upward — a ramp/burst profile — reporting max sustainable RPS: the highest achieved throughput whose p99 stayed inside --knee-factor (default 2×) of the lowest-concurrency baseline and which was not erroring.

That bound matters. Throughput usually keeps climbing well past the point where latency collapses, so a peak RPS with no latency condition attached describes a system nobody would actually run. The report also names the concurrency level where the knee was crossed, so a reader can see where it went rather than only that it did.

just bench-sim     # sequential: per-request added latency (fast, unchanged)
just load-sim      # concurrency sweep: max sustainable RPS, ITL, error rate
just test-bench    # harness unit tests — stdlib only, no engine or network

load-* is deliberately a separate recipe: a sweep across five concurrency levels takes minutes, and silently making the added-latency run that much slower would be a bad trade. Tune with LOAD_LEVELS, LOAD_REQUESTS, LOAD_WARMUP and LOAD_MAX_TOKENS.

Two measurement rules worth knowing when reading the JSON:

• Failed requests are counted, never timed. A connection refused in 0.1 ms is not a fast request; letting it into the percentiles would make a saturated system look quicker than a healthy one. They appear in error_rate instead.
• ITL keys are absent, not zero, when nothing streamed. A reported ITL of 0 would read as “instant tokens” rather than “not measured”.

Baseline

No baseline snapshot is recorded yet. It needs a run on fixed, documented hardware, and a number taken on a laptop under thermal throttling would be worse than none — an unreproducible baseline invites false regressions. Record hardware, engine version and rolter version alongside the numbers when one is taken.

Inference engines

The ROL-238 suite checks rolter against OpenAI-compatible engine servers. Output is intentionally meaningless; this validates the HTTP, OpenAI JSON, and SSE contracts rather than model quality.

The default engine, sim, is llm-d-inference-sim: a ~30MB multi-arch vLLM API simulator that needs no model downloads and boots in milliseconds, which makes the suite cheap enough to run as a regular PR check. The real CPU vLLM and SGLang profiles remain for on-demand runs; they use trl-internal-testing/tiny-random-LlamaForCausalLM only for its configuration/tokenizer and initialize random weights with --load-format dummy (with a head_dim=64 override, since the CPU attention kernels reject the model’s native head_dim=4). The CPU vLLM profile uses eager execution to avoid expensive compilation warm-up during CI smoke runs.

It runs on CPU in Docker and therefore works on GitHub-hosted runners. Each engine profile starts two independent dummy upstreams so the gateway exercises a real target pool. Run one engine locally:

just integration-sim
just integration-vllm
just integration-sglang

Each command boots routes for every balancing strategy (round-robin, random, power-of-two, consistent-hash, cache-aware, weighted, pipeline, cheapest, and fastest). It verifies /v1/models, non-streaming chat, and SSE both directly and through rolter, and explicitly confirms round-robin reaches both targets. Logs are kept in artifacts/engines/<engine>/.

Local end-to-end run

just integration-vllm and just integration-sglang start two CPU dummy-weight servers, render the gateway configuration, and run the OpenAI JSON and SSE assertions. The runner cleans up all containers and child processes on exit and preserves the combined engine log plus the gateway log under artifacts/engines/<engine>/.

For manual inspection, start the selected two-server pool and leave it running:

docker compose -f docker/docker-compose.engines.yml --profile vllm up -d
# use profile sglang for ports 30000 and 30001

Render the gateway configuration in another terminal and start rolter:

config=$(mktemp)
sed \
  -e 's/__ROLTER_PORT__/4010/g' \
  -e 's/__ENGINE_1_PORT__/8000/g' \
  -e 's/__ENGINE_2_PORT__/8001/g' \
  integration/engines/rolter-dummy.toml.in >"$config"
cargo run -p rolter-gateway -- --config "$config"

Verify non-streaming JSON and streaming SSE through the gateway:

curl -i http://127.0.0.1:4010/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"dummy-round-robin","messages":[{"role":"user","content":"Reply with one token."}],"max_tokens":1,"temperature":0}'

curl -N http://127.0.0.1:4010/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"dummy-round-robin","stream":true,"messages":[{"role":"user","content":"Reply with one token."}],"max_tokens":1,"temperature":0}'

Exercise all configured strategies; each request must return a non-empty OpenAI choices array:

for model in dummy-round-robin dummy-random dummy-power-of-two \
  dummy-consistent-hash dummy-cache-aware dummy-weighted dummy-pipeline \
  dummy-cheapest dummy-fastest; do
  curl -fsS http://127.0.0.1:4010/v1/chat/completions \
    -H 'content-type: application/json' \
    -d "{\"model\":\"$model\",\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}],\"max_tokens\":1}" \
    | jq -e '.choices | length > 0' >/dev/null
done

Clean up the pool and temporary gateway configuration:

docker compose -f docker/docker-compose.engines.yml --profile vllm down --volumes
rm -f "$config"

For non-gating direct-versus-gateway samples, run just bench-vllm or just bench-sglang. They record non-streaming and streaming p50/p95/p99 latency and streaming first-byte time in JSON. Results only compare runs on the same host, CPU image, engine versions, and host configuration; throughput thresholds are deliberately not merge gates.

The engine integration workflow runs the sim smoke as a regular check on pull requests that touch engine paths. Dispatch it manually with engine=vllm to smoke the real CPU engine (Actions tab, or gh workflow run "engine integration" -f engine=vllm). SGLang remains available through the local just integration-sglang command, but its source-built CPU image is currently too heavy for the shared CI gate. This suite is for compatibility, not a performance gate. When ROL-67 lands, add the equivalent /v1/messages assertion through the gateway.

Architecture Decision Records

Lightweight decision log. Each entry: Status · Context · Decision · Consequences. Supersede rather than rewrite.

A standalone record opens with its title and a single metadata line — no table:

# Title of the decision

**Status:** Accepted · **Date:** 6 Aug 2026 · **Issues:** [#812](https://github.com/rolter-ai/rolter/issues/812)
**Supersedes:** ADR-0007 for exact vLLM modes
**Relates:** ADR-0022 (config-vs-DB tiering)

## Context

Status, Date and Issues are the line; Supersedes and Relates get their own line and are omitted when there is nothing to say. Nothing else belongs there — authorship and dates are what git is for, and a field whose value is either constant across every record or a — (unassigned) placeholder is noise that makes the actual decision harder to find. Records are English only.

ADR-0001 — Rust + Axum/Hyper/Tower for the data plane

Accepted. Need maximum proxy throughput with rich API semantics and SSE streaming. Chose Axum/Hyper/Tower on Tokio over Pingora/Actix for ecosystem fit and ergonomics. Consequence: idiomatic async stack; revisit Pingora only if profiling demands it.

ADR-0002 — Two-binary topology over shared crates

Accepted. Keep the hot proxy path lean and independently scalable from management. rolter-gateway (data plane) and rolter-control (management + UI host) share library crates. Consequence: clear seam; some duplicated wiring.

ADR-0003 — Vite + React + shadcn/ui SPA, Bun toolchain, served by Rust

Accepted. shadcn/ui (Radix + Tailwind) for the dashboard, built with Vite and managed with Bun; output is static assets served by rolter-control (no Node runtime in prod). Consequence: simple prod footprint; Bun used for install/dev/build.

ADR-0004 — Postgres + Redis + ClickHouse

Accepted. Postgres = source of truth (config/RBAC/keys/pricing); Redis = cache + rate-limit counters + config pub/sub; ClickHouse = high-volume request/cost logs. No SQLite. Consequence: three datastores to operate; each fits its job.

ADR-0005 — Org → Team → Project → Virtual Key tenancy

Accepted. Budgets and rate limits attach at any scope, most-restrictive-wins. Consequence: flexible multi-tenancy; enforcement must resolve a scope chain.

ADR-0006 — Local accounts + virtual keys + roles; SSO/LDAP later

Accepted. v1 ships local accounts (argon2id) and roles admin/member/viewer; OAuth2/OIDC and LDAP arrive as pluggable identity providers. Consequence: usable day one without an IdP.

ADR-0007 — Approximate cache-aware balancing behind a pluggable trait

Accepted. v1 uses an approximate per-target prefix trie (no engine coupling) behind LoadBalancer. Precise (KV-events) and lmcache-aware land later without API changes. Consequence: immediate wins; precise mode is additive.

ADR-0008 — Reload-free config: Postgres truth + Redis pub/sub + ArcSwap

Accepted. Control plane writes Postgres, bumps a version, publishes on Redis; gateways fetch and atomically swap an in-memory snapshot, reconciling by version. Consequence: instant fan-out, self-healing, lock-free reads.

ADR-0009 — Envelope encryption for provider secrets

Accepted. Upstream keys are AES-256-GCM envelope-encrypted with a master key from env/file; Vault/KMS backends later. Consequence: no plaintext secrets at rest; master key is the critical secret.

ADR-0010 — Packaging: maturin (uv) + cargo + Docker

Accepted. Ship a maturin-built PyPI wheel bundling the unified rolter launcher (uv tool install rolter), cargo install rolter, and a multi-stage Docker image. The rolter binary dispatches to gateway/control subcommands so one wheel/crate ships the whole system. Consequence: three distribution paths from a single named artifact.

ADR-0011 — API surface v1

Accepted. OpenAI /v1/chat/completions, /v1/completions, /v1/models and Anthropic /v1/messages. Embeddings, images, audio and other modalities follow. Consequence: drop-in for the two dominant client SDKs first.

ADR-0012 — Conventional Commits + CI PR-title lint

Accepted. Commit messages and PR titles follow Conventional Commits; enforced by commitlint, conventional-pre-commit, and a CI PR-title check. Consequence: consistent history, automatable changelogs/releases.

ADR-0013 — OpenTelemetry-based observability with engine propagation

Accepted. Export traces/metrics via OTLP to any compatible backend (SigNoz, Datadog, Grafana, Langfuse, …); propagate W3C trace context to vLLM/SGLang so engine spans join the same trace; federate upstream engine metrics. Consequence: vendor-neutral observability; an OTel Collector is the recommended hub.

ADR-0014 — Extensible API protocol translation

Accepted. Resolve translation by client/upstream protocol pair in rolter-proxy, including incremental SSE, while the gateway retains transport, caching and accounting ownership. Consequence: new provider dialects extend one translation boundary; non-equivalent modalities remain explicit and are never silently dropped.

ADR-0015 — OpenAI Responses API translation

Development. Add OpenAI Responses as a protocol pair for native OpenAI, Chat Completions and Anthropic Messages, while model-less lifecycle operations remain uniformly unsupported until tenant-scoped storage exists.

ADR-0016 — Routing OpenAI Responses resources through a tenant-scoped registry

Development. Pin Responses lifecycle operations to a bounded tenant-scoped process-local record, preserving the original provider credential while making unknown and cross-tenant IDs indistinguishable.

ADR-0017 — Provider/model addressing to disambiguate identical model names

Accepted. First-class provider-slug/model addressing coexists with named routes: a stable, URL-safe provider slug resolves provider-slug/model to a pinned (provider, upstream_model) target (avoiding LiteLLM’s base_url ambiguity). Pinning bypasses cross-provider fan-out but still balances within the provider. Slugs are org-scoped and collision-safe: deterministic reported migration backfill with -N de-dup, 409+suggestion on runtime create-conflict (never silent suffixing), soft-delete-preferred reclaim. Addendum: provider groups unify fleets (e.g. ten vLLM instances) under one group slug in the same namespace — vllm-cluster/qwen3 fans out across members with a chosen balancing strategy and passthrough model names, so no route-per-model. Consequence: new immutable slug column + proxy parsing + /v1/models + UI work + provider-groups entity (see follow-up issues).

ADR-0018 — Config mutated via granular CRUD, not whole-config replace

Accepted. The normalized Postgres store (ADR-0004, source of truth) is mutated only through the scoped CRUD API — providers, routes, targets, virtual keys — each write bumping the config version and hot-reloading gateways via the snapshot poll (ADR-0008). PostgresConfigStore::save is deliberately read-only: there is no whole-config “apply” endpoint, because a full replace would fight the normalized model and clobber concurrent edits. The dashboard Config page is a read-only effective-config viewer; the /gw reverse-proxy that fronts the gateway for the Playground is unrelated to config writes. Consequence: one live-edit path (CRUD) with instant fan-out; a raw whole-config editor would need a deliberate transactional diff/apply (guarding config-owned entries, preserving key hashes) and is out of scope (closed #494).

ADR-0019 — Per-provider egress proxy pools

Accepted. Rotate across a provider-local proxy pool, fail over only connection/tunnel failures, and quarantine repeatedly failing members. Authenticated URLs are resolved exclusively from whole-value environment references. Consequence: resilient egress without leaking credentials; health state remains process-local.

ADR-0020 — Bounded semantic response caching in Redis

Accepted. Run semantic lookup only after an exact miss, embed through an explicitly configured provider, and scan a bounded recent Redis window. Consequence: similarity reuse without another datastore; embedding and cache failures fail open and candidate search remains deliberately bounded.

ADR-0021 — External cache telemetry for routing

Accepted. Extend ADR-0007 with opt-in vLLM KV-event and LMCache occupancy scorers whose network I/O runs in background tasks. Consequence: exact/capacity-aware routing stays allocation-light and fail-open; stale or untrusted telemetry becomes neutral and least-load selection remains available.

ADR-0022 — Uniform config-vs-DB tiering for models, providers, and provider groups

Accepted. Give models, providers, and provider groups the same two-tier config wrapper over the DB (CRUD) tier: readonly entries are immutable and config-owned; default entries are seeded into the default project once at startup and then owned/editable via API/UI; pure DB rows come from CRUD. readonly wins resolution; a default colliding with a readonly key is a load-time error. Top-level [[routes]]/[[providers]]/[[provider_groups]] stay as deprecated readonly aliases for back-compat. Consequence: providers gain a seed-then-edit story and provider groups gain a full DB lifecycle (new tables/repos/seed/CRUD, tracked as follow-up PRs); the model tiering pattern is reused rather than reinvented per entity.

ADR-0023 — Propagating access-profile model policy to the data plane

Accepted. Resolve each virtual key owner’s merged access-profile policy when the snapshot is built, carry it on the key record, and enforce it on the gateway’s model and route selection; the key’s own allow-list and the owner’s policy must both permit a model. Rejects key-mint-time resolution because a later policy edit would never reach issued keys, so a revocation would silently fail to revoke. Adds bump_config_version() triggers to exactly the four tables that feed the resolution, changing migration 0058’s premise rather than its rule. Consequence: the policy /rbac/effective reports is the policy enforced, and a deployment with no access profiles is unaffected.

ADR-0024 — Dashboard UX telemetry as a structural-only event stream

Accepted. Collect dashboard usability events into a ui_events ClickHouse table beside request_logs rather than adding a product-analytics vendor, and make the schema itself the privacy guarantee: every column is a key, an enum, a duration or an id, so a form value or prompt has nowhere to land. Attribution is server-side; the ingest endpoint is guarded by CurrentUser with no capability row, because a create capability granted to any authenticated caller would break the RBAC invariant that a viewer writes nothing. Consequence: usability data with no new vendor, egress or retention policy, joinable to gateway traffic on trace_id; the cost is that any question needing a value rather than a key is unanswerable by design.

ADR-0025 — Events as logs, not span events

Accepted. OpenTelemetry is deprecating the Span Events API, so log-based events become rolter’s event model: no new explicit add_event/record_exception, and #809 (OTLP log export) sequences before #808 (GenAI conventions) because those conventions are specified as log-based events and cannot be emitted until logs are exported at all. A grep finds zero uses of the deprecated API, but tracing-opentelemetry turns every tracing event fired inside a span into a span event, so rolter is in practice an exclusive implicit user of it — which makes the migration a dependency upgrade in one bridge rather than a sweep of 57 call sites. Consequence: tracing stays; #809’s trace_id/span_id correlation becomes a blocking acceptance criterion rather than a nicety; new event types have no correct home until it lands, and should wait rather than add a span event that must later be removed.

ADR-0026 — Client control over telemetry: a kill switch now, collector-routed tenant destinations later

Accepted. Splits #812 in two. ROLTER_TELEMETRY_ENABLED ships now as one explicit off for every signal — traces, metrics, the dashboard’s browser tracing, and logs once #809 lands — that can only subtract, defaults to enabled so no existing deployment changes, and leaves export on for an unrecognized value so a typo cannot silently blind a deployment. It is environment-only despite the issue asking for a config key: telemetry::init installs the subscriber before any config file is read, so a config-file switch could not gate trace export at all. Per-tenant destinations are deferred and, when built, belong in the OpenTelemetry Collector’s routing processor keyed on an org resource attribute rather than in-process exporters — that keeps one egress path in the gateway and keeps tenant-supplied URLs, an SSRF surface, out of the data plane entirely. Consequence: operators get a reviewable “no telemetry leaves here” today; per-tenant routing later requires running a collector, which is already the recommended topology.

ADR-0027 — End-to-end test harness: Python/uv project driving a black-box stack

Accepted. The e2e suite is a Python driver on the latest CPython managed by uv, exercising the real HTTP APIs of a docker-compose stack (postgres, redis, clickhouse, control, gateway, and N llm-d-inference-sim fake-vLLM engines) with no in-process shortcuts, so the tests see exactly what an operator or tenant sees. Numbered after ADR-0026 despite its earlier date because ADR numbers are append-only and never reused. Consequence: the suite needs a container runtime rather than cargo test alone; in exchange it covers the wiring between the two binaries and the three datastores that unit tests cannot reach.

ADR-0028 — Disaggregated prefill/decode routing belongs to the engine, not the gateway

Accepted. rolter will not implement P/D disaggregation; a disaggregated fleet is one upstream and its own coordinator owns phase selection and the KV handoff. Engines move KV tensors through connector-selected transports such as NIXL/UCX. Some can coordinate that transfer through engine-specific HTTP fields, but rolter would still have to own compatible-worker topology, version-specific metadata, two upstream calls and per-request handoff state — exactly the lifecycle coupling ADR-0014 prevents. The distinguishing test is whether rolter can consume an input through a stable, engine-independent contract without joining the engine’s request lifecycle. Consequence: disaggregated fleets work with rolter today for any engine, with no code and no mid-request-handoff failure modes; revisit if a stable phase-placement contract emerges on the OpenAI or Anthropic surface.

OpenAI Responses API translation

Status: Development · Date: 13 Jul 2026 · Issues: ROL-252

Context

ROL-252 adds the public POST /v1/responses. The gateway already routes on model, and rolter-proxy already holds an isolated registry of OpenAI Chat Completions and Anthropic Messages translations with incremental SSE handling. Responses carries text, multimodal input, tools, tool results and its own streaming event model. Lifecycle operations use a response identifier without a model, so they cannot safely select a route without storing the tenant/key → upstream response association.

Options considered

Option 1 — Native passthrough only

Forward /v1/responses to every provider without translation.

flowchart LR
    C[Responses client] --> G[rolter gateway]
    G --> P[upstream /v1/responses]

Option 2 — Protocol registry with Responses

Add Responses as a protocol: native OpenAI gets passthrough, while Chat Completions and Anthropic Messages get adapted requests and return Responses objects or SSE events.

flowchart LR
    C[Responses client] --> G[rolter gateway]
    G -->|OpenAI| R["/v1/responses"]
    G -->|Chat-compatible| CC["/v1/chat/completions"]
    G -->|Anthropic| AM["/v1/messages"]
    R --> G
    CC --> G
    AM --> G

Comparison

Decision

Option 2 was chosen. Protocol::OpenAiResponses is added to the rolter-proxy registry. ProviderKind::Openai uses the native /v1/responses; Chat-compatible providers use /v1/chat/completions; Anthropic uses /v1/messages. Common fields are translated, and responses and streaming events are returned in the Responses representation.

Rationale

This preserves the existing architectural boundary: the gateway owns authentication, tenant scope, target selection, tracing, metrics and accounting, while rolter-proxy owns only the wire protocol. Native Responses loses no provider-specific fields, and the existing Chat and Anthropic routes stay reachable for clients on the new OpenAI SDK.

Consequences

Benefits:

• one shared Responses endpoint works across three families of upstream APIs;
• SSE is translated incrementally, without buffering the live stream;
• input_tokens/output_tokens are counted by the existing cost and rate-limit accounting.

Drawbacks and risks:

• background, store, previous_response_id and provider-specific reasoning have no safe Chat/Anthropic equivalent and are not forwarded there;
• lifecycle for native OpenAI is implemented by a separate tenant-scoped registry in ADR-0016; translated Chat/Anthropic resources still return 501 response_lifecycle_unsupported;
• supporting new Responses event types requires extending the converter and its tests.

System impact:

rolter-gateway, rolter-proxy, the OpenAPI spec and the API documentation change; the storage schema does not.

Related records

• ADR-0014 — Extensible API protocol translation
• ROL-252 — OpenAI Responses API passthrough and streaming

Open questions

• Consider a distributed backend registry for multi-replica deployments without sticky routing.
• Extend streaming translation of function-call and reasoning event types as supported upstream contracts are confirmed.

Routing OpenAI Responses resources through a tenant-scoped registry

Status: Development · Date: 13 Jul 2026 · Issues: ROL-252, ROL-264

Context

ROL-252 added OpenAI Responses creation, routed on the model field. ROL-264 adds GET and DELETE /v1/responses/{id}, POST /v1/responses/{id}/cancel and GET /v1/responses/{id}/input_items. These requests carry no model, tenant or provider, and the upstream identifier cannot safely be used to re-balance: the request could go to a different provider, target or provider key and expose another tenant’s resource.

The gateway already authenticates the virtual key, atomically refreshes the routing snapshot and selects a concrete provider credential. A safe lifecycle requires remembering that decision after a successful POST /v1/responses, streaming responses included. Translated Chat Completions and Anthropic Messages do not create a persistent upstream Responses resource.

Options considered

Option 1 — Re-select the route from the response ID

Hash response_id and use the current balancer or iterate over providers.

flowchart LR
    C[Lifecycle client] --> G[rolter gateway]
    G --> B{Current balancer}
    B --> P1[Provider A]
    B --> P2[Provider B]

Option 2 — Bounded process-local tenant registry

After a successful creation, store the composite key virtual-key digest + response_id and pin it to the provider, target, model and provider-credential fingerprint. The entry has a TTL and is removed after a successful DELETE.

sequenceDiagram
    participant C as Client
    participant G as Gateway
    participant R as Tenant registry
    participant P as Native OpenAI provider
    C->>G: POST /v1/responses with model
    G->>P: create via selected target and credential
    P-->>G: response ID or SSE response event
    G->>R: store tenant plus pinned route and capabilities
    C->>G: GET, DELETE, cancel or input_items by ID
    G->>R: lookup by authenticated tenant plus ID
    R-->>G: pinned native route
    G->>P: lifecycle request with same credential

Option 3 — Distributed registry in Redis

Store the same entries in shared Redis with a TTL, so that any gateway replica can serve a lifecycle request.

flowchart LR
    C[Client] --> G1[Gateway replica 1]
    C --> G2[Gateway replica 2]
    G1 --> R[(Redis registry)]
    G2 --> R
    G1 --> P[OpenAI provider]
    G2 --> P

Comparison

Decision

Option 2 was chosen. rolter-gateway keeps a bounded process-local registry. The key is composed of a peppered digest of the virtual key and the public response_id; the plaintext key is not stored. The value records the provider, target, public model, provider-native ID, the fingerprint of the selected provider credential and capability flags.

By default an entry lives for 24 hours and a single process holds at most 100,000 entries. Both are configured through [responses] registry_ttl_secs and registry_max_entries; a zero value disables registration. Expiry and overflow are cleaned up lazily on lookup/insert. A successful DELETE removes the entry immediately.

Rationale

A process-local registry closes the main security invariant without requiring Redis on the inference path. The composite key makes unknown, expired and cross-tenant lookups indistinguishable. Pinning to the provider credential prevents reaching the resource through a different upstream account. The fingerprint makes it possible to find the same credential after the key pool is reordered, without storing the secret a second time.

Translated Chat Completions and Anthropic Messages get an entry with empty lifecycle capabilities. That allows returning a precise 501 response_lifecycle_unsupported to the owner while keeping a uniform 404 response_not_found for any other tenant.

Consequences

Benefits:

• a lifecycle request is never re-balanced and always uses the original provider account;
• unknown, expired, deleted, cross-tenant and post-config-change unreachable resources all return the same 404 with no route metadata;
• the native upstream status and body are forwarded without translation;
• non-streaming JSON and completed SSE are registered from a single completion-observer point.

Drawbacks and risks:

• multi-replica deployments require sticky routing by client or response ID;
• a gateway restart drops the registry earlier than upstream retention;
• the full response is temporarily buffered by the existing accounting stream; the registry reuses that buffer, but the memory footprint is not reduced;
• deleting a provider, changing its kind or rotating the original credential makes the entry unreachable and returns 404.

System impact:

rolter-core (retention configuration), rolter-gateway (registry, lifecycle handlers and response observer), rolter-proxy (model-less forwarding), the OpenAPI spec, the API documentation and the engine smoke suite change. The Postgres/Redis schemas do not.

Related records

• ADR-0005 — Org → Team → Project → Virtual Key tenancy
• ADR-0014 — Extensible API protocol translation
• ADR-0015 — OpenAI Responses API translation
• ROL-252 — OpenAI Responses API passthrough and streaming
• ROL-264 — Responses API lifecycle resources

Open questions

• Add an optional Redis registry backend if lifecycle must work without sticky routing across replicas.
• Extend capability discovery once OpenAI-compatible providers reliably implement the native Responses lifecycle.

Provider/model addressing to disambiguate identical model names

Status: Accepted · Date: 14 Jul 2026 · Issues: ROL-266

Context

When several providers serve the same model name, a client has no first-class way to say which one it wants. This is common with self-hosted OpenAI-compatible backends: two vLLM/SGLang instances at different base URLs both serving qwen3, or an openai-kind provider whose base_url points at a non-OpenAI upstream.

How rolter addresses models today

• A client sends model = a route name (routes.model, unique per project).
• A route fans out to one or more route_targets, each {provider_id, upstream_model?, weight}.
• The proxy rewrites the outgoing model to upstream_model when set (crates/rolter-proxy/src/lib.rs:333, maybe_rewrite_model); target selection and failover happen in crates/rolter-gateway/src/handlers.rs (weighted balancer over entry.route.targets).
• Providers are unique(org_id, name); there is no URL-safe slug.

flowchart LR
    C[client: model = route name] --> R[route lookup]
    R --> B[balancer over targets]
    B --> T1["target A · provider vllm-msk · upstream qwen3"]
    B --> T2["target B · provider vllm-spb · upstream qwen3"]

So disambiguation today means the operator invents distinct route names (qwen3-msk, qwen3-spb). There is no client-facing provider/model addressing and no convention linking a route name to a concrete provider.

Prior art

• LiteLLM — a two-tier scheme: clients send a public model_name alias; each deployment carries litellm_params.model with a provider prefix (openai/gpt-4o) plus its own api_base/credentials. Multiple deployments sharing one model_name are load-balanced. The prefix is a routing hint tied to a base_url, so you can point openai/... at a Qwen endpoint by swapping base_url — the prefix is semantically muddy and is not a stable identity.
• Envoy AI Gateway — extracts the model field from the request body into an x-ai-eg-model header before routing, then applies ordinary header-match rules in an AIGatewayRoute to pick a backend. Selection is out-of-band (header/route config), not encoded in the model string the client sends.

The lesson: keep the address segments stable identities, not free-form hints (avoid LiteLLM’s base_url ambiguity), while still letting a route fan out.

Options considered

Option A — Status quo (distinct route names)

Operators keep inventing unique route names per provider (qwen3-msk).

• Pro: zero code; already works; route name stays the single routing key.
• Con: no convention; clients must know deployment-specific names; poor discovery; the provider is invisible in the address.

Option B — First-class provider-slug/model addressing (coexisting)

Introduce a stable, URL-safe slug on providers. A client may send either a bare route name (unchanged) or provider-slug/model. The gateway resolves provider-slug/model to the concrete (provider, upstream_model) pair, pinning the provider and using model as the upstream model.

flowchart LR
    C["client: model = 'vllm-spb/qwen3'"] --> P{"contains '/' and<br/>left = known provider slug?"}
    P -- no --> RN[resolve as route name today]
    P -- yes --> PR["pin provider vllm-spb"]
    PR --> UM["upstream model = 'qwen3'"]
    UM --> FW[forward to that provider only]

• Pro: unambiguous, self-describing addressing; slug is a stable identity (fixed kind + base_url), avoiding LiteLLM’s base_url muddiness; coexists with named routes, so it is additive and backward-compatible; maps cleanly to “pick a provider from the list, or add one inline, then reference its models” in the UI.
• Con: needs a new slug column (migration + uniqueness/charset rules + CRUD/UI); parsing precedence and slash-collision rules to define; a pinned provider bypasses multi-provider fan-out (see open questions — it can still fan out across that provider’s own key pool / targets).

Option C — Auto-derived provider/model aliases (read-only convenience)

No new column: derive an alias by slugifying the existing provider name and pairing it with each target’s upstream_model, exposed only through /v1/models and accepted on input.

• Pro: no schema change; quick.
• Con: provider name is mutable and only unique(org_id, name) — renames silently break addresses; slugifying a display name yields collisions and unstable ids. Same fragility LiteLLM has. Rejected as the primary path.

Option D — Out-of-band provider selector header

Keep model as today; add an optional x-rolter-provider: <slug> header (Envoy-style) to pin the provider.

• Pro: no change to the model string; no slash-parsing.
• Con: not expressible in stock OpenAI/Anthropic SDK model fields, so clients that can only set model (the common case) cannot use it; discovery is worse. Useful as a complement to B, not a replacement.

Comparison

Recommendation

Adopt Option B — first-class provider-slug/model addressing that coexists with today’s named routes — and optionally add Option D later as a complementary header for clients that need to pin a provider without touching model.

Rationale: B gives unambiguous, self-describing, rename-safe addressing that fits the OpenAI/Anthropic model field clients already use, and matches the desired UX (“pick an existing provider or add one inline, then reference provider-slug/model”). It is purely additive: existing bare-model routes keep working.

Proposed resolution semantics (to confirm)

1. Precedence: try the whole model string as a route name first (preserves any existing route whose name contains /). If unmatched and the string contains /, split on the first /: if the left segment is a known provider slug in the caller’s scope, treat the right segment as the upstream model and pin that provider; otherwise fall through to the normal not-found path.
2. Slug: ^[a-z0-9][a-z0-9-]{0,62}$, unique(org_id, slug), immutable-by-default (renaming display name never changes the slug). Backfill existing providers from a slugified name with numeric de-dup on migration.
3. Balancing: a provider-slug/model request pins the provider and bypasses cross-provider fan-out, but still uses that provider’s key pool, cooldowns, and (if the provider has multiple same-model targets) intra-provider selection. Bare route names keep full multi-provider balancing.
4. /v1/models: list both — existing route ids and provider-slug/model ids (grouped by provider in a follow-up UI), so either address is discoverable.

Slug collision handling

Slugs are unique(org_id, slug), so the DB is the final arbiter — but “just add a unique index” leaves the behaviour around collisions unspecified. This section pins it down for the three moments a collision can happen. Note first that slugs are org-scoped: there is no cross-org collision, and provider-slug/model always resolves within the caller’s org (enforced in the query where org_id = $caller), so the blast radius of any collision is a single org.

1. Migration backfill (deterministic). Shipped in crates/rolter-store/migrations/0014_provider_slug.sql:

• Providers are processed per org in a stable order (id ascending via row_number() over (partition by org_id, base order by id)), so the result is reproducible.
• Slugify name: lowercase, map every run outside [a-z0-9] to a single -, trim leading/trailing -; an empty result falls back to provider.
• The first claimant of a base slug keeps it bare (truncated to 63); each subsequent collision gets a -N suffix from its row number (vllm, vllm-2, vllm-3, …), the base truncated to 58 chars so base-N still fits. The migration never fails on a collision — it always converges.
• Follow-up: emit a migration report (one line per adjusted provider: org_id, provider_id, name, base_slug, final_slug) so operators can reconcile any externally-published address.

2. Runtime creation (validate, never silently mangle). Shipped in crates/rolter-control/src/crud.rs:

• An explicit slug is validated against ^[a-z0-9][a-z0-9-]{0,62}$; an omitted one is derived from name (a name with no ascii alphanumerics requires an explicit slug). A conflict surfaces as the DB unique-violation error — the API never auto-appends a suffix behind the operator’s back, because an address is a contract the human picks explicitly.
• Follow-up: map the unique violation to 409 with the next free suggestion ({"error": "slug taken", "suggestion": "vllm-2"}), and have the UI pre-check availability before submit.
• Slug is immutable by default (it is the stable identity). An in-place change requires the explicit allow_slug_change=true opt-in on update, which warns that the provider-slug/model address changes with it.

3. Deletion & reclaim (explicit, not automatic). A hard-deleted provider frees its slug immediately; the next create may reuse it. This is intentional but load-bearing: reusing openai after deleting the old openai repoints that address to a new upstream. Therefore:

• Prefer soft-delete for providers that ever served traffic, so a freed slug is not silently re-pointed; a reused slug is then an explicit operator action on a tombstoned name.
• Reclaim is a new identity, never a restore — document that provider-slug/model after reclaim may resolve to different weights/base_url than before.

Decision (14 Jul 2026)

• Adopt Option B, coexisting with named routes. provider-slug/model resolves to a pinned (provider, upstream_model); bare route names keep full multi-provider balancing. Named routes are not replaced.
• Balancing when pinned: a provider-slug/model request pins the provider and bypasses cross-provider fan-out, but still fans out within that provider — its key pool, cooldowns, and any same-model targets stay in rotation.
• Precedence & slug rules as proposed above (route-name-first, then first-/ split; slug ^[a-z0-9][a-z0-9-]{0,62}$, unique(org_id, slug), immutable by default).
• Collision policy as in Slug collision handling above: slugs are org-scoped (no cross-org collision); migration backfill is deterministic with -N de-dup (shipped in migration 0014); runtime creation validates and rejects conflicts rather than silently suffixing (409 + suggestion as follow-up); slug changes need the explicit allow_slug_change opt-in; hard-delete frees a slug immediately, but soft-delete is preferred so a reclaimed slug is an explicit operator action, not a silent re-point.
• Option D (header selector): deferred, not part of the initial implementation.

Addendum (20 Jul 2026) — provider groups: unifying slugs under one address

Problem

Per-provider slugs alone push fleet operators the wrong way. Run ten vLLM instances (distinct base_url + credentials each) and you get ten providers — vllm-1 … vllm-10 — where pinning any single one is the opposite of what the operator wants. Named routes can unify them, but a route binds one public model name to targets, so a cluster serving 30 models needs 30 hand-made routes. LiteLLM’s “model group / alias” answers the per-model case; nothing answers “this whole fleet, all of its models, one address”.

Decision

Introduce provider groups — an org-scoped, named set of providers that shares the slug namespace with providers (unique(org_id, slug) enforced across both), so the left segment of X/model resolves through a single unified lookup and is never ambiguous.

• Membership: groups are fully operator-defined — any slug, any member set — and membership is many-to-many: one provider may belong to several groups. Overlapping groups are the intended way to express scopes, e.g. vllm-cluster-msk (five Moscow instances), vllm-cluster-nsk (five Novosibirsk instances), and vllm-cluster-all (all ten) — the client picks the scope by address. No group nesting (groups of groups) initially: flat membership covers the scoping cases, nesting adds cycle-detection and resolution complexity for little gain; deferred.
• Resolution: precedence unchanged — whole string as route name first, then split on first /. The left segment is looked up in the unified slug namespace: a provider slug pins that provider (as decided above); a group slug fans out across the group’s members, balanced with the group’s configured strategy (any rolter_balancer::LoadBalancer), honoring per-member weight, each member’s key pool, and cooldowns.
• Model handling: default is passthrough — the right segment is forwarded unchanged as the upstream model (the homogeneous-cluster case: every member serves the same model set). An optional per-member upstream_model rewrite covers heterogeneous groups, mirroring route_targets.
• Credentials: unchanged. Clients authenticate with a rolter virtual key; member credentials stay per provider. A group gives one address, one client key, N upstreams with N distinct upstream creds — which is the unification the fleet operator actually needs.
• Relation to routes: a group is effectively a wildcard route family (vllm-cluster/*) without creating a route per model. Named routes remain the curated-alias layer for cherry-picked public names; groups cover fleets.
• /v1/models: group addresses are listed as the deduplicated union of member-served models (vllm-cluster/qwen3, vllm-cluster/llama4, …) alongside provider-slug and route ids.
• Collision interplay: groups also soften the collision pressure from the section above — individual instances can carry mundane slugs (vllm-msk-1), while the group owns the meaningful address (vllm-cluster, or even vllm).

Proposed follow-up implementation issues

1. store: add immutable slug to providers — migration, unique(org_id, slug), deterministic reported backfill with -N de-dup, 409+suggestion on create-conflict, soft-delete-preferred reclaim, CRUD wiring. (relates to ROL-81, see Slug collision handling)
2. proxy/gateway: provider-slug/model parsing + resolution with the precedence rule and provider pinning; interaction with maybe_rewrite_model/upstream_model; tests.
3. gateway: extend /v1/models to surface provider-slug/model ids.
4. ui: model-management surfaces the provider-slug/model address; inline add-provider flow. (relates to ROL-222)
5. (optional) proxy: x-rolter-provider header selector (Option D).
6. store/gateway/ui: provider groups — provider_groups entity (slug in the unified namespace, strategy, members with weights), group-slug/model resolution with passthrough + per-member rewrite, /v1/models union listing, group CRUD/UI. (see Addendum: provider groups)

Sources

• LiteLLM proxy config — model_name / litellm_params / load balancing / wildcard
• Envoy AI Gateway — model-name-based routing via x-ai-eg-model

Per-provider egress proxy pools

Status: Accepted · Date: 18 Jul 2026 · Issues: #305

Context

A provider may need several outbound proxies for regional reachability, IP-based rate limits, or resilience. The existing singular egress_proxy setting cannot spread traffic or recover when its proxy becomes unavailable. Proxy credentials must not enter database rows, config snapshots, API responses, or metrics labels.

Options considered

1. Keep one proxy and rely on an external proxy load balancer.
2. Select a random proxy independently for every attempt.
3. Maintain a per-provider round-robin pool with local failure quarantine.

Decision

Adopt option 3. egress_proxies is the canonical ordered pool, while the legacy egress_proxy remains a backward-compatible one-element pool. Each member owns a cached reqwest client and may use HTTP, HTTPS, SOCKS5, or SOCKS5H. Requests start at the next round-robin member and retry connect or tunnel failures through the remaining eligible members. Three consecutive failures quarantine a member for 30 seconds; a successful request clears its failure streak.

Authenticated URLs are accepted only as whole-value ${ENV_VAR} references. Metrics identify members with a redacted stable label and never expose userinfo.

Consequences

• Providers can spread traffic and survive an individual proxy failure without a separate egress load balancer.
• Existing singular configuration continues to work unchanged.
• Retry is deliberately limited to connection/tunnel failures; retrying arbitrary HTTP responses could duplicate a request whose upstream already processed it.
• Quarantine state and round-robin position are process-local, so gateway replicas may temporarily make different choices.
• Operators must inject authenticated proxy URLs into every gateway replica.

Bounded semantic response caching in Redis

Status: Accepted · Date: 18 Jul 2026 · Issues: #261

Context

The exact response cache only reuses byte-equivalent normalized requests. Similar prompts still reach an upstream model even when an existing response would be acceptable. Semantic reuse adds an embedding request and a nearest-neighbour search, but it must not make gateway availability depend on the embedding provider or Redis and must keep request-path work bounded.

Options considered

1. Keep exact caching only.
2. Introduce a dedicated vector database and approximate-nearest-neighbour index.
3. Store embeddings alongside Redis cache entries and scan a bounded recent window.

Decision

Adopt option 3 as an opt-in route capability. Exact lookup always runs first. On an exact miss, rolter embeds the normalized prompt using an explicitly configured provider/model and compares cosine similarity against at most max_candidates recent entries in the route/key-isolated Redis index. A response is reused only when its score reaches threshold. Semantic metadata and the exact response share the route cache TTL.

Embedding, Redis, decoding, dimension, missing-entry, and stale-entry failures all fail open to normal upstream routing. Streaming responses enter both indexes only after the stream completes successfully.

Consequences

• Similar prompts can avoid model cost and latency without a new datastore.
• Exact-cache behavior and its faster lookup remain unchanged.
• Search cost is predictable but linear in the configured candidate bound; this is not intended as a large corpus vector-search engine.
• The embedding provider adds cost and latency on semantic misses and becomes part of cache-result quality, though never gateway availability.
• Operators own the similarity threshold and must account for unsafe reuse in domains where small wording changes alter the correct answer.

External cache telemetry for routing

Status: Accepted · Date: 18 Jul 2026 · Issues: #258, #259 Supersedes: ADR-0007 for exact vLLM and LMCache-aware modes

Context

ADR-0007 deliberately introduced approximate prompt-prefix affinity without coupling the gateway to an inference engine. vLLM can now publish exact KV block residency events, while LMCache deployments can expose availability and occupancy. Using those signals can improve reuse, but blocking network calls or trusting stale telemetry on the request path would weaken gateway latency and availability.

Options considered

1. Keep only the engine-independent approximate prefix trie.
2. Query cache state synchronously during every routing decision.
3. Consume telemetry in background tasks and expose bounded in-memory scorers.

Decision

Adopt option 3 as two opt-in strategies behind the existing balancer interface:

• precise_cache_aware subscribes to the supported vLLM V1 three-frame ZMQ msgpack KV-event protocol, derives stable token-block identities, and scores the resident leading fraction of tokenizer-aligned request blocks.
• lmcache_aware polls a rolter-defined HTTP JSON signal containing occupancy and cache_available, then scores available targets as 1 - occupancy.

Both sources update bounded process-local state outside the request path. Missing request token IDs, malformed input, disconnects, stale data, and vLLM sequence gaps produce a neutral score and preserve least-load fallback. A sequence gap clears the precise index until an explicit all-blocks-cleared boundary re-establishes trust.

Consequences

• Exact vLLM residency and LMCache capacity can influence routing without request- path network I/O.
• Approximate cache_aware remains available and engine-independent.
• vLLM exact scoring requires token IDs from the same tokenizer and a compatible event protocol; version assumptions are part of the operator documentation.
• LMCache integration depends only on the small documented controller contract, not an unstable internal LMCache API.
• Telemetry state is local to each gateway replica and intentionally disposable; restart, staleness, or desynchronization temporarily reduces routing quality but does not prevent requests.

Uniform config-vs-DB tiering for models, providers, and provider groups

Status: Accepted · Date: 20 Jul 2026 · Issues: #306 Relates: ADR-0017 (provider/model addressing), ADR-0018 (granular CRUD)

Context

rolter bootstraps from a TOML config and (optionally) a Postgres store, merged at startup via MergedConfigStore. Today the three routable entity types treat the config/DB boundary inconsistently:

Three problems:

1. Providers cannot be seeded-then-edited. A [[providers]] entry is permanently config-owned and rejected by CRUD (require_not_config_owned), so an operator who wants to bootstrap a provider from config but later edit its credentials/base_url through the UI has no path.
2. Provider groups have no DB tier at all — they exist only as read-only config (ADR-0017 addendum / #571), so they cannot be created via the API or UI.
3. The model shape is idiosyncratic — top-level [[routes]] and a parallel [models.readonly] both mean “immutable config route”, which is redundant and does not generalize to providers/groups.

The desired behaviour is uniform across all three: an operator can declare an entry as readonly (immutable, served straight from config), as a default (seeded into the DB once at startup, then owned and editable via API/UI), or create it purely through the API/UI (a DB row with no config presence).

Decision

Adopt one uniform two-tier config wrapper for every routable entity type — models, providers, and provider groups — layered over the existing DB (CRUD) tier:

[models]
readonly = [ /* immutable, config-owned, rejected by CRUD */ ]
default  = [ /* seeded into the default project once, then DB-owned */ ]

[providers]
readonly = [ ... ]
default  = [ ... ]

[provider_groups]
readonly = [ ... ]
default  = [ ... ]

Semantics, identical for each entity type:

• readonly — merged into the effective running config and into the gateway snapshot directly; tracked in ConfigOwned; CRUD create/update/delete against a name/slug in this set is rejected. Immutable for the process lifetime.
• default — seeded into the bootstrap default/default/default tenancy exactly once (idempotent: an existing row by natural key is never overwritten on restart), then it is an ordinary DB row — editable and deletable via API/UI, and not config-owned. Mirrors today’s seed_default_models.
• DB (API/UI) — pure store rows with no config presence, already the CRUD path.

Resolution precedence when the same natural key (routes.model, providers.slug, group slug) appears in more than one tier: readonly wins (it is immutable and must never be shadowed by a DB row), then the DB row, then a default that has not yet been claimed as a DB row. Seeding a default whose key already exists as readonly is a config error, surfaced at load time (as models.default already does against readonly routes).

Migration & back-compat

• The current top-level [[routes]] and [[providers]] arrays are kept as deprecated aliases for their readonly tier, so existing configs and the bootstrap rolter.example.toml keep working unchanged. New docs steer operators to [models.readonly] / [providers.readonly].
• [[provider_groups]] (introduced in the ADR-0017 addendum / #572, before this ADR) becomes the provider_groups.readonly alias; the config-only groups slice is forward-compatible with this decision.
• Slugs remain the stable identity across tiers (ADR-0017): a default provider or group keeps its slug when it becomes a DB row, so its provider-slug/model / group-slug/model address is stable across the config→DB transition.

Consequences

• One mental model and one code path shape (readonly merge + default seed + ConfigOwned guard) for all three entity types; the provider-groups and provider work reuses the model tiering rather than inventing per-entity rules.
• Providers gain a seed-then-edit story; provider groups gain a full DB lifecycle.
• New store tables/repos are required for provider groups (provider_groups + membership), plus seed functions for providers.default and provider_groups.default, plus group CRUD — tracked as separate PRs (see below).
• The deprecated top-level arrays add a small amount of parse-time aliasing to maintain until a future breaking release removes them.

Follow-up implementation issues

1. core: uniform tier wrapper for providers + provider groups (readonly/default), with top-level [[routes]]/[[providers]]/[[provider_groups]] kept as deprecated readonly aliases; load-time validation that a default never collides with a readonly key.
2. store/control: providers.default seed tier — seed function mirroring seed_default_models; providers seeded this way are DB-owned, not config-owned.
3. store: provider_groups + membership tables, repo, and MergedConfigStore wiring so DB-defined groups reach the gateway snapshot.
4. control: provider-group CRUD (create/update/delete, slug validation shared with providers, 409+suggestion on slug conflict) + provider_groups.default seed.
5. ui: provider-group management screens (create/edit/membership/strategy), surfaced alongside providers and models. (separate issue — not in the initial backend PRs)

Propagating access-profile model policy to the data plane

Status: Accepted · Date: 4 Aug 2026 · Issues: #791, #534 Relates: ADR-0022 (config-vs-DB tiering)

Context

#534 added org-scoped access profiles that can carry a model and route allow/deny policy. The control plane resolves the profiles a caller holds, merges their policies, and reports the result on GET /api/v1/rbac/effective — but nothing enforced it on the gateway, so the policy described access it did not actually control.

Enforcement was deferred because the obvious mechanism does not work. The RBAC tables deliberately carry no bump_config_version() trigger: control-plane authorization is evaluated per request against the live database, so there is nothing to propagate, and bumping the version on every role edit would wake the whole gateway fleet for a change it could not observe. Migration 0058 records that reasoning and states the tables must never grow such a trigger.

The gateway, meanwhile, never sees a user. It authenticates a virtual key and knows only what the snapshot told it about that key. A policy is a property of a person; a request carries a credential. Something has to bridge the two.

Decision

Resolve the merged policy per key owner when the snapshot is built, carry it on the virtual-key record, and enforce it on the gateway’s model and route selection.

Concretely:

• rolter_core::ModelPolicy becomes the single definition of the policy shape, its merge rule, and its allow/deny matching. The control plane’s MergedPolicy is now an alias for it.
• load_virtual_keys resolves each key’s created_by user to a merged policy in one batched query for the whole key set, and publishes it on VirtualKeyRecord::access_policy. None means unrestricted.
• KeyMeta::model_permitted and KeyMeta::route_permitted gate the request path. The key’s own model allow-list and the owner’s policy must both permit a model: they are separate grants and neither may widen the other.
• Migration 0060 adds bump_config_version() triggers to exactly the four tables that feed this resolution: access_profile_policies, access_profile_assignments, access_profiles, and memberships.

Why not resolve at key-mint time

#791 floated stamping the policy onto the key when it is minted. That is unsound: a profile edited after the mint would never reach keys already issued, so revoking a model would not revoke it. A policy change that fails to restrict is a security bug, and the failure is silent — the control plane would report the new policy on /rbac/effective while the gateway kept honouring the old one. Snapshot resolution has the opposite failure mode: the worst case is a bounded staleness window that the existing config-version machinery already closes.

Why this does not contradict migration 0058

0058 says these tables must never bump the config version because the data plane does not consume them. This ADR changes that premise rather than overruling the rule: the gateway now consumes them, and the repository’s standing convention is that any table the data plane reads must bump the version inside the write transaction. 0060 therefore applies the rule to the tables whose premise changed, and only those.

custom_roles and custom_role_grants stay untriggered. They decide control-plane authorization, which is still evaluated live per request, so 0058’s reasoning holds for them unchanged.

memberships is included because it is the second path by which a profile reaches a user: a profile assigned to a team applies to everyone in it, so adding a member changes that user’s effective policy without any row in the profile tables changing. Membership is not a hot table, the trigger is statement-level, and the gateway coalesces on the version number — so even a bulk SCIM sync costs one snapshot poll rather than one per member.

Consequences

• The policy /rbac/effective reports is now the policy that is enforced. One implementation of “deny beats allow” serves the control plane, the store and the gateway, so the reported and enforced answers cannot drift.
• A deployment with no access profiles is bit-for-bit unaffected: every key carries None, and model_permitted reduces to the pre-existing model_allowed check.
• Editing a profile or a team membership now bumps the config version, so those edits wake the gateway fleet. This is the intended cost of making the policy observable; it is bounded to four tables, none of them on a request path.
• Enforcement is keyed on created_by. A key with no owner — admin-created and config-defined keys — carries no policy, because there is no person whose profiles could apply. Restricting those remains the key’s own model list.
• The staleness window is the snapshot poll interval, as it is for every other DB-backed config. A revocation is not instantaneous; it is as fast as a provider or route change.

Dashboard UX telemetry as a structural-only event stream

Status: Accepted · Date: 6 Aug 2026 · Issues: #805 Relates: ADR-0023 (access-policy propagation)

Context

Browser tracing (#807) tells us how long a dashboard screen took and whether it threw. It does not tell us whether the screen was usable: which screens are slow to become interactive, where people back out, which forms get abandoned, which error and empty states are actually reached. Those are the questions that decide what to build next, and spans cannot answer them.

Collecting that is ordinarily where a product-analytics vendor is added. For a self-hosted gateway that sits in the request path of its operators’ LLM traffic, a second data processor is a hard sell: a different retention policy, a different redaction story, a different jurisdiction, and an outbound connection operators did not ask for. Several rolter deployments are air-gapped, where it simply does not work at all.

There is also a real hazard specific to this kind of telemetry. UX event streams are where free text leaks. An “abandoned form” event is one careless field away from carrying what was typed into the form, and the dashboard’s forms hold provider API keys, virtual keys and prompts.

Decision

Collect UX events in-house, into a ui_events ClickHouse table beside request_logs, and make the schema the privacy guarantee.

Structural only. Every column is a key, an enum, a duration or an id. There is deliberately no column a form value, prompt or free-text body could be written into. target names the form, control or validation rule; it never carries what failed the rule. A future column that could hold free text is a change to this ADR, not a routine migration.

This is chosen over the usual alternative — accept arbitrary payloads and redact on write — because redaction is a policy that has to keep being correct as the dashboard grows, while a schema with nowhere to put a value is correct by construction. logging.payload_capture already exists for the case where someone genuinely wants raw bodies, and it is off by default; this stream is deliberately not that.

Server-side attribution. user_id is taken from the authenticated session and a client-supplied one is ignored, so a caller cannot file events against another user.

Authentication, not authorization. The ingest endpoint is guarded by CurrentUser alone, following me.rs, and has no row in the capability table. The RBAC model holds that a viewer writes nothing — rbac_matrix.rs asserts exactly that — and a ui_event:create capability granted to any authenticated caller breaks the invariant. A viewer who could not file their own screen views would be absent from every funnel, which makes the data quietly wrong rather than absent. Keeping this off the capability table states plainly that it is not an operator surface.

On by default, with two off switches. Absent clickhouse_url the stream is inert. logging.ui_events = false is a deployment-level opt-out. It defaults on — unlike payload_capture — precisely because the schema cannot carry sensitive content, so the usual reason to make telemetry opt-in does not apply.

Consequences

Operators get usability data with no new vendor, no new egress and no new retention policy: same deployment, same TTL, same 90-day partitioning as request logs. Because events carry trace_id, a slow screen and the gateway request behind it are one join rather than two systems.

The cost is expressiveness. Any question that needs a value rather than a key is unanswerable by design, and some genuinely useful analyses are foreclosed. That is the intended trade: the class of incident this prevents — dashboard secrets in an analytics table — is worse than the analyses it forbids.

Adding a screen means adding a stable screen key rather than letting a URL through. screen, target and from_screen are LowCardinality columns capped at 96 characters, which a real URL does not fit; sending one would degrade the table, so the bound fails the write instead.

Per-tenant routing of this stream is out of scope and tracked separately (#812).

Events as logs, not span events

Status: Accepted · Date: 6 Aug 2026 · Issues: #814, #808, #809, #805 Relates: ADR-0024 (dashboard UX telemetry)

Context

OpenTelemetry is deprecating the Span Events API — Span.AddEvent, Span.RecordException and friends. Maintaining two overlapping ways to emit a correlated event (span events via the Tracing API, log events via the Logs API) produced split guidance for instrumentation authors and a slower-evolving event model. The replacement is events as logs: emitted through the Logs API, named, and correlated to the active span through context rather than attached to it. Semantic-convention authors are now explicitly told to document events as log-based events.

Nothing is being removed soon. The deprecation is phased, existing data stays valid in the OTLP trace model, and compatibility layers are a stated priority. This is direction-setting, not an outage.

What rolter emits today

A grep for the deprecated API finds nothing:

$ grep -rn "add_event\|record_exception" --include='*.rs' crates/
(no matches)

That result is misleading, and reading the bridge is what the audit actually required. rolter instruments through tracing + tracing-opentelemetry, and that layer’s on_event turns every tracing event fired inside an active span into an OTel span event (tracing-opentelemetry-0.33.0/src/layer.rs:1468, span.add_event(...); the pre-export path pushes onto builder.events identically). An ERROR-level event additionally sets the span status to error.

So rolter has zero explicit uses of the deprecated API and is, in practice, an exclusive user of it: the ~57 tracing event macros across rolter-gateway, rolter-proxy and rolter-core become span events whenever they fire inside one of the pipeline spans. The migration surface is the bridge, not our call sites.

Why the ordering between #808 and #809 changes

The GenAI semantic conventions (#808) are precisely the area moving to log-based events. Implementing them against span events would be building onto a deprecated API on the day it is written.

Emitting log-based events requires exporting logs at all, and rolter does not: there is no OTLP logs exporter in the workspace and signoz_logs is empty by construction. #809 therefore stops being an independent nice-to-have and becomes a prerequisite.

Decision

Log-based events are rolter’s event model. New instrumentation targets the Logs API; span events are legacy, produced only incidentally by the tracing bridge.

Concretely:

1. No new explicit span events. add_event and record_exception are not to be called directly. Nothing calls them today and nothing should start.
2. #809 (OTLP log export) sequences before #808 (GenAI conventions). The conventions #808 implements are specified as log-based events, so the log pipeline has to exist before they can be emitted correctly. #808’s acceptance criteria adopt log-based events as the target.
3. tracing stays. The bridge is idiomatic Rust and the ecosystem’s own integration, not a bespoke house wrapper — the same distinction drawn in the #815 wrapping audit. It is where the span-event mapping lives, so it is also the single place a future migration happens: when tracing-opentelemetry routes events through the Logs API, rolter follows by upgrading, without touching 57 call sites.
4. Correlation is the acceptance bar. A log-based event is only a replacement for a span event if it carries trace_id and span_id, which is already called out as most of the value in #809. An exported log that cannot be joined back to its span is a regression against what the bridge does today.
5. Content stays off. The GenAI opt-in content attributes (gen_ai.input.messages / gen_ai.output.messages) remain disabled whether they are carried as span events or as log records. Moving event model does not move the privacy line.

Consequences

• The migration is a dependency upgrade rather than a code sweep, because rolter never adopted the deprecated API directly. This is the payoff of having gone through tracing instead of calling OTel by hand.
• #809 gains a second justification beyond “logs are missing”: it unblocks #808. Its trace_id/span_id correlation requirement is now load-bearing rather than a nicety, and should be treated as a blocking acceptance criterion.
• Until #809 lands, error detail continues to reach backends as span events via the bridge. That is fine — the deprecation is phased and the data stays valid — but it means the GenAI work should not start early and improvise its own event channel.
• Anything that wants a new event type before #809 has no correct home. The answer is to wait rather than to add a span event that will have to be removed.

Alternatives considered

Implement #808 now against span events, migrate later. Rejected. The conventions are specified as log-based events, so this builds a second migration into work that has not shipped yet, on the exact attributes most likely to be re-specified.

Emit both span events and log events during a transition. Rejected. Duplicate correlated events on both signals is the split-guidance problem OpenTelemetry is deprecating span events to escape, and it doubles egress for data that already joins on trace_id.

Drop tracing and call the OTel Logs API directly. Rejected. It is a large rewrite of every call site to avoid a bridge that will itself be updated upstream, and it contradicts the #815 finding that the ecosystem bridge is not the wrapping the “don’t wrap” guidance targets.

Client control over telemetry: a kill switch now, collector-routed tenant destinations later

Status: Accepted · Date: 6 Aug 2026 · Issues: #812, #805, #809 Relates: ADR-0019 (egress proxy pools), ADR-0025 (events as logs)

Context

Telemetry is currently an operator decision baked into the deployment: OTEL_EXPORTER_OTLP_ENDPOINT for the backend exporter and ROLTER_UI_OTEL_ENDPOINT for the dashboard. A client of the gateway has no say in either — they cannot turn tracing off for their own traffic, and they cannot have their spans delivered to a backend they own.

#812 raises two things of very different size, and filed them together so the relationship is on the record. This ADR splits them.

The switch. “Off” is implicit today: you achieve it by leaving an endpoint unset. That works but is not discoverable, does not survive somebody setting the endpoint for one signal, and gives an operator nothing to point at in a security review.

Per-tenant destinations. rolter is multi-tenant (org / team / project, virtual keys, RBAC) while telemetry is single-destination, which does not match. A team running rolter as shared infrastructure wants its spans in its backend; some tenants want no export at all for data-residency reasons; a customer debugging their own integration should not have to ask the operator to fetch traces. This half raises questions — scope, fan-out, egress safety, cardinality — that are design work rather than implementation.

Decision

Ship the kill switch now as an environment variable. Do not build per-tenant fan-out in rolter; route it in the collector when it is built.

1. ROLTER_TELEMETRY_ENABLED

One switch covering traces, metrics, the dashboard’s browser tracing, and logs once #809 lands.

• It can only subtract. When false, no exporter is built regardless of which OTEL_* endpoints are set. It is never a second way to turn export on.
• Unset means enabled, so no existing deployment changes behaviour. That is not a weaker default than it sounds: with no endpoint configured nothing is exported anyway, so the effective default remains “exports nothing”.
• An unrecognized value leaves export on. A typo must not silently blind a deployment; only the explicit falsy spellings (0, false, no, off) disable it.
• Environment-only, with no config-file key, despite #812 asking for “the matching config key”. telemetry::init installs the subscriber in main before any config file is read — the gateway has to be able to log a config parse failure — so a config-file switch could not gate trace export at all. Honouring one only for the signals initialized later would mean the same key meant different things depending on which signal you asked about, which is worse than not having it. The OTEL_* contract this composes with is environment-based for the same reason.

2. Per-tenant destinations belong in the collector

When the second half is built, it is configuration of an OpenTelemetry Collector that sits in front of the tenants’ backends, not a fan-out inside rolter.

• Scope: org. It is the easiest to administer and the coarsest unit anyone actually asks about (“my team’s spans in my team’s Honeycomb”). Virtual key is finer and matches how traffic is attributed, but a destination per key is a cardinality problem with no matching demand. rolter’s job is to stamp the attribute; the routing key can be refined later without changing where fan-out happens.
• Fan-out: the collector’s routing processor. Multiple SDK exporters in-process does not scale with tenant count — each is a connection, a queue and a retry buffer, all inside the request-serving process. rolter emits one OTLP stream to one collector with a resource attribute naming the tenant, and the collector’s routing processor sends each tenant’s data onward. This keeps exactly one egress path in the gateway and makes per-tenant destinations an operator’s configuration change rather than a rolter deploy.
• Egress safety follows from that. A tenant-supplied endpoint is an arbitrary URL, and the deciding argument against in-process fan-out is that it would make the gateway POST to one. Terminating tenant endpoints in the collector keeps that SSRF surface out of the data plane entirely. If a tenant-supplied endpoint is ever accepted through the API, it must go through the same EgressPolicy that provider api_base values do (ADR-0019), and its credentials through the encrypted-secret path — never plaintext config.
• Opt-out is the switch, one level down. A tenant wanting no export at all is a routing rule that drops, not a second mechanism.

Consequences

• An operator gets a single, documentable, reviewable “no telemetry leaves this deployment” today, without waiting on the design work.
• The gateway keeps exactly one exporter and one egress path no matter how many tenants exist. Cardinality and buffer growth become the collector’s problem, where they are a sizing question rather than a data-plane risk.
• The cost is that per-tenant destinations require running a collector. That is already the recommended topology (docs/architecture/observability.md), so it is not a new dependency for anyone following it — but a deployment exporting straight to a vendor endpoint would have to add one.
• Nothing here is blocked on #809, but when OTLP log export lands it inherits both decisions automatically: the switch covers it, and its records carry the same resource attributes the routing processor keys on. That is why the switch is worded per-deployment rather than per-signal.
• ROLTER_TELEMETRY_ENABLED is now part of the public configuration surface and cannot quietly change meaning. In particular it must stay “can only subtract” — making it required-true would break every existing deployment on upgrade.

Alternatives considered

A config-file [telemetry] enabled key. Rejected on initialization order, as above. Worth restating because it is the obvious request and the reason it does not work is not obvious.

Per-signal switches (ROLTER_TRACES_ENABLED, …). Rejected. The problem #812 names is that “off” is currently spread across several variables; adding more variables reproduces it. Per-signal control already exists through the signal-specific OTEL_*_ENDPOINT variables.

In-process per-tenant exporters. Rejected on all three of scaling, egress safety and cardinality, as above.

Reflecting a tenant-supplied endpoint straight from a virtual key. Rejected outright: it turns every key holder into someone who can aim the gateway’s egress, which is the SSRF primitive ADR-0019 exists to prevent.

End-to-end test harness: Python/uv project driving a black-box stack

Status: Accepted · Date: 21 Jul 2026 · Issues: #613, #614

Context

Per-crate unit tests and the two narrow suites we already have (#414 in-process integration, #449 compose smoke) never exercise rolter as a whole: a freshly booted stack, multiple tenants, live RBAC changes, a fleet of upstream engines, and balancing across clusters. #613 needs a broad governance/routing E2E suite, and #614 is the harness every other scenario (RBAC matrix, balancing, key lifecycle, security) builds on.

Two decisions had to be made: what drives the stack, and how the driver code is packaged.

Options considered

1. Rust in-process integration test — spin control+gateway in-process against testcontainers. Fast and CI-native, but not a real deployment: it bypasses the HTTP edge, the compose network, and the container boundaries where isolation and auth actually matter.
2. Bash script driving docker-compose — closest to “run fresh rolter”, but bash grows unmaintainable for a table-driven RBAC matrix with JSON assertions.
3. Python driver over docker-compose — real HTTP against a real composed stack, with a proper test framework for the matrix.

For packaging the Python driver:

• PEP 723 inline script metadata (uv add --script) — deps live in a single file header, uv run file.py auto-installs. Ideal for one-file scripts.
• A uv project (pyproject.toml + uv.lock) — deps and a lockfile for a multi-file package.

Decision

Adopt option 3: a Python driver, latest CPython managed by uv, hitting the real HTTP APIs of a stack brought up by docker-compose (postgres, redis, clickhouse, control, gateway, and N llm-d-inference-sim fake-vLLM engines). No in-process shortcuts — the tests see exactly what an operator/tenant sees.

Package it as a uv project under integration/e2e/, not inline PEP 723 script metadata. The harness is a pytest package: a shared helper/client library plus multiple scenario modules (test_rbac.py, test_balancing.py, test_keys.py, test_security.py) sharing fixtures. Inline metadata is per-single-file and cannot express cross-module fixtures or a lockfile, so it is the wrong grain here. Inline PEP 723 remains fine for any standalone one-off helper run by hand.

The harness project is kept separate from the root maturin pyproject.toml: that one packages the shipping wheel at requires-python >=3.9; the test harness has no such floor and pins the latest stable CPython via uv python pin.

RBAC only enforces when ROLTER_ADMIN_TOKEN is set (otherwise the control plane runs in open/superadmin mode, per #454/ROL-250), so the harness compose sets the admin token and shares ROLTER_KEY_PEPPER/ROLTER_SESSION_PEPPER between control and gateway. Bootstrap uses the admin token as superadmin; per-role scenarios create local accounts and drive them via session tokens.

The suite is heavy, so it is gated (manual dispatch + nightly), never on the default per-PR gate.

Consequences

• One harness backs every #613 sub-issue; scenarios are Python modules, not new bespoke rigs.
• Tests run against the real container edge, so cross-tenant isolation, auth, and balancing are exercised where they actually live.
• Fake engines only (llm-d-inference-sim) — offline/air-gapped safe, no secrets, no model downloads, deterministic.
• A lockfile makes the harness reproducible; the cost is a second Python project in the tree, deliberately isolated from the wheel’s packaging metadata.
• Full stack-boot verification requires Docker and building the rolter images, so it lives in CI (dispatch/nightly), not the fast PR path.

Disaggregated prefill/decode routing belongs to the engine, not the gateway

Status: Accepted · Date: 11 Aug 2026 · Issues: #853 Relates: ADR-0014 (protocol translation boundary), ADR-0007 (cache-aware balancing)

Context

#853 carried three unclaimed ideas from llm-d. The second — routing by inference phase to specialised pools — was flagged as “the largest of the three and the one most coupled to how the fleet is deployed”, with the note that it may not fit rolter’s position in the stack and that the useful output in that case is a written “no, because”. This is that note.

Disaggregated serving splits inference in two. Prefill processes the prompt, is compute-bound, and produces a KV cache. Decode generates tokens, is memory-bandwidth-bound, and consumes that KV cache. Running them on separately sized pools improves utilisation, because the two phases stress different resources and otherwise contend on the same accelerator.

Making that work requires, per request: selecting a prefill worker, running prefill there, transferring the resulting KV cache to a decode worker, and then streaming generated tokens from the decode worker.

What rolter is

ADR-0014 fixed the boundary: rolter-gateway owns authentication, tenancy, target selection, tracing, metrics and accounting; rolter-proxy owns the wire protocol. rolter forwards one client HTTP request to one upstream and streams the response back. Its routing inputs come from gateway-owned configuration, the client request, and bounded telemetry adapters; it does not participate in an engine’s multi-stage request lifecycle.

Options considered

Option 1 — Implement P/D disaggregation in rolter

The gateway selects a prefill target and a decode target, drives both, and coordinates the KV handoff between them.

Option 2 — Treat a disaggregated fleet as one upstream

The engine-side coordinator owns phase routing and the KV transfer. rolter balances across the fleet’s entry points exactly as it does for any other pool, and does not know a request was split.

Option 3 — Hybrid: rolter routes phases, engine transfers KV

rolter picks both workers and issues two calls, but the KV cache moves over the engine’s own sidechannel rather than through rolter.

Comparison

Decision

Option 2. rolter will not implement disaggregated prefill/decode routing. A P/D-disaggregated deployment is a single upstream fleet from rolter’s point of view, and its own coordinator owns phase selection and the KV handoff.

Rationale

The KV handoff is the central mechanism. The tensors do not travel in an OpenAI or Anthropic response; engines move them through connector-selected transports such as NIXL/UCX. A gateway that carried the tensors itself would therefore have to join that engine-specific data path.

Coordination can still be initiated over HTTP. For example, vLLM’s experimental disaggregated-prefill API accepts kv_transfer_params on separate prefill and decode requests while its connector moves the KV data out of band. That makes option 3 technically possible, but does not create a stable, engine-independent contract: rolter would still own engine-specific fields, compatible-worker topology, two-call orchestration and state across the handoff. That is precisely the version coupling and lifecycle ownership ADR-0014 was written to prevent.

It would also be redundant. vLLM ships disaggregated serving with a connector API, and llm-d’s router is co-designed with its engine and scheduler — it can assume things about worker topology and KV placement that a general-purpose gateway in front of heterogeneous providers cannot. rolter’s value is that it sits in front of many engines and hosted APIs; a feature that only works for one engine, at one version, with one transport configured, is not a good trade against that.

The distinguishing question is: can rolter consume the input through a stable, engine-independent contract without joining the engine’s request lifecycle? The answer is yes for prefix-cache affinity, queue depth, KV-event residency, LMCache occupancy and LoRA-adapter affinity, which fit the bounded scorer and telemetry interfaces behind ADR-0007 and ADR-0021. P/D placement does not: the placement decision and transfer metadata are part of one engine-specific, multi-call lifecycle.

Consequences

Benefits:

• the ADR-0014 boundary is preserved: no engine-specific KV connector or phase-control extension in the data plane, and no per-request state held across an upstream handoff;
• disaggregated fleets are usable with rolter today, with no code, for any engine that implements disaggregation — including engines that do not exist yet;
• rolter does not inherit the failure modes of a mid-request handoff, which has no clean recovery once prefill has completed and decode is unreachable.

Drawbacks and risks:

• rolter cannot improve on the engine’s phase placement, and cannot report on it beyond whatever the fleet exposes over its metrics endpoint;
• if a stable, engine-independent phase-placement contract emerges on the OpenAI or Anthropic surface, this decision should be revisited, because the portability and lifecycle objections would no longer hold.

System impact:

None. This is a decision not to build something; no code changes.

Related records

• ADR-0007 — Approximate cache-aware balancing behind a pluggable trait
• ADR-0014 — Extensible API protocol translation
• ADR-0021 — External cache telemetry for routing

Open questions

• Should rolter document a recommended topology for putting a disaggregated fleet behind it (which entry point to target, and how to configure health checks against a coordinator rather than a worker)? That is documentation rather than a routing feature, and would be useful independently of this decision.

API surface

rolter speaks the OpenAI and Anthropic HTTP APIs so existing SDKs work unchanged — point them at the gateway base URL and use a rolter virtual key.

Authentication

• OpenAI-style: Authorization: Bearer <virtual-key>
• Anthropic-style: x-api-key: <virtual-key>

When no virtual keys are configured the gateway runs open (useful for local dev).

Endpoints (v1)

Realtime WebSocket

Connect with the usual gateway bearer key and the public route model as a query parameter:

wss://gateway.example.com/v1/realtime?model=gpt-realtime

rolter authenticates and selects an upstream before accepting the client upgrade, then pins that upstream and its selected provider key for the session. Text, binary audio and WebSocket control frames are relayed in both directions without application-level buffering. If the upstream drops, the client must reconnect; rolter does not fail a live session over to another target because replaying audio or tool events is unsafe.

The WebSocket-first implementation supports the OpenAI Realtime event stream, including session.update, input_audio_buffer.*, response.*, and function-call events. WebRTC/browser ephemeral-token handoff is not exposed by the gateway yet.

MCP gateway

/mcp/{server} and any path beneath it proxy an organization-scoped MCP server registered in the control plane. The caller supplies a normal rolter virtual key. Unlike LLM routes, MCP calls require a database-backed key minted by a user: that owner must have a live, unrevoked and unexpired OAuth session for the named server, and its scopes must cover the server’s required_scopes. The gateway fails closed before connecting when any owner, server or scope check fails.

The caller’s virtual key is never forwarded. Rolter replaces it with the session’s OAuth bearer token and preserves end-to-end MCP headers, body, query, status and response stream. Streamable HTTP and legacy SSE registrations use this path; stdio and WebSocket registrations currently return mcp_transport_unsupported rather than bypassing authorization.

Routing

The model field in the body selects a route. The route’s strategy picks a target; rolter rewrites model to the target’s upstream model id and forwards with the provider’s credentials. Session affinity uses x-session-id when present.

When the selected upstream speaks the other chat protocol, rolter translates OpenAI Chat Completions and Anthropic Messages in both directions. Translation includes system/developer instructions, sampling and stop parameters, function tools and tool results, token usage, finish reasons, and live SSE events. Image and document inputs retain URL, base64 media type/data, and file references. Blocks with no equivalent in the target protocol (for example OpenAI input audio sent to an Anthropic Messages upstream) are preserved as opaque content blocks; the target may reject them rather than rolter silently dropping data.

The Gemini dialects — generateContent and Interactions — are the exception: their wire formats are typed part unions with no opaque carrier, so an unrecognized part cannot be passed through. Rather than drop it, rolter fails the request closed with 400 unsupported_content_part, naming the part type and the upstream dialect:

{
  "error": {
    "message": "the interactions upstream has no equivalent for content part type 'input_audio'; remove it or route this model to a provider that accepts it",
    "type": "invalid_request_error",
    "code": "unsupported_content_part",
    "param": "messages"
  }
}

This applies to every client dialect (Chat Completions, Messages, Responses) and covers new part types a client SDK starts sending. Only text/input_text and image_url/input_image have Gemini equivalents today; route models that need other modalities to a provider whose dialect carries them.

OpenAI Responses

POST /v1/responses is routed by its required model field. Native OpenAI providers receive the request and SSE events unchanged. For Chat Completions or Anthropic Messages upstreams, rolter translates the common text, multimodal, function-tool, tool-result, sampling, and usage fields in both directions and emits Responses-shaped events to the caller. Responses-only features without a wire equivalent (for example background, store, previous_response_id, and provider-specific reasoning controls) are not forwarded to those older surfaces; use a native Responses provider when those features are required.

For native OpenAI providers, rolter records the selected provider, target, upstream model, provider credential fingerprint, and native response ID after a successful creation. GET/DELETE /v1/responses/{id}, cancellation, and input-item retrieval are then pinned to that record. Records are isolated by virtual key, retained for 24 hours by default, bounded to 100,000 entries per gateway process, and removed after a successful delete. Configure these limits with [responses] registry_ttl_secs and registry_max_entries; setting either to 0 disables registration.

The registry is process-local. Multi-replica deployments must keep lifecycle requests sticky to the gateway replica that accepted creation; records do not survive a gateway restart. Route changes do not retarget an existing response. If its provider is removed, its provider kind changes, or its credential is rotated away, the record becomes unavailable. Unknown, expired, deleted, cross-key, and unavailable records all return the same 404 response_not_found error so route ownership is not leaked.

Responses translated through Chat Completions or Anthropic Messages retain an ownership record but expose no lifecycle capabilities, because those upstream contracts do not retain an OpenAI Responses resource. Their lifecycle calls return 501 response_lifecycle_unsupported. Compaction and input-token counting remain unsupported for all providers.

Examples

# openai chat (streaming)
curl -N http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-rolter-dev" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o","stream":true,"messages":[{"role":"user","content":"hi"}]}'

# anthropic messages
curl http://localhost:4000/v1/messages \
  -H "x-api-key: sk-rolter-dev" \
  -H "Content-Type: application/json" \
  -d '{"model":"claude","max_tokens":256,"messages":[{"role":"user","content":"hi"}]}'

# openai embeddings
curl http://localhost:4000/v1/embeddings \
  -H "Authorization: Bearer sk-rolter-dev" \
  -H "Content-Type: application/json" \
  -d '{"model":"text-embedding-3-small","input":["hello","world"]}'

# self-hosted vllm pool via a public model name
curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-rolter-dev" \
  -H "x-session-id: user-123" \
  -H "Content-Type: application/json" \
  -d '{"model":"llama","messages":[{"role":"user","content":"hi"}]}'

Multipart audio (/v1/audio/transcriptions, /v1/audio/translations) forwards the upload verbatim and routes on the model form field; the route target’s upstream model name is not rewritten into the multipart body, and variant routing / per-model param defaults (JSON-only) do not apply.

Development setup

Prerequisites

• Rust (stable) via rustup — the workspace pins the toolchain in rust-toolchain.toml.
• Bun for the UI — curl -fsSL https://bun.sh/install | bash.
• prek for repository Git hooks — install with uv tool install prek or brew install prek.
• Docker + Compose for Postgres/Redis/ClickHouse.
• uv (optional) for the PyPI-wheel install path and tooling.

Clone & build

git clone https://github.com/rolter-ai/rolter.git
cd rolter
cargo build --workspace
cargo nextest run --workspace   # or `cargo test --workspace`; install: cargo install cargo-nextest

Run the gateway (no external services needed)

cp rolter.example.toml rolter.toml
export OPENAI_API_KEY=sk-...        # referenced by api_key_env in the config
cargo run -p rolter-gateway -- --config rolter.toml
# -> http://localhost:4000  (/healthz, /metrics, /v1/*)

Run the control plane + UI

cargo run -p rolter-control          # http://localhost:4001
cd ui && bun install && bun run dev  # http://localhost:3000 (proxies /api -> :4001)

Run the full stack

docker compose -f docker/docker-compose.yml up -d                 # postgres, redis, clickhouse, gateway, control

Handy tasks

just wraps the common commands:

just build | just test | just fmt | just lint
just gateway | just control | just ui-dev | just up

Build these docs

This book is mdBook. Diagrams are ```mermaid fences rendered by the mdbook-mermaid preprocessor, so both tools have to be on PATH — with mdbook alone the build still succeeds and every diagram silently ships as a plain code block.

cargo install mdbook mdbook-mermaid --locked
just docs         # build to docs/book/ (gitignored)
just docs-serve   # live-reloading preview on http://localhost:3001

The mermaid runtime is vendored at docs/mermaid.min.js and docs/mermaid-init.js so the book renders air-gapped. mdbook-mermaid install docs regenerates both, but mermaid-init.js carries a local fix — upstream still binds theme buttons by their pre-0.5 ids (ayu, navy, …), which now throws and leaves diagrams on the light palette after a theme switch — so re-apply it after any refresh. Write labels in mermaid’s own dialect rather than GitHub’s — quote any label containing / (R["/v1/responses"], since [/…] is parallelogram syntax) and break lines with <br/>, never \n.

Before committing

cargo fmt --all
cargo clippy --workspace --all-targets -- -D warnings
cargo nextest run --workspace && cargo test --doc --workspace   # or `just test`
prek install --prepare-hooks
prek run --all-files
prek run --all-files --hook-stage pre-push

The hooks add staged-file hygiene and secret scanning, Conventional Commit validation, Rust/workflow/TOML/spelling checks, workspace tests, dependency policy checks, and UI lint/build checks. Install the system tools used by the project-specific hooks:

brew install actionlint taplo typos-cli
cargo install cargo-nextest cargo-deny

cargo-nextest is recommended but optional for the push hook; it falls back to cargo test. CI remains authoritative for database-backed tests that need ROLTER_TEST_DATABASE_URL.

Testing

Run

Tests run under nextest (the same runner CI uses), plus a separate doc-test pass since nextest does not run doc tests:

cargo nextest run --workspace   # unit + integration tests
cargo test --doc --workspace    # doc tests
cd ui && bun run lint           # ui typecheck

Install the runner once with cargo install cargo-nextest (or see the nextest install docs). just test runs both Rust passes for you. Plain cargo test --workspace still works if you haven’t installed nextest, but CI runs nextest so prefer it locally.

The Ollama Cloud live smoke sends a billed request and is ignored by default:

OLLAMA_API_KEY=... ROLTER_OLLAMA_LIVE_MODEL=gpt-oss:20b \
  cargo test -p rolter-gateway --test ollama_cloud live_smoke -- --ignored

The Gemini Interactions smoke is gated the same way. It exists because Google publishes no full JSON schema for the interactions wire format, so parts of the adapter — the multimodal part field names and some step.delta variants — were inferred from prose docs and only a real request confirms them (#764):

GEMINI_API_KEY=... ROLTER_GEMINI_LIVE_MODEL=gemini-3.6-flash \
  cargo test -p rolter-gateway --test gemini_interactions_live -- --ignored

Both run in CI only from dispatch-gated workflows, never the per-PR gate: quality.yml takes no secrets by design (#734) so dependabot and fork PRs pass exactly the same checks. Assertions in the live suites carry the upstream response body in their failure message — with an inferred field name, the provider’s complaint is the finding, and a bare status-code assertion would throw it away.

Configuring the Gemini smoke

gemini-interactions-smoke.yml needs one secret before it can verify anything:

1. Create a live-providers repository environment (Settings → Environments). Keeping the key there rather than at repository scope means a run against a billed provider is reviewable, not something any workflow can reach for.
2. Add GEMINI_API_KEY to that environment.
3. Dispatch the workflow (gh workflow run gemini-interactions-smoke.yml), optionally with -f model=<id>.

Until the secret exists the workflow fails rather than skipping. A green tick from a run that made no request reads as “the wire format is still confirmed” when nothing was checked — worse than no sweep at all. Pass -f allow_unconfigured=true for a deliberate dry run of the workflow itself.

Each run records the wire shapes it observed into the job summary and uploads the full log as an artifact. A billable run should leave evidence behind: the next question about a field name is then answered by reading the last run rather than by spending another call.

What the suite covers, and why each probe exists:

A content part the dialect cannot carry is rejected at the gateway with 400 unsupported_content_part rather than being dropped (#882), so an unconfirmed part shape fails loudly instead of producing a shortened body.

Test grouping is configured in .config/nextest.toml: the Postgres-backed rolter-store/rolter-control suites share one database and reset the schema per test, so they run in a single-threaded group to avoid clobbering each other.

Layout

• Unit tests live next to the code in #[cfg(test)] mod tests. Current coverage: balancer strategies (round-robin cycling, consistent-hash stability, cache-aware affinity, empty targets), the prefix trie, config parsing, model rewrite, auth checks, and the in-memory store.
• Keep the pure crates (rolter-core, rolter-balancer, rolter-auth) fully unit-testable without I/O.

Strategy as the project grows

• Integration tests for the gateway: spin up the Axum app with a mock upstream (wiremock/httpmock) and assert routing, auth, model rewrite, error mapping and streaming passthrough.
• Property tests (proptest) for the balancer: distribution fairness, affinity invariants.
• DB tests for rolter-store Postgres backend behind a feature, using a disposable container.
• Load tests (oha/k6) against a mock upstream to track added latency and max RPS (see performance.md).

Chaos & resilience contracts

Resilience is asserted in two places, split by what each harness can drive deterministically.

The e2e chaos suite (integration/e2e/tests/test_chaos.py, compose chaos profile) drives a static-config gateway against mock upstreams whose failure mode is fixed by an env var. It covers what is observable purely over the wire: retry and failover on 5xx/429, a clean 5xx when every target is down, the request timeout bound on a slow upstream, the circuit breaker’s OPEN transition (asserted via rolter_breaker_opened_total) and flap degrade/recovery.

The gateway chaos tests (crates/rolter-gateway/tests/chaos.rs) cover the two contracts that need a request pinned at a known point inside the gateway, which a black-box harness can only approximate with sleeps:

• bounded-queue backpressure — with queue.capacity = 1, queue.workers = 1 and backpressure = "error", one request pins the sole worker, exactly one surplus request takes the queue slot, and the rest must come back as 429 with error.code = "queue_full" while rolter_provider_queue_rejections_total advances. Memory is bounded by the queue, not by client burst size.
• graceful SIGTERM drain — a real rolter-gateway child process is sent SIGTERM while a request is pinned upstream. The in-flight request must still return 200, new connections must be refused, and the process must exit 0.

Both use a mock upstream that blocks on a semaphore the test owns, so every step is driven by a signal rather than by elapsed time — there are no sleeps to race. Run them with:

cargo test -p rolter-gateway --test chaos

Benchmarks

Hot-path micro-benchmarks run under criterion. They live in crates/<crate>/benches/ with a [[bench]] harness = false entry per file, and cover the per-request cost that shows up as pure gateway overhead:

just bench                       # cargo bench --workspace
cargo bench -p rolter-balancer   # just the balancer benches
cargo bench -p rolter-balancer --bench pick   # one bench target

Current coverage:

rolter-balancer

• pick — LoadBalancer::pick for every built-in strategy over a ~24-target pool with a populated RouteContext.
• trie — prefix-trie insert (bounded/unbounded, so LRU eviction is measured) and longest_prefix on a warm trie.

rolter-core

• snapshot — the CPU side of config-snapshot generation at 10/100/1000 routes: sanitize_for_snapshot, validate, and the JSON encode. /internal/snapshot is polled by every gateway in the fleet, so this cost is paid fleet-wide on every poll. The encode dominates — ~2.8 ms at 1000 routes against ~150 µs for sanitize — which is why payload size is its own metric (#845).

rolter-gateway

• admission — the two registries every upstream attempt consults before anything else: Breaker::allows and Cooldowns::is_parked, plus the outcome-recording calls beside them. Covers the healthy steady state (no entries recorded), a warm fleet, a tripped/parked target and a 64-model fleet, so the cost is measured where a real gateway actually sits rather than only in the worst case (#1050).

criterion writes HTML reports to target/criterion/. Benches are not run in CI (timings are noisy on shared runners), but cargo clippy --workspace --all-targets -- -D warnings compiles them on every PR, so they cannot silently bit-rot. Use just bench-check (cargo bench --workspace --no-run) to compile them locally without running.

Coverage

Workspace line coverage is measured with cargo llvm-cov:

cargo install cargo-llvm-cov
cargo llvm-cov --workspace --all-features --summary-only   # quick %
cargo llvm-cov --workspace --all-features --html           # browsable report

CI runs coverage in the coverage job of quality.yml and enforces a ratcheting baseline: the committed baseline lives in .github/coverage-baseline.txt, and .github/scripts/coverage-ratchet.sh fails the step if the current percentage drops more than COVERAGE_TOLERANCE points (default 0.5) below it. The job also uploads the lcov.info report as a CI artifact.

Policy (ROL-246):

• New code must not push coverage below baseline − tolerance. If a PR legitimately lowers coverage, edit .github/coverage-baseline.txt in the same PR and explain why.
• When coverage climbs well above the baseline, raise the baseline to lock in the gain (the ratchet only goes up).
• The job is informational (continue-on-error: true) until the baseline is trusted; promote it to blocking by removing that flag on the coverage job.

CI

.github/workflows/ci.yml delegates to the shared quality.yml gate, which runs cargo fmt --check, cargo clippy -D warnings, cargo nextest run --workspace --all-features plus a cargo test --doc pass, the feature matrix, cargo doc (warnings as errors), cargo-deny, gitleaks, the UI lint/build, and a Conventional Commit PR-title check on every push/PR.

Secret scanning

The gitleaks job runs the gitleaks CLI from a digest-pinned container, not gitleaks-action. The action gates org-owned repositories behind a license key, and license secrets are invisible to both dependabot runs (a separate secret store) and fork PRs (no secrets at all), so every such PR failed the job and with it ci-ok. The CLI is free and unrestricted, so quality.yml now takes no secrets and behaves identically for forks, dependabot and direct pushes.

Two passes run with the shared .github/config/gitleaks.toml policy: gitleaks dir over the working tree (everything the commit ships) and, on PRs, gitleaks git --log-opts base..head over the branch history (catches a secret added and then removed inside the same PR). The pinned digest is v8.30.1 — the version prek.toml already uses for the staged-content hook, so local and CI scans agree.

Reproduce a CI run locally:

docker run --rm -v "$PWD:/repo" -w /repo \
  ghcr.io/gitleaks/gitleaks@sha256:c00b6bd0aeb3071cbcb79009cb16a60dd9e0a7c60e2be9ab65d25e6bc8abbb7f \
  dir . --config .github/config/gitleaks.toml --redact --exit-code 1

Storybook play tests

The storybook job builds the static Storybook, serves it, and runs the interaction (play) tests with @storybook/test-runner against a headless chromium. It is a merge gate (#753): a failing play test fails quality, which fails ci-ok. Locally:

cd ui
bunx playwright install --with-deps chromium chromium-headless-shell
bun run build-storybook
python3 -m http.server 6006 --directory storybook-static &
bun run test-storybook --url http://127.0.0.1:6006

ui/package.json pins playwright and playwright-core through overrides. The test-runner declares its own loose playwright range, so without the pin it resolves a different version from @playwright/test and launches a browser revision playwright install never downloaded — the test-runner then fails at launch and the play tests silently stop running (#737). Keep both on one version, and install chromium-headless-shell alongside chromium, since the test-runner launches the shell rather than the full build.

The job is informational (continue-on-error: true) pending the ROL-124 promotion path.

The screen-story harness

A screen story renders the real page component against a stubbed fetch, so it exercises the same query wiring, empty/error branches and editor sheets that ship. ui/src/pages/story-harness.tsx holds the shared pieces — it is not a .stories.tsx file, so Storybook never tries to render it as a screen:

Two traps this encodes. Scope endpoints are matched on the whole pathname: a screen’s own endpoint often contains one of them (/api/v1/projects/{id}/virtual-keys), and a substring match would answer it with the project list. And most screens disable their primary action until the three-request scope chain resolves, so findByRole followed by a click races and throws pointer-events: none — clickWhenEnabled is the fix.

Each screen should carry Loaded, Loading, Empty and an error/forbidden story, one interaction story that opens the primary editor and saves, and at least one story exercising the discard guard. Where a sheet opens pre-filled (budgets seed 100 / 30d), assert the seed too: its dirty flag means “differs from the seed”, not “is non-empty”, and getting that backwards makes an untouched form prompt on every close.

Full-stack compose smoke

The compose-smoke job boots the production-shaped Docker Compose topology (Postgres, Redis, ClickHouse, gateway, control) and exercises it end-to-end. Run it locally with the same script CI uses:

bash docker/smoke/smoke.sh

It layers docker/docker-compose.ci.yml over the base compose file: the overlay mounts docker/smoke/rolter.smoke.toml (a keyless open gateway config) so the built-in fake-llm model answers without any provider secret. The script waits for both /healthz endpoints, checks /v1/models and fake-llm chat (non-streaming + SSE) on the gateway and the postgres-backed /internal/snapshot on the control plane, then always dumps compose logs and runs down -v. It is informational (continue-on-error) until the image-build cost and flake profile are trusted (ROL-245).

Contributing

Thanks for helping build rolter.

Workflow

1. Branch from master: feat/<issue-number>-<short> or fix/<issue-number>-<short>.
2. Make focused changes; add unit tests next to the code.
3. Run cargo fmt --all, cargo clippy --workspace --all-targets -- -D warnings, and the tests (just test, or cargo nextest run --workspace + cargo test --doc --workspace).
4. Use Conventional Commits for messages and the PR title.
5. Link issues with Closes #N / Refs #N.
6. Open a PR; fill in the template; CI must be green.

For parallel human or agent work, use the Worktrunk worktree workflow so each task has an isolated branch, index, and directory.

Code standards

• Rust 2021; rustfmt defaults; clippy clean with -D warnings.
• thiserror for library errors, anyhow in binaries.
• Keep the data-plane hot path allocation-light; never block on locks (use arc-swap).
• Code comments start lowercase, no trailing punctuation; /// doc comments use normal prose.
• New balancing strategy → implement rolter_balancer::LoadBalancer + wire into build().
• New storage backend → implement rolter_store traits behind a cargo feature.

Agent commits

Automated contributions include the trailer:

Co-Authored-By: Oz <oz-agent@warp.dev>

Don’t

• Don’t commit secrets (use env / the encrypted store).
• Don’t force-push master; force-push is fine on your own feature branches.
• Don’t --amend after pushing shared history.

Parallel development with Worktrunk

Rolter uses Worktrunk as a thin lifecycle and visibility layer over standard Git worktrees. Each development agent gets an independent directory, index, and branch while normal Git history and GitHub remain authoritative.

The workflow is agent-neutral. Codex, Claude, Z.ai, Warp, and other agents all use the same worktree layout and branch rules. Tool-specific Worktrunk plugins are optional local integrations; they are not required by the repository.

Install

On macOS or Linux with Homebrew:

brew install worktrunk
wt config shell install

Alternatively, install the Rust binary:

cargo install worktrunk
wt config shell install

Restart the shell, then confirm that the shell wrapper is active:

type wt
wt --version
wt config show

The repository’s .config/wt.toml identifies GitHub as the forge and starts a background copy-on-write cache transfer for entries in .worktreeinclude. Only reproducible build caches are selected. Credentials and .env files must be configured independently in each worktree and are never copied by the repository hook.

Project hooks require one-time approval. Review the rendered command before approving it; agents should use --yes only after that review.

Start an independent task

Fetch first, then create the issue branch from the remote default branch:

git fetch origin master
wt switch --create fix/123-short-description --base origin/master

Use the repository branch format <type>/<issue-number>-<short-description>. Never add an agent or person name as a prefix.

Start the chosen agent inside the worktree that wt switch selected. An orchestrator without shell integration can obtain paths from structured output:

wt list --format=json

Automation that persists this output should explicitly select Worktrunk’s new schema until it becomes the default:

wt --config-set list.json-schema=2 list --format=json

Every agent must own exactly one branch and worktree. Never let two agents push the same feature branch. Worktrees isolate files and indexes, but branch refs and remote-tracking refs are shared by the repository.

Dependent tasks

Use an explicit parent branch as the base:

wt switch --create feat/124-dependent-change --base feat/123-foundation

The child pull request targets the parent branch. After the parent merges, fetch origin/master, rebase the child in its own worktree, validate it, and push with --force-with-lease. Do not run repository-wide branch synchronizers across active worktrees.

Inspect the agent fleet

wt list
wt list --full

Before assigning or cleaning work, inspect dirty state, divergence, conflicts, CI status, duplicate branches, and prunable registrations. Activity markers from Worktrunk plugins are advisory: a crashed or disconnected agent may leave a stale marker.

Commit and publish

Worktrunk manages worktree lifecycle only. Use normal repository commands for commits and publication:

prek run --all-files
git push --set-upstream origin HEAD
gh pr create --base master --draft

Use the immediate parent instead of master for a stacked pull request. Fill the PR template, use a Conventional Commit title, and link the issue with Closes #N only when the PR completes its acceptance criteria.

Do not use wt merge, wt step commit, wt step squash, or wt step push for Rolter delivery. Merge through GitHub only after hosted ci-ok, review, and acceptance-criteria verification. Worktrunk hooks are convenience automation, not a security boundary, and --no-hooks can bypass them.

Remove completed work

Confirm the PR is merged and the worktree is clean before removal:

wt list --full
wt remove <branch>

The shared pre-remove hook runs cargo clean inside that worktree before Worktrunk deletes it. This reclaims the copied target/ cache while the path still exists; source files and other worktrees are unaffected.

Worktrunk deletes a branch only when it can prove the branch adds no changes to the default branch. When the merge state is uncertain, preserve the branch:

wt remove --no-delete-branch <branch>

Never use --force or --force-delete in an automated cleanup path. Treat prunable legacy registrations separately: inspect git worktree prune --dry-run, verify every path, and only then run git worktree prune.

Optional local integrations

Install only the plugins for agents used on a particular machine:

wt config plugins codex install
wt config plugins claude install
wt config plugins opencode install
wt config plugins gemini install

Agents without a Worktrunk plugin, including Z.ai or Warp-based agents, simply run inside the path created by wt switch. Repository behavior must never depend on a specific agent plugin being installed.

Dashboard localization (i18n)

The dashboard SPA under ui/ is translated with i18next + react-i18next. English is the base catalog; Russian ships alongside it. Switching language is instant — react-i18next re-renders the tree, nothing reloads and no route changes.

Everything is bundled. There is no CDN, no language-detection service and no runtime fetch to a translation host, because rolter must run fully air-gapped.

Layout

en is imported statically because it is the fallback and must always resolve. Every other locale is a dynamic import(), so Vite emits it as its own chunk and a user only downloads the language they actually pick.

Key naming

Keys are nested and grouped by where the copy lives:

<screen-key> is the nav key, which is also the route path — so a screen’s label, header and body copy are all found by the same string.

Keep keys semantic (pages.dashboard.noTraffic), never the English text itself: re-wording the English copy should not force a key rename.

Writing translatable copy

import { useTranslation } from "react-i18next";

const { t } = useTranslation();
return <h2>{t("pages.dashboard.recentTitle")}</h2>;

Interpolation — name the value, never concatenate:

// en.json
"roleWithOrg": "Admin · {{org}}"

t("shell.roleWithOrg", { org: orgName })

Plurals — use count, and let i18next pick the form. English needs two forms, Russian needs four; the catalogs differ on purpose and the gate knows the difference:

// en.json                    // ru.json
"errors_one":   "{{count}} error",     "errors_one":   "{{count}} ошибка",
"errors_other": "{{count}} errors"     "errors_few":   "{{count}} ошибки",
                                       "errors_many":  "{{count}} ошибок",
                                       "errors_other": "{{count}} ошибки"

t("pages.dashboard.errors", { count: errors })

Markup inside a sentence — use <Trans> with numbered slots, so each language can put the emphasised part where its grammar wants it:

"deleteHint": "This removes <0>{{name}}</0>. This cannot be undone."

<Trans
  i18nKey="scope.deleteHint"
  values={{ name }}
  components={[<span key="name" className="font-mono" />]}
/>

Never interpolate a noun into a sentence frame. "Add {{level}}" cannot be declined correctly in Russian (and most inflected languages). Spell each case out as its own key — scope.addOrg, scope.addTeam, scope.addProject — and select it with a lookup table in the component.

Numbers, money and dates

Use useFormat(), never bare toLocaleString(). A bare call follows the browser locale, so a ru-RU browser rendered Russian separators inside an otherwise English panel.

const fmt = useFormat();
fmt.number(1234567);     // 1,234,567   / 1 234 567
fmt.currency(0.0004);    // $0.0004     (sub-cent costs stay visible)
fmt.percent(0.734);      // 73.4%
fmt.dateTime(row.ts);

Adding a locale

1. Copy ui/src/lib/i18n/locales/en.json to <code>.json and translate it. Leave locale.en / locale.ru / … as native names in every catalog — a language is always listed in its own language, so a user who picked the wrong one can find their way back.
2. Register it in ui/src/lib/i18n/index.ts: add the code to LOCALES, a native name to LOCALE_NAMES, a two-letter badge to LOCALE_SHORT, and a dynamic import to loaders.
3. Run bun run check:i18n — it will list every key you still owe, and every plural form the language requires.

No other file changes. The picker builds itself from LOCALES.

Locale selection

detectLocale() resolves, in order:

1. the stored choice in localStorage["rolter.locale"],
2. the browser’s navigator.languages, matched on the primary subtag so ru-RU resolves to ru,
3. en.

Picking a language persists the choice and sets <html lang>. This is dashboard-only state — it is not written to the account, so it needs no control-plane change. (Per-user server-side persistence is tracked separately.)

Keeping catalogs honest

cd ui
bun run check:i18n     # the CI gate
bun test src/lib/i18n  # the same rules, plus runtime behaviour

The gate fails on: a key missing from a locale (which would silently fall back to English mid-sentence), an orphaned key whose screen is gone, an empty string, a dropped {{placeholder}} or <0> slot, and an incomplete set of plural forms. Both entry points share ui/src/lib/i18n/parity.ts, so bun test and CI cannot disagree.

When you add or re-word copy, update en.json and every other catalog in the same PR — a half-translated catalog is a CI failure, not a follow-up.

Catching strings that never reach a catalog

Catalog parity is a property of the catalogs. It cannot see the failure that actually happens: a string hardcoded in JSX, which is not in any catalog by definition. That is how EditorSheet shipped window.confirm("Discard unsaved changes?") and a default "Cancel" label with a green merge gate (#871).

cd ui
bun run check:literals            # fail on any literal not in the baseline
bun run check:literals --update   # re-record the baseline (only to shrink it)

It scans src/components/** and src/pages/** for three shapes:

A line containing t( is skipped, because that line is already doing the right thing. Acronyms, identifiers, URLs and bare numbers are not copy and are ignored. Stories and tests are not scanned: a story’s job is to render a component with concrete sample text, and routing that through the catalogs would make the stories test the catalogs instead of the component.

Why there is a baseline

The rule is repo-wide and the debt predates it — several hundred literals already exist. Failing on all of them would mean either a several-hundred-string translation PR nobody asked for, or a gate that is permanently red and therefore ignored. So the existing set is recorded in literals-baseline.json and the gate fails on anything new.

The baseline only shrinks. When you translate a recorded literal, the gate tells you the entry is stale and asks you to drop it, so the same string cannot come back unnoticed.

It is a lexical scan, not a TypeScript parse. The question is “does a user-visible English string appear where a t() call belongs”, and every candidate is a string literal or a JSX text node — both of which a regex reads accurately enough to answer it, without a compiler dependency in a script that runs on every push.

Dashboard error states

A screen that cannot load its data has to say why. Before #962 every screen rendered the same sentence — Failed to load X. — for causes needing entirely different responses, so it pointed at none of them.

That is not a hypothetical cost. During the #924 dogfooding pass the Keys screen showed “Failed to load your keys.” while the real cause was every /api/v1/me/* route returning 401 (#942). The message sent the operator to check their key configuration; the actual cause was found afterwards by reading traces. An error that cannot separate you are not signed in from the server is down costs more time than no error at all, because it invites a wrong hypothesis and the operator spends their attention there first.

The rule

Never render a load failure by hand. Use LoadError:

{keys.error && (
  <LoadError
    error={keys.error}
    resource={t("errors.resources.virtualKeys")}
    onRetry={() => keys.refetch()}
  />
)}

resource is the translated noun for what failed — it is interpolated into the title, so it reads as a sentence in every locale. Pass onRetry whenever the caller holds a query handle; the component decides whether offering it is honest.

What it distinguishes

classifyLoadError in ui/src/lib/load-error.ts maps a thrown value to one of six kinds. ApiError already carries status and the control plane’s code, so no screen has to parse a message to find out what happened.

Two of these are easy to collapse and must not be. A plain 401 is fixed by signing in; open_mode_no_session is a control plane running with no admin token, which has no accounts to sign into at all — signing in again is exactly the wrong advice. And a retry button on a 403 suggests the failure was transient when it was a permission, so isRetryable withholds it.

Two things that are not this component

An empty result is not a failure. A successful request returning zero rows renders an empty state. Routing it here would tell an operator something is broken when nothing is.

The control plane’s own message is never swallowed. LoadError prints it beneath the summary. The dashboard’s classification is a helpful gloss, not a replacement — #962 happened because the gloss was the only thing on screen and it was wrong.

Adding a screen

Add the resource noun to errors.resources.* in every catalog under ui/src/lib/i18n/locales/ (see i18n) and use it as above. The six errors.load.* kinds already exist; a new screen needs no new error copy.

Commit conventions

rolter uses Conventional Commits for commit messages and PR titles. CI checks PR titles; the conventional-pre-commit hook managed by prek checks local messages.

Format

<type>(<scope>): <subject>

<body>

<footer>

• type (required): feat, fix, perf, refactor, docs, test, build, ci, chore, revert
• scope (recommended): gateway, balancer, proxy, core, store, auth, control, ui, docs, infra, ci, deps, release
• subject: imperative, lowercase, ≤ 72 chars, no trailing period
• breaking change: add ! after the scope and a BREAKING CHANGE: footer

Examples

feat(balancer): add precise kv-event cache-aware scorer
fix(gateway): stream anthropic sse without buffering
perf(proxy): reuse pooled client per egress proxy
docs(architecture): document reload-free config propagation
refactor(core)!: rename ModelRoute.targets to upstreams

BREAKING CHANGE: config field `targets` is now `upstreams`.

Issues & PRs

• Link issues from the body/footer: Closes #123, Refs #123.
• PR title must be a single valid Conventional Commit line (enforced by CI via amannn/action-semantic-pull-request).
• Squash-merge so the PR title becomes the commit on master; keeps history releasable and changelog-friendly.

Tooling

• .config/commitlint.config.mjs — rules (types, scopes, lowercase subject, 72-char header).
• prek.toml — fast commit-time hygiene, secret scanning, formatting/linting, commit-message validation, and pre-push test/security gates.
• Install all configured hook stages with prek install --prepare-hooks. The configuration installs pre-commit, commit-msg, and pre-push shims.
• Run commit-time checks manually with prek run --all-files.
• Run the push gate manually with prek run --all-files --hook-stage pre-push.

The commit stage uses prek’s built-in checks plus pinned Gitleaks and Conventional Commit hooks. Project-specific checks require actionlint, taplo, and typos on PATH. The push stage also requires cargo-deny and Bun when UI files are part of the push. Install cargo-nextest for CI-equivalent test execution; the hook falls back to cargo test when it is unavailable.

Packaging & distribution

rolter ships three ways.

The unified rolter binary dispatches to both planes via subcommands:

rolter gateway --config rolter.toml     # data plane
rolter control --database-url postgres://…   # control plane + UI host

The standalone rolter-gateway / rolter-control binaries remain available.

cargo

cargo install rolter            # unified launcher (from crates.io)
# or from source:
cargo install --path crates/rolter

uv (PyPI wheel via maturin)

The wheel bundles the compiled rolter launcher so Python users can install the CLI with uv. pyproject.toml uses the maturin backend (bindings = "bin", manifest-path = crates/rolter/Cargo.toml).

Each release publishes five wheels plus a source distribution:

The macOS x86_64 wheel is cross-compiled rather than built on an Intel runner — the macOS SDK carries both architectures, so it needs no extra runner. The sdist is the fallback for anything with no matching wheel: without one, pip install rolter fails outright on an unlisted platform instead of building from source. verify-parity asserts all six are present for the version, so a silently missing platform fails the release rather than reaching a user.

uv tool install maturin       # one-time
uvx maturin build --release   # build a wheel into target/wheels/
uv tool install rolter        # once published to PyPI

Docker

Multi-stage docker/Dockerfile builds the Rust binaries and the Bun-built UI, then assembles a slim runtime:

docker build -f docker/Dockerfile -t rolter:dev .
docker compose -f docker/docker-compose.yml up -d          # full stack with postgres/redis/clickhouse

Release pipeline

Releases are fully automated from Conventional Commits. Two workflows do the work, and the handoff between them is the part worth understanding.

merge to master
      │
      ▼
release-plz.yml ── release-pr ──►  "Release PR" (version bump + changelogs)
      │
      │ (that PR is merged)
      ▼
release-plz.yml ── verify ──► release ──►  crates.io publish
                                           tag v{version}
                                           github release
      │
      │ workflow_dispatch -f tag=v{version}      ← dispatch-artifact-release
      ▼
release.yml
  │
  ├─ gate ──── verify-external-checks (ci-ok + CodeQL green for the tagged sha)
  │
  ├─ build ─── build-wheels  (5 wheels + sdist)
  │            build-image   (per arch, pushed as untagged digests)
  │
  ├─ smoke ─── smoke-wheels  (install each wheel, run `rolter --version`)
  │            smoke-image   (run each image digest, check its version)
  │
  ├─ publish ─ publish-pypi    (trusted publishing, OIDC)
  │            publish-docker  (assemble tag manifests: GHCR + Docker Hub)
  │
  └─ check ─── verify-parity  (all channels serve {version})

The stages are a barrier, not decoration. Every publish job depends on every build and smoke job, so a release is all-or-nothing: a failed wheel can no longer leave container images published against a version that has nothing on PyPI. Before this split, publish-docker did not depend on build-wheels at all, and exactly that partial release was possible.

Two properties make the barrier real:

• Images are built as untagged digests (push-by-digest). A digest nobody can resolve by tag is not a release; publish-docker only assembles the :{version} and :latest manifests once everything else has passed, from those same digests — so the image is never rebuilt.
• Nothing is published untested. The smoke stage installs each wheel and runs each image digest, asserting rolter --version matches the tag, while the artifacts are still private. The wheel install uses --no-index, so it can only resolve from the freshly built dist/ and can never pass by silently pulling an older rolter from PyPI.

Each build is its own job, so a single flaky platform can be re-run on its own without re-publishing anything that already succeeded.

Why the explicit dispatch

release.yml also has a push: tags trigger, but it never fires for a real release. release-plz creates the tag with the repository GITHUB_TOKEN, and GitHub suppresses downstream workflow events for token-created refs to prevent recursive runs. workflow_dispatch is the documented exception — it always creates a run, even from the GITHUB_TOKEN — so release-plz.yml ends with a dispatch-artifact-release job that calls gh workflow run release.yml -f tag=vX.Y.Z. That job holds actions: write and nothing else, and it fails if the dispatch produces no run.

Without it the pipeline half-works in the worst way: the GitHub release and crates.io advance while no wheel is ever built, and every job stays green. That is how v0.0.6 through v0.0.10 shipped while PyPI sat on 0.0.5 (#903). scripts/check-release-handoff.sh (a merge gate in quality.yml and a prek hook) asserts the wiring is still in place.

Which tag the dispatch carries

releases, the output release-plz hands back, contains a tag for every published crate — not only the one with git_tag_enable. The crates.io-only members get a derived <crate>-v<version> string that was never pushed to git. Only rolter-gateway is configured with git_tag_name = "v{{ version }}", so resolve release tag selects that entry by package name; an empty result means nothing was released on this push and the dispatch job is skipped.

Taking the first tagged entry instead is what broke v0.0.11 (#1026): the array starts with rolter-core, the dispatch carried rolter-core-v0.0.11, and release.yml failed at checkout on a ref that does not exist — so v0.0.11 went to crates.io and GitHub Releases with no wheel, and PyPI stayed on 0.0.10. The step now also rejects any resolved tag that is not vX.Y.Z, so a bad ref fails before it is dispatched rather than halfway through the artifact build.

Parity gate

verify-parity runs with always() at the end of release.yml and asserts that every enabled channel actually serves the tagged version: the GitHub release exists, crates.io has rolter {version}, PyPI has it (when PYPI_PUBLISH_ENABLED is true), and GHCR has the manifest (when DOCKER_PUBLISH_ENABLED is true). A skipped or failed publish turns the run red instead of quietly leaving a channel behind.

Publishing gates

Wheels are built with maturin-action but uploaded with pypa/gh-action-pypi-publish: maturin upload is deprecated and slated for removal (PyO3/maturin#2334). The publisher identity PyPI matches on is the repository, workflow filename and environment — not the tool — so the swap is transparent to the trusted-publisher config, and it adds PEP 740 attestations on the id-token grant the job already holds.

The gate is asserted, not re-run

release.yml does not run quality.yml itself. It asserts that the tagged commit already passed it, by requiring ci-ok among the check-runs recorded for that SHA. That is deliberate, and it is what makes the gate correct:

A local reusable workflow (uses: ./…) always checks out the caller’s ref. On a workflow_dispatch the caller ref is master, while build-wheels checks out inputs.tag — so a re-run verified master and shipped the tag (#988). It passed, and told you nothing about what was being packaged. Threading the tag into quality.yml fixes that but makes the shared workflow check out an arbitrary dispatch-supplied ref in a default-branch context, whose caches trusted runs later restore — cache poisoning, and CodeQL flags it.

Asserting settles both. Every commit on master carries a ci-ok check-run from ci.yml, and release-plz re-runs the gate on the release commit before tagging, so a tagged commit is verified by construction. The assertion binds to the tagged SHA — which re-running never did — costs no duplicate 20-minute run, and checks out nothing.

Because release-plz dispatches the moment it finishes tagging, ci-ok is often still running for that commit. A pending check is therefore expected, not a failure: the job waits up to 45 minutes for a verdict, fails immediately on a real non-success, and fails closed if a required check never appears.

RELEASE_REQUIRED_CHECKS holds exact check-run names, so it rots whenever a scanner is renamed or reconfigured — and since the gate is fail-closed, a stale name silently blocks every release instead of failing at the source. This bit rolter once already: the variable still named the CodeQL default setup jobs (Analyze (rust), …) after the repo moved to advanced setup (codeql (rust), …), so no release could publish even with a working tag dispatch. If the gate reports “required check … not found”, compare it against the check-run names the job log prints and update the variable.

Releasing a tag by hand

For a backfill, or if a dispatch was lost, run the artifact half yourself from the default branch:

gh workflow run release.yml -f tag=v0.0.10
gh run watch "$(gh run list --workflow=release.yml --limit 1 --json databaseId -q '.[0].databaseId')"

The upload uses --skip-existing, so re-running a tag that partly published is safe. Verify the result:

curl -s https://pypi.org/pypi/rolter/json | jq -r .info.version
uv tool install rolter && rolter --version

Backfill policy

Only the current release is backfilled to PyPI. The versions the broken handoff skipped (v0.0.6 – v0.0.9) stay unpublished: they are superseded pre-1.0 releases, uv tool install rolter / pip install rolter resolve to the latest version regardless, and publishing them retroactively would put four versions on the index that no user ever pinned, dated years after their tags. Anyone who needs one of them can build from the tag or cargo install rolter@0.0.x from crates.io, which has the complete series.

Zero-config quickstart: no keys, add providers at runtime

rolter starts with zero LLM credentials and serves the built-in fake-llm model out of the box. Real providers, models, and upstream API keys are added later — at runtime, over the management API, persisted in Postgres, and picked up by the gateway without a restart.

1. Start with no credentials

rolter easy-up

That’s it. No provider keys, no database, no config file (one is created from the bundled example on first run). The gateway answers immediately:

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-rolter-dev" \
  -H "Content-Type: application/json" \
  -d '{"model": "fake-llm", "messages": [{"role": "user", "content": "hi"}]}'

sk-rolter-dev is the local-dev virtual key from the generated rolter.toml; delete the [[virtual_keys]] section to run open, or replace it before exposing the gateway anywhere.

2. Switch on runtime management (Postgres mode)

Runtime CRUD over providers/models/keys needs the database-backed control plane:

export ROLTER_ADMIN_TOKEN="$(openssl rand -hex 24)"   # protects the management API
export ROLTER_KEK="$(openssl rand -hex 32)"           # encrypts provider keys at rest

rolter easy-up --database-url postgres://user:pass@localhost:5432/rolter

easy-up migrates, seeds a default org/team/project, imports the bootstrap toml, and starts both planes. The gateway port (4000) now also serves the management API: /admin/* proxies to the control plane’s /api/v1/*.

Two deployment secrets matter here:

• ROLTER_ADMIN_TOKEN — bearer token required on the management API and the internal snapshot endpoint. Without it those endpoints are open (fine on localhost; a startup warning reminds you).
• ROLTER_KEK — key-encryption key. Provider API keys submitted over the API are sealed with AES-256-GCM before they reach Postgres; the KEK never leaves the process environment. Set the same value on the control plane and gateway (with easy-up it is one process, so one export). Without a KEK, requests that include an api_key are rejected — there is no plaintext fallback.

3. Add your first real provider — with its key — via curl

BASE=http://localhost:4000/admin

# ids seeded by easy-up
ORG=$(curl -s $BASE/orgs -H "Authorization: Bearer $ROLTER_ADMIN_TOKEN" | jq -r '.[] | select(.name=="default") | .id')
TEAM=$(curl -s $BASE/orgs/$ORG/teams -H "Authorization: Bearer $ROLTER_ADMIN_TOKEN" | jq -r '.[0].id')
PROJECT=$(curl -s $BASE/teams/$TEAM/projects -H "Authorization: Bearer $ROLTER_ADMIN_TOKEN" | jq -r '.[0].id')

# provider + upstream credential (sealed at rest; never returned by the API)
PROVIDER=$(curl -s -X POST $BASE/orgs/$ORG/providers \
  -H "Authorization: Bearer $ROLTER_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"name": "openai", "kind": "openai", "api_base": "https://api.openai.com", "api_key": "sk-..."}' \
  | jq -r .id)

# public model name + target
ROUTE=$(curl -s -X POST $BASE/projects/$PROJECT/routes \
  -H "Authorization: Bearer $ROLTER_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o", "strategy": "round_robin"}' | jq -r .id)

curl -s -X POST $BASE/routes/$ROUTE/targets \
  -H "Authorization: Bearer $ROLTER_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d "{\"provider_id\": \"$PROVIDER\", \"upstream_model\": \"gpt-4o\"}"

Within the snapshot poll interval (instantly with --redis-url) the gateway serves the new model — no restart:

curl http://localhost:4000/v1/models -H "Authorization: Bearer sk-rolter-dev"

4. Rotate or remove a credential

PUT /admin/providers/{id} updates a provider in place. For api_key, api_key_env, and egress_proxy: omit the field to leave it unchanged, send an empty string to clear it, send a value to set/rotate it.

# rotate
curl -X PUT $BASE/providers/$PROVIDER \
  -H "Authorization: Bearer $ROLTER_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"api_key": "sk-new-key"}'

# remove the stored key (falls back to api_key_env, if set)
curl -X PUT $BASE/providers/$PROVIDER \
  -H "Authorization: Bearer $ROLTER_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"api_key": ""}'

How the pieces fit

• Persistence — providers/routes/keys live in Postgres; credentials in provider_keys, AES-256-GCM-sealed with the ROLTER_KEK-derived key.
• Propagation — every write bumps config_version (a database trigger, transactional with the write). Gateways poll GET /internal/snapshot?version=N and hot-swap their routing snapshot; with Redis configured the control plane also publishes a bump for instant refetch. See Config & hot reload.
• Two surfaces, one API — /admin/* on the gateway is a thin reverse proxy to the control plane’s /api/v1/* (enable on a standalone gateway with --admin-url http://control:4001); pointing tooling at either works identically. Authentication is enforced by the control plane in both cases.
• Config file still wins — anything declared in the bootstrap rolter.toml is a read-only “config model” (LiteLLM-style): the API rejects runtime mutations to it with 409.
• Reads never leak secrets — GET /api/v1/config (the dashboard read) redacts api_key; only the token-guarded snapshot endpoint carries decrypted keys, because the gateway needs them to call upstreams.

Configuration reference

The gateway boots from a TOML file (--config, default rolter.toml); see rolter.example.toml. At runtime, the control plane is the source of truth and applies changes without a restart (config-and-hot-reload.md).

Schema

[server]

• host (string, default 0.0.0.0)
• port (u16, default 4000)
• metrics_path (string, default /metrics) — path the Prometheus metrics endpoint is served on; change it to avoid colliding with an upstream app or sidecar that already owns /metrics. Must be rooted (/…) and must not collide with a built-in route (/healthz, /v1/*).

[tls]

• ca_bundles (string[], default []) — PEM CA-bundle files added to the normal public-root trust store for outbound upstream TLS. ROLTER_CA_BUNDLE replaces this global list with a single deployment-local path. Files are checked for missing, unreadable, empty, and malformed content while config is loaded.

[[providers]]

• name (string, unique) — referenced by route targets
• kind (openai | anthropic | openai_compatible | ollama | ollama_cloud | llama_cpp | openrouter | tei | azure_openai | bedrock | vertex | gemini | gemini_native | gemini_interactions | mistral | groq | xai | meta_llama_api | cohere | perplexity | together | fireworks | databricks | aleph_alpha | nebius | ovhcloud | scaleway | deepseek | qwen | zhipu | kimi | ernie | doubao | hunyuan | yi | minimax | baichuan | gigachat | yandex_gpt | cloud_ru | mts_ai | naver | upstage | rinna | rakuten | sarvam | krutrim | falcon)
• api_base (string) — base URL, no trailing slash
• api_key (string, optional) — prefer api_key_env
• api_key_env (string, optional) — environment variable to read the key from
• role_profile (openai | system_only | anthropic, optional) — explicit instruction-role semantics. The default is openai for kind = "openai", anthropic for kind = "anthropic", and conservative system_only for every OpenAI-compatible kind. system_only converts leading developer messages to system in place; it rejects a system or developer message after a user/assistant/tool turn with role_capability_unsupported rather than silently changing it.
• model_role_profiles (table, optional) — upstream-model-specific role_profile overrides. Use this only for a custom template whose developer-role support is explicitly known; rolter never probes a vLLM template at runtime.

Role-capability profiles

openai_compatible describes the HTTP surface only. vLLM, in particular, renders roles using the selected model’s chat template, so an endpoint’s role support must not be inferred from its /v1 API. The default system_only profile is suitable for Qwen-style templates that do not define developer. Set role_profile = "openai" or a model_role_profiles entry only after confirming that the deployed template supports distinct developer messages.

Anthropic targets collect leading OpenAI developer and system messages into ordered top-level system blocks. Instruction messages placed after a conversation turn are rejected for anthropic and system_only profiles; rolter returns an OpenAI-style 400 with code role_capability_unsupported instead of dropping or reclassifying them.

Ollama: local daemon vs Cloud

Use ollama for a local/self-hosted daemon such as http://localhost:11434 (no authentication). Use ollama_cloud for direct programmatic Cloud access. Cloud requires api_key_env (normally OLLAMA_API_KEY); inline keys and key pools are rejected. Configure api_base = "https://ollama.com"; rolter uses the OpenAI-compatible /v1/chat/completions and /v1/models endpoints with bearer authentication. Ollama’s native /api/* endpoints are distinct.

[[providers]]
name = "ollama-cloud"
kind = "ollama_cloud"
api_base = "https://ollama.com"
api_key_env = "OLLAMA_API_KEY"

Azure OpenAI, Amazon Bedrock, and Vertex AI

These providers use their current OpenAI-compatible APIs. Set api_base to the provider’s OpenAI-compatible prefix and use an environment-sourced credential:

[[providers]]
name = "azure"
kind = "azure_openai"
api_base = "https://RESOURCE.openai.azure.com/openai/v1"
api_key_env = "AZURE_OPENAI_API_KEY"

[[providers]]
name = "bedrock"
kind = "bedrock"
api_base = "https://bedrock-runtime.us-east-1.amazonaws.com/v1"
api_key_env = "AWS_BEARER_TOKEN_BEDROCK"

[[providers]]
name = "vertex"
kind = "vertex"
api_base = "https://aiplatform.googleapis.com/v1/projects/PROJECT/locations/global/endpoints/openapi"
api_key_env = "VERTEX_ACCESS_TOKEN"

Azure credentials are sent in the api-key header. Bedrock and Vertex credentials are sent as bearer tokens. The default active-health probes use Azure’s model list, Bedrock ListFoundationModels, and Vertex’s publisher model list, respectively; none invokes a model.

Google Gemini, Mistral, Groq, and xAI (Grok) expose hosted OpenAI-compatible APIs. Their api_base already carries the version segment, so rolter strips the leading /v1 from the gateway path before appending it. Keys are bearer tokens sourced from api_key_env (inline keys are rejected); the free health probe lists {api_base}/models.

gemini_native targets Gemini’s native generateContent surface instead of its OpenAI-compatible shim. rolter translates OpenAI Chat / Anthropic Messages / OpenAI Responses requests into Gemini’s contents/parts wire format and converts the response (and SSE stream) back, so clients keep speaking their usual protocol. The model and method are embedded in the URL ({api_base}/models/{model}:generateContent, or :streamGenerateContent?alt=sse for streaming), the key is sent as x-goog-api-key, and api_base points at the version root with no /openai suffix.

gemini_interactions targets Gemini’s stateful Interactions API. Every chat-shaped request is translated onto the single {api_base}/interactions endpoint: turns become input items, system/developer messages become system_instruction, tool calls and tool results become function_call / function_result items, and sampling parameters become generation_config. The response steps[] (and the interaction.created / step.delta / interaction.completed SSE events) are converted back into the client’s dialect. The thread is client-driven: the interaction id is returned as the response id, and a client resumes it by sending previous_response_id (OpenAI Responses) or previous_interaction_id — rolter keeps no interaction state of its own. Auth is x-goog-api-key, and api_base points at the version root. Endpoints with no interactions equivalent (embeddings, audio) are rejected rather than forwarded.

[[providers]]
name = "gemini"
kind = "gemini"
api_base = "https://generativelanguage.googleapis.com/v1beta/openai"
api_key_env = "GEMINI_API_KEY"

# native generateContent wire format (translated from OpenAI/Anthropic)
[[providers]]
name = "gemini-native"
kind = "gemini_native"
api_base = "https://generativelanguage.googleapis.com/v1beta"
api_key_env = "GEMINI_API_KEY"

# stateful Interactions API (translated from OpenAI/Anthropic/Responses)
[[providers]]
name = "gemini-interactions"
kind = "gemini_interactions"
api_base = "https://generativelanguage.googleapis.com/v1beta"
api_key_env = "GEMINI_API_KEY"

[[providers]]
name = "mistral"
kind = "mistral"
api_base = "https://api.mistral.ai/v1"
api_key_env = "MISTRAL_API_KEY"

[[providers]]
name = "groq"
kind = "groq"
api_base = "https://api.groq.com/openai/v1"
api_key_env = "GROQ_API_KEY"

[[providers]]
name = "xai"
kind = "xai"
api_base = "https://api.x.ai/v1"
api_key_env = "XAI_API_KEY"

• [[providers.api_keys]] (optional) — multiple weighted API keys for one provider; when present it takes precedence over the single api_key/api_key_env pair. Providers cap throughput per key, so rotating across keys multiplies effective RPM/TPM
  • key (string, optional) — inline key value; prefer env
  • env (string, optional) — environment variable to read the key from
  • weight (u32, default 1) — relative selection weight
• api_key_env (string, optional) — env var to read the key from
• egress_proxy (string, optional) — legacy single HTTP/HTTPS/SOCKS5 outbound proxy; treated as a one-element pool
• egress_proxies (string[], optional) — round-robin HTTP, HTTPS, SOCKS5, or SOCKS5H proxy pool. A connect/tunnel failure retries the next member; three consecutive failures quarantine a member for 30 seconds. Authenticated proxy URLs must be supplied as whole-value environment references such as "${PROVIDER_PROXY_EU}", keeping credentials out of config snapshots and database/API output
• ca_bundles (string[], optional) — provider-specific replacement for global [tls].ca_bundles; [] explicitly selects public roots only
• [providers.kv_events] (optional) — vLLM V1 ZMQ KV-event source for precise_cache_aware: endpoint (tcp://…), topic (default kv-events), max_blocks (default 1,000,000), and stale_secs (default 30)
• [providers.lmcache] (optional) — LMCache controller signal for lmcache_aware: endpoint (HTTP JSON occupancy signal), refresh_secs (default 2), and stale_secs (default 10)
• also_track_via_llm_call (bool, default false) — when set, active health checks send a real max_tokens = 1 completion to this provider instead of the free /v1/models liveness probe, so a healthy result proves end-to-end inference. This burns a few tokens on every sweep (interval_secs); leave it off unless you need inference-level health. Recorded as source = llm_call in provider_health_events.
• llm_probe_model (string, optional) — the upstream model id the also_track_via_llm_call completion targets (e.g. gpt-4o-mini). Required when the flag is on; without it (or an api key) the checker logs a warning and falls back to the free probe.
• status_page_url (string, optional) — statuspage.io-style status.json URL (e.g. https://status.anthropic.com/api/v2/status.json). When set, a slow background poll records the provider’s public status as a secondary status_page health signal — it surfaces in provider_health_events, the dashboard and rolter_status_page_degraded_total, but never marks the provider unhealthy or affects routing on its own. Parse/transport failures are logged and skipped.

See Custom CA bundles for rotation behavior and Docker/Kubernetes mount examples.

[[routes]]

• model (string) — public model name clients request
• strategy (round_robin | random | power_of_two | consistent_hash | cache_aware | weighted | pipeline | cheapest | fastest | precise_cache_aware | lmcache_aware | adaptive | lora_aware | predicted_latency, default round_robin)
• [[routes.targets]]
  • provider (string) — a provider name
  • model (string, optional) — upstream model id; defaults to the requested model
  • weight (u32, default 1)
• [routes.params] (table, optional) — admin default inference params injected into the request body (e.g. temperature, max_tokens, stop). Provider-agnostic: keys are whatever the upstream accepts. An unset param passes through untouched.
• [routes.param_policy] — whether callers may override the params defaults
  • mode (allow | deny, default allow) — baseline override policy
  • allow (string[], default []) — params callers may override when mode = "deny"
  • deny (string[], default []) — params callers may not override when mode = "allow"
  • when an override is denied and the caller sends the param anyway, the admin default silently wins
• [[routes.variants]] (optional) — weighted variants for A/B, canary, and key-split traffic. When present, the route ignores the top-level targets pool: a request samples one variant by weight (the primary) and, on failure, falls over to the remaining variants in declared order. Within a variant the route’s strategy picks which target leads; the remaining targets follow in declared order as the deterministic fallback tail.
  • name (string) — variant identifier, attributed in request logs (the variant column)
  • weight (u32, default 1) — relative traffic share for the primary draw
  • [[routes.variants.targets]] — same shape as [[routes.targets]]
  • [routes.variants.params] (table, optional) — variant-scoped param defaults, layered over [routes.params] (the variant wins) under the route’s param_policy
• [routes.advanced.guardrails] (optional) — per-route selection layered over the global [guardrails] rule set. Rules are named, so adding one globally still reaches every route that has not opted out of it by name.
  • disable (string[], default []) — rules that do not apply on this route
  • enable (string[], default []) — rules that apply on this route; wins over disable on a conflict
  • a name matching no configured rule fails validation rather than being ignored — a typo in disable would otherwise read as “this rule is off here” while the rule kept running

[[virtual_keys]]

• key (string) — the bearer token clients present
• name (string, optional)
• models (string[], default []) — allow-list; empty = all

[adaptive_routing]

Deployment-wide policy for routes using the adaptive strategy. See load balancing.

• enabled (bool, default false) — kill switch; while off, every adaptive route serves the pipeline stack
• latency_weight (f32, default 1.0), cost_weight (f32, default 0.5), load_weight (f32, default 0.25) — blend weights; negatives are clamped to 0, and all-zero disables the blend
• exploration_ratio (f32, default 0.05) — share of picks made at random to keep latency samples fresh; clamped to [0, 0.5]
• min_samples (u32, default 50) — requests a route must serve before the blend engages

[logging]

• clickhouse_url (string, optional)

[health]

• enabled (bool, default false) — master switch for active upstream probing
• interval_secs (u64, default 10) — seconds between probe sweeps
• timeout_secs (u64, default 2) — per-probe timeout
• path (string, default /) — probe path; the default resolves to each provider kind’s free liveness endpoint (normally /v1/models, or the provider-native Azure, Bedrock, or Vertex model-list endpoint)
• probe_concurrency (usize, default 2) — max probes in flight at once during a sweep, so probing never stampedes upstreams
• consecutive_failure_threshold (u32, default 3) — consecutive probe failures before a provider is marked unhealthy
• recovery_success_threshold (u32, default 2) — consecutive successes before an unhealthy provider recovers
• status_page_interval_secs (u64, default 60) — seconds between provider status-page polls; only providers with a status_page_url are polled, and the poller runs even when enabled = false
• probes are jittered across the first quarter of the interval (per-provider stable offset), and a 429 on the probe itself pauses that provider’s probing with exponential backoff (1, 2, 4, 8 sweeps) without marking it unhealthy

[realtime]

Guardrails for persistent /v1/realtime WebSocket sessions. All limits are per gateway process; set a value to 0 to disable that limit.

• max_connections (u64, default 1000) — concurrent sessions admitted by this gateway instance
• max_session_secs (u64, default 3600) — hard session-duration limit
• idle_timeout_secs (u64, default 300) — closes a session when neither side sends a frame

[egress]

Where the gateway may send upstream traffic. A provider’s api_base (and any egress_proxy) is operator-supplied, so without a destination policy a crafted or compromised provider row turns the gateway into an SSRF primitive pointed at whatever sits inside its network position — cloud instance metadata being the classic target.

• block_link_local (bool, default true) — deny 169.254.0.0/16 and fe80::/10. This is the cloud instance-metadata range; leave it on unless you have a concrete reason
• block_loopback (bool, default false) — deny 127.0.0.0/8 and ::1. Off by default because sidecar and single-host deployments legitimately serve models on localhost
• block_private (bool, default false) — deny 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 and fc00::/7. Off by default because on-prem clusters are the common case
• allow_hosts (string[], default []) — hosts exempt from every check above, matched verbatim against the URL host (an IP literal or a hostname). The escape hatch for a deployment that must reach one otherwise-denied address

Enforcement happens twice. IP literals are classified during config validation, so a bad api_base is rejected at startup or when the control plane writes it. Hostnames are classified at connect time, against the address DNS actually returned — validation never resolves DNS itself (that would make config validation depend on a live resolver and break air-gapped deployments), and a connect-time check is what makes the policy DNS-rebinding-safe. A denied address fails the request; it is not silently retried elsewhere.

[egress]
block_link_local = true
block_private = true
allow_hosts = ["vllm.internal.example.com"]  # the one private upstream we mean to reach

[guardrails]

Built-in, zero-dependency regex guardrails and PII redaction, evaluated inside the gateway with no external service and no network hop. Disabled by default; a disabled or empty block adds no hot-path cost. Complements — never replaces — the custom guardrail webhook (ROL-257) and external PII engines (ROL-258).

• enabled (bool, default false) — master switch
• max_scan_bytes (usize, default 262144) — cap on total request text scanned per request; oversized content is forwarded unscanned so work stays bounded
• streaming_post_call (string, default reject) — what a streamed request does on a route that has post_call rules; see Output masking and streaming

Each [[guardrails.rules]] entry:

• name (string, required) — stable, unique; surfaced in metrics, never carries match text
• builtin (string) — one of email, phone, api_token, payment_card; or pattern (string) for a custom regex. Set exactly one.
• stage (string, default pre_call) — pre_call scans request content before proxying; post_call masks the response body before it reaches the client
• action (string, default annotate) — annotate (count only, forward unchanged), block (reject with an OpenAI-compatible guardrail_blocked error), or redact (replace each match with replacement)
• replacement (string) — redaction token; defaults to the built-in entity token (e.g. [REDACTED:EMAIL]) or [REDACTED]
• include_system (bool, default false) — also scan operator-authored system/developer messages; excluded by default

default_on was removed. It documented a per-request client opt-in that was never implemented, so it never selected anything: every configured rule applied regardless of its value. Existing configs that still set it keep loading — the key is ignored — and behaviour is unchanged. To turn a rule off somewhere, name it in that route’s [routes.advanced.guardrails] disable list (see [[routes]]).

Patterns use the linear-time (RE2-style) regex engine with no catastrophic backtracking, and are compiled under a bounded program size during config validation — an invalid or unbounded pattern fails at startup/snapshot validation, never on the request path. The request path never logs raw matched values; metrics expose rolter_guardrail_blocks_total and rolter_guardrail_redactions_total only.

Scanned surfaces: OpenAI /v1/chat/completions and /v1/responses (messages + input), /v1/completions (prompt), and Anthropic /v1/messages (system + messages). String, string-array, and typed text parts are all covered.

[guardrails]
enabled = true

[[guardrails.rules]]
name = "email"
builtin = "email"
action = "redact"

[[guardrails.rules]]
name = "card"
builtin = "payment_card"
action = "block"

[[guardrails.rules]]
name = "leaked-key"
builtin = "api_token"
stage = "post_call"
action = "redact"

Output masking and streaming

A post_call rule runs on the response body after the upstream replies and before the client sees it. redact rewrites the matched text in place; block withholds the completion and returns a 403 with code guardrail_blocked and the rule name — nothing was malformed and no upstream failed, so it is neither a 400 nor a 5xx, and it stays out of the range clients retry on.

Masked surfaces: choices[].message.content and choices[].text (OpenAI chat and legacy completions), the top-level content parts array (Anthropic /v1/messages), and output[].content[].text plus output_text (/v1/responses).

Tool-call arguments are scanned too. An address the model passes to a send_email tool has left the completion just as surely as one it printed. Arguments that arrive as a JSON document inside a JSON string (choices[].message.tool_calls[].function.arguments, the legacy function_call.arguments, and /v1/responses output[].arguments) are parsed, masked and re-encoded, so a replacement token containing a quote or a backslash cannot turn valid arguments into something the client fails to parse; non-string values are untouched. Anthropic tool_use parts are masked through their input object. Arguments that are not valid JSON are masked as plain text.

Output masking requires the whole completion. A match can straddle any number of token boundaries, so a rule redacting a@b.com cannot act on a frame holding only a@b, and buffering the full stream would remove the only property streaming has. streaming_post_call therefore decides what a streamed request does on a route with post_call rules:

• reject (default) — refuse with a 400 carrying code guardrail_streaming_unsupported, counted in rolter_guardrail_stream_rejections_total. The default fails closed: a masking rule that silently stops applying because the client passed "stream": true is the failure mode worth ruling out.
• passthrough — serve the stream with output rules not applied. pre_call rules still run on the request.

Non-streamed responses are buffered by the gateway when (and only when) a post_call rule applies to the route, so a route without them keeps its existing forwarding behaviour. Cached responses are stored as the upstream returned them and masked on every delivery, not once at store time — so a rule added after an entry was cached still applies to it, and an entry shared by two routes is masked per the route serving it. Output metrics: rolter_guardrail_output_redactions_total and rolter_guardrail_output_blocks_total.

[guardrail_webhook]

A vendor-neutral hook to a self-hosted semantic guardrail service (e.g. Guardrails AI, LLM Guard). Before proxying, the gateway POSTs a stable JSON envelope to the configured endpoint; the service replies with an allow/block/transform/annotate decision. Disabled by default; complements the built-in regex guardrails.

• enabled (bool, default false) — master switch
• url (string) — http(s) endpoint the envelope is POSTed to (required when enabled)
• stage (string, default pre_call) — pre_call inspects the request. post_call is validated but not yet enforced (output/SSE stage deferred).
• timeout_ms (u64, default 2000) — per-call timeout
• max_retries (u32, default 0) — extra attempts on a transient failure (connect/timeout/non-2xx)
• failure_mode (string, default fail_open) — fail_open forwards unchanged when the service is unreachable; fail_closed rejects with an OpenAI-compatible error
• max_body_bytes (usize, default 65536) — cap on the content forwarded; oversized content is sent as a truncated preview with truncated: true
• auth — optional credential resolved from the environment at call time, never inlined:
  • { bearer = { token_env = "GUARD_TOKEN" } } → Authorization: Bearer <env>
  • { shared_secret = { secret_env = "GUARD_SECRET" } } → X-Rolter-Guardrail-Secret: <env>

Contract. Request envelope: { direction, stage, model, route, trace_id, tenant: { org, team, project, key }, truncated, content }. Only these fields are sent; prompt content is never logged by the gateway. Response: { "action": "allow" | "block" | "transform" | "annotate", "content"?, "reason"?, "annotations"? }. An unrecognized or malformed decision defaults to allow (transport failures are governed by failure_mode). Metrics: rolter_guardrail_webhook_blocks_total, _transforms_total, _errors_total; the trace id is propagated in the X-Rolter-Trace-Id header.

[guardrail_webhook]
enabled = true
url = "https://guard.internal/check"
failure_mode = "fail_closed"
auth = { bearer = { token_env = "GUARD_TOKEN" } }

[prompt_templates]

Centrally-managed, versioned prompt templates and deterministic route decorators (ROL-256). Applications reuse approved system instructions through a named template without being granted arbitrary prompt-authoring privileges. Disabled by default; an empty or disabled block adds no hot-path cost.

• enabled (bool, default false) — master switch

Each [[prompt_templates.templates]] entry is one immutable version:

• id (string, required) — stable identifier surfaced in safe metadata, never in content logs
• version (u32, required, ≥ 1) — immutable version; the operator lists exactly the versions to activate. (id, version) must be unique.
• routes (array of string, default all) — public model names this template applies to; empty means every route
• [[prompt_templates.templates.variables]] — a named variable a decorator may reference as {{ name }}:
  • name (string, [A-Za-z_][A-Za-z0-9_]*)
  • required (bool, default false) — the caller must supply it; mutually exclusive with default
  • default (string) — value used when the caller omits it
• [[prompt_templates.templates.decorators]] — a message injected around the caller’s own messages:
  • role (string, default system) — system, assistant, or user
  • position (string, default prepend) — prepend (before the caller’s messages) or append (after), both in declared order
  • content (string) — message text, with optional {{ variable }} placeholders

Variables and escaping. Callers pass values in a rolter_template_vars object on the request body; it is always stripped before forwarding upstream. A caller value overrides the declared default; an unknown variable, a missing required variable, or an oversized value (variable > 4 KiB, rendered message > 16 KiB) is rejected with an invalid_prompt_template error. Substitution is structural: each rendered message is emitted as a JSON string through the serializer, never string-concatenated into raw JSON, so a variable value can never break out of its string or inject additional messages. Every {{ placeholder }} is validated at config-load time to reference a declared variable.

Surfaces and ordering. Applied to /v1/chat/completions, /v1/responses, and Anthropic /v1/messages. Prepend decorators wrap before, append after, preserving the caller’s own message order and semantics. For Anthropic, system decorators fold into the top-level system field (joined by blank lines); assistant/user decorators wrap the messages array. Surfaces without a chat message array (e.g. /v1/completions) are not decorated. The gateway applies only the configured immutable version from its reload-free snapshot. Applied template id/version and decoration count are recorded in safe metadata; metrics expose rolter_prompt_template_decorations_total and rolter_prompt_template_rejections_total.

[prompt_templates]
enabled = true

[[prompt_templates.templates]]
id = "support-preamble"
version = 3
routes = ["gpt-4o"]

[[prompt_templates.templates.variables]]
name = "persona"
default = "a helpful support assistant"

[[prompt_templates.templates.decorators]]
role = "system"
position = "prepend"
content = "You are {{persona}}. Follow the company policy and be concise."

A caller then supplies variables per request:

{
  "model": "gpt-4o",
  "messages": [{ "role": "user", "content": "hi" }],
  "rolter_template_vars": { "persona": "a billing specialist" }
}

When the control plane uses PostgreSQL, prompt templates can also be authored through the org-scoped /api/v1/orgs/{org_id}/prompt-templates endpoints. Draft versions are immutable once published, publication and rollback update the reload-free snapshot, and activation can be limited to an organization, project, route, or virtual key. Seed import creates missing versions idempotently and refuses to overwrite an existing immutable version with different content.

Organization skills use the adjacent /api/v1/orgs/{org_id}/skills endpoints. A skill has mutable metadata and access policy, immutable inline or reference-backed versions, explicit publish/rollback/retire lifecycle, and a deterministic /resolve/{slug} endpoint. Reference-backed versions accept https, git+https, oci, and s3 references without embedded credentials; secret-bearing metadata keys are rejected.

Environment variables

• ROLTER_CONFIG, ROLTER_HOST, ROLTER_PORT — gateway
• ROLTER_CONTROL_HOST, ROLTER_CONTROL_PORT, ROLTER_UI_DIR — control plane
• ROLTER_KEK — AES-256-GCM KEK for provider-secret encryption
• ROLTER_PUBLIC_URL — the control plane’s externally reachable base URL (default http://localhost:4001). The OIDC redirect URI is derived from it, so single sign-on needs it set correctly behind a proxy; see Single sign-on
• DATABASE_URL, REDIS_URL, CLICKHOUSE_URL — datastores
• RUST_LOG — tracing filter (e.g. info, rolter_gateway=debug)
• provider key vars referenced by api_key_env (e.g. OPENAI_API_KEY)

CLI flags override env, which override file values.

Secure configuration (rolter init and rolter check)

rolter has two deployment paths, and they are deliberately not converging.

Local (rolter easy-up) is frictionless and may stay loose on security: no database, no provider keys, no secrets at all, answering immediately on the built-in fake-llm model. It prompts for nothing, and rolter check does not change that.

Production is guided and should be hard to misconfigure. rolter check validates a production deployment before the process starts, so a misconfiguration fails loudly instead of starting degraded.

rolter init                       # generate a production config and its secrets
rolter check                      # validate the current environment
rolter check --config rolter.toml # also parse and validate the config file
rolter check --strict             # treat warnings as failures too
rolter check --connect            # also probe that the datastores accept a connection

The two are a pair. check tells an operator what is wrong; init tells them what to write. A test in the tree renders a fresh production environment and runs the full check suite over it, so what the generator emits is exactly what the gate accepts — if they ever drift, an operator would follow the documented path and still be told their deployment is broken.

It exits non-zero when anything fatal is found, which makes it usable as a container entrypoint step, a Kubernetes init container, or a Helm pre-install hook.

Why this exists

The motivating case is worth stating plainly, because it is the shape of every problem this command is meant to catch.

Provider credentials are encrypted at rest with a key-encryption key read from ROLTER_KEK. When that variable is missing, the control plane’s default-provider seed does not fail — it logs a warning and stores the inline api_key unsealed. Nothing goes red. The deployment serves traffic normally. A credential the operator believed was encrypted at rest simply is not, and there is no signal saying so.

Until recently several docs pages — including the production deployment guide and .env.example — told operators to set ROLTER_MASTER_KEY, a variable nothing in the codebase has ever read. An operator who followed those instructions exactly ended up in precisely that degraded state. Those pages are now corrected, and rolter check reports the mistake by name if the old variable is still set anywhere.

That is the class of failure this gate targets: not a crash, but a deployment that looks healthy while quietly not doing what it was configured to do.

What it checks

--connect is opt-in because a check that opens sockets cannot be the default for a command meant to run offline and without side effects. It is a TCP connect, not a protocol handshake: it needs no database driver and answers the question that actually goes wrong during a rollout — the name does not resolve, or nothing is listening. A datastore that accepts TCP but rejects the credentials is a different failure, and one the process reports loudly on its own.

Redis and the bind address are warnings rather than errors because a single-replica deployment behind an ingress is legitimately fine without either. Use --strict in an environment where they are not.

Anything resembling credentials in a URL is redacted before printing, since this output routinely lands in CI logs.

Generating the configuration (rolter init)

rolter check failing on a missing ROLTER_KEK and suggesting openssl rand -hex 32 still leaves an operator to invent the rest of the file around it. rolter init writes it.

rolter init                                  # rolter.toml + .env, production profile
rolter init --profile local                  # the loose defaults, for development
rolter init --print                          # stdout only, for piping into a secret manager
rolter init --env-only --database-url ...    # just the environment

It generates four distinct secrets — the KEK, the admin token, the internal token and the virtual-key pepper — each 256 bits from the system CSPRNG. They are distinct by construction and a test asserts it: reusing one value in two roles would mean a leaked admin token also decrypts every stored credential.

Nothing here is interactive. A generator that prompts cannot run in the Dockerfile, the Helm hook or the CI job where a production deployment is actually assembled, and an operator who has to answer six questions will hand-write a .env instead. Flags carry the choices and the defaults are the secure ones.

It will not overwrite without --force, and the refusal says why: regenerating ROLTER_KEK strands every credential encrypted under the old one. That failure is total, delayed, and looks like data corruption.

On unix the generated files are created 0600 at creation time rather than chmod’ed afterwards — a .env holding the KEK is a credential, and fixing the mode after the fact leaves a window.

What the profiles differ on

The data plane binds 0.0.0.0 under both: it is the public surface.

In a container

Run it as a pre-start step so the container never reaches a serving state while misconfigured:

CMD ["sh", "-c", "rolter check --strict && rolter control"]

Docker Compose

The bundled docker/docker-compose.yml is the local stack and is deliberately loose — example postgres credentials, no KEK, management plane wide open. Gating local bring-up on production rules would break the one path that is meant to have no friction, so the check lives behind a profile and never runs on docker compose up:

docker compose -f docker/docker-compose.yml --profile preflight \
               run --rm --env-file /path/to/production.env preflight

Kubernetes and Helm

The chart ships the init container already, enabled by default:

preflight:
  enabled: true
  strict: true   # a warning means the deployment works but is not what you meant
  connect: false # opt in to also probe the datastores

It renders with the exact env block the workload container gets — both come from one shared template, because three deployment paths that each re-implement “is this configured safely” will drift, and the one that drifts is the one nobody notices until a credential was stored unencrypted. A helm chart CI job lints the chart and renders every branch of that template.

If you are writing your own manifests, prefer an init container so the failure is visible as a distinct pod status rather than a crash-looping main container:

initContainers:
  - name: preflight
    image: ghcr.io/rolter-ai/rolter:latest
    command: ["/usr/local/bin/rolter", "check", "--strict"]
    envFrom:
      - secretRef:
          name: rolter-secrets

Custom CA bundles

Rolter can add private CA certificates to the normal public-root trust store for outbound HTTPS connections to upstream providers. Certificate-chain and hostname verification remain enabled; this feature does not affect inbound TLS or configure mTLS client certificates.

Minimal air-gapped configuration

Mount a PEM file containing one or more CA certificates, then use either the environment variable:

ROLTER_CA_BUNDLE=/etc/rolter/ca/private-root.pem rolter-gateway --config /app/rolter.toml

or the matching global TOML field:

[tls]
ca_bundles = ["/etc/rolter/ca/root.pem", "/etc/rolter/ca/intermediate.pem"]

[[providers]]
name = "private-vllm"
kind = "openai_compatible"
api_base = "https://llm.internal.example"

ROLTER_CA_BUNDLE replaces the global TOML list. A provider can replace the global selection independently:

[[providers]]
name = "isolated-cluster"
kind = "openai_compatible"
api_base = "https://llm.cluster.internal"
ca_bundles = ["/etc/rolter/ca/cluster-root.pem"]

Other providers keep using the global private roots plus the built-in public roots. Set a provider’s ca_bundles = [] to use public roots only.

Docker Compose

Mount the bundle read-only and pass its in-container path:

services:
  gateway:
    environment:
      ROLTER_CA_BUNDLE: /etc/rolter/ca/private-root.pem
    volumes:
      - ./pki/private-root.pem:/etc/rolter/ca/private-root.pem:ro

Kubernetes

Store the public CA certificate in a ConfigMap or Secret and mount it read-only:

apiVersion: v1
kind: ConfigMap
metadata:
  name: rolter-upstream-ca
data:
  private-root.pem: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rolter-gateway
spec:
  template:
    spec:
      containers:
        - name: gateway
          image: rolter:latest
          env:
            - name: ROLTER_CA_BUNDLE
              value: /etc/rolter/ca/private-root.pem
          volumeMounts:
            - name: upstream-ca
              mountPath: /etc/rolter/ca
              readOnly: true
      volumes:
        - name: upstream-ca
          configMap:
            name: rolter-upstream-ca

Validation and rotation

Startup fails with the bundle path and an actionable error when a configured file is missing, unreadable, contains no certificates, or has malformed PEM. Snapshot updates are rejected under the same conditions.

HTTP clients capture trust roots when their connection pool is created. After replacing a mounted certificate, publish or fetch a new configuration snapshot—even if the path is unchanged—to clear configured pools and rebuild them from the new bundle. With static bootstrap configuration, restart the gateway. Existing in-flight connections finish with their original trust configuration; subsequent connections use the rotated bundle.

Self-hosted Ollama

rolter supports a local or privately hosted Ollama daemon through Ollama’s OpenAI-compatible API. This provider does not require an API key.

Native setup

Install Ollama, start the daemon, and pull a small smoke-test model:

ollama serve
ollama pull qwen2.5:0.5b

Configure the daemon origin, without /v1 (rolter appends endpoint paths):

[[providers]]
name = "ollama-local"
kind = "ollama"
api_base = "http://localhost:11434"

[[routes]]
model = "local-qwen"
strategy = "round_robin"
[[routes.targets]]
provider = "ollama-local"
model = "qwen2.5:0.5b"

Start rolter and exercise model discovery, chat, legacy completions, embeddings, and streaming:

curl http://localhost:4000/v1/models
curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"local-qwen","messages":[{"role":"user","content":"hello"}]}'
curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"local-qwen","stream":true,"messages":[{"role":"user","content":"hello"}]}'
curl http://localhost:4000/v1/completions \
  -H 'content-type: application/json' \
  -d '{"model":"local-qwen","prompt":"hello"}'
curl http://localhost:4000/v1/embeddings \
  -H 'content-type: application/json' \
  -d '{"model":"local-qwen","input":"hello"}'

/v1/models lists rolter’s configured public route names, so the example returns local-qwen; it does not expose unrelated models installed in Ollama.

Docker setup

Containers must address Ollama by its Compose service name:

services:
  ollama:
    image: ollama/ollama:0.9.6
    volumes:
      - ollama-data:/root/.ollama

Use api_base = "http://ollama:11434" in the gateway container’s config. The opt-in smoke suite under integration/ollama/ provides a complete reproducible Compose setup and pulls qwen2.5:0.5b automatically.

Compatibility and known gaps

rolter passes OpenAI request JSON and response bodies through unchanged (apart from the configured model-name rewrite), preserving retry, cooldown, health, logging, error mapping, routing, and SSE semantics. Ollama currently documents chat and legacy completions, streaming, JSON mode (response_format), tools, vision message content, seed, and usage fields. The gateway also passes stream_options through, though Ollama may ignore unsupported options.

Support depends on the installed Ollama release and model: tool calling and vision require capable models, JSON schemas are not guaranteed to be obeyed by every model, and some OpenAI fields are accepted but ignored. Ollama’s OpenAI-compatible embeddings endpoint accepts models with embedding support; for production, route it to a dedicated embedding model. Ollama’s native /api/* endpoints and Ollama Cloud authentication are outside this provider’s scope.

OpenRouter

rolter’s openrouter provider targets OpenRouter’s OpenAI-compatible API while keeping OpenRouter model identifiers and routing controls intact.

Configuration

Create an API key in OpenRouter and expose it only through the environment:

export OPENROUTER_API_KEY='...'

[[providers]]
name = "openrouter"
kind = "openrouter"
api_base = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"

[[routes]]
model = "router-chat"
strategy = "round_robin"
[[routes.targets]]
provider = "openrouter"
model = "anthropic/claude-sonnet-4"

The public rolter model (router-chat) is rewritten only to the target model override. OpenRouter identifiers such as anthropic/claude-sonnet-4, including their provider prefix and optional variants, are otherwise forwarded verbatim. rolter fallback chooses another configured target after a retryable failure; OpenRouter’s own provider request object then controls routing among upstreams inside the selected OpenRouter target.

For example, this body preserves OpenRouter’s provider ordering and fallback policy:

{
  "model": "router-chat",
  "messages": [{"role": "user", "content": "hello"}],
  "provider": {
    "order": ["Anthropic", "Google"],
    "allow_fallbacks": true,
    "data_collection": "deny"
  }
}

Chat completions, SSE chunks, usage/cost fields, response metadata, and OpenRouter error JSON pass through without normalization. rolter still applies its normal authentication, route policy, retries, cooldowns, health checks, request logging, and routing headers. /v1/models lists configured rolter route aliases rather than exposing every model in OpenRouter’s catalog.

Attribution headers

OpenRouter recommends HTTP-Referer and X-Title for application attribution; they are not required for authentication. rolter omits them by default so it does not disclose deployment identity. Set either explicitly when desired:

export OPENROUTER_HTTP_REFERER='https://example.com'
export OPENROUTER_X_TITLE='Example gateway'

These values are forwarded only by openrouter providers. Never put API keys, user identifiers, or private internal hostnames in attribution headers.

Live smoke

The ignored live test makes a billable request and therefore requires both a credential and an explicitly selected model:

OPENROUTER_API_KEY=... ROLTER_OPENROUTER_LIVE_MODEL=openai/gpt-4.1-mini \
  cargo test -p rolter-gateway --test openrouter live_openrouter_smoke -- --ignored

Docker deployment

Compose (full stack)

docker/docker-compose.yml brings up Postgres, Redis, ClickHouse, and the rolter gateway + control services.

cp .env.example .env            # set OPENAI_API_KEY etc.
docker compose -f docker/docker-compose.yml up -d
docker compose -f docker/docker-compose.yml logs -f gateway

• Gateway: http://localhost:4000
• Control + UI: http://localhost:4001
• Postgres 5432, Redis 6379, ClickHouse 8123/9000

DB schemas auto-apply on first start, by two different routes. The Postgres schema is owned by sqlx::migrate!, which rolter-control and rolter-seed both run on startup; Compose deliberately does not mount migrations/ into the container’s docker-entrypoint-initdb.d, because letting Postgres apply those files itself would bypass sqlx’s _sqlx_migrations bookkeeping and make the next startup replay every migration against a populated database (#499). ClickHouse has no such runner, so clickhouse/ is mounted into its initdb directory and is that schema’s only provisioning path.

Image

The multi-stage docker/Dockerfile produces a slim Debian runtime with both binaries and the built UI at /app/ui/dist. Its default command is rolter easy-up, so one image serves the gateway and dashboard with the built-in fake-llm model — no compose file, provider key, or config mount required.

docker build -f docker/Dockerfile -t rolter:dev .
docker run --rm -p 4000:4000 -p 4001:4001 rolter:dev

Then open http://localhost:4001 and verify the data plane with:

curl -s http://localhost:4000/v1/chat/completions \
  -H 'Authorization: Bearer sk-rolter-dev' \
  -H 'Content-Type: application/json' \
  -d '{"model":"fake-llm","messages":[{"role":"user","content":"hello"}]}'

Override the command to run just the gateway or control plane:

docker run --rm -p 4000:4000 rolter:dev rolter-gateway --config /app/rolter.toml
docker run --rm -p 4001:4001 rolter:dev rolter-control

Published images

Release tags publish an image to GHCR (and, when configured, Docker Hub) under the same repo name and tags. Each release is tagged with its version and latest:

docker pull ghcr.io/<owner>/rolter:latest
docker pull ghcr.io/<owner>/rolter:0.0.4

Publishing is fail-closed and opt-in, mirroring the PyPI flow. The publish-docker job in .github/workflows/release.yml runs only when:

• repo variable DOCKER_PUBLISH_ENABLED = true, and
• the verify + external-check gates pass for the tagged commit.

GHCR always publishes via the built-in GITHUB_TOKEN. To also push to Docker Hub, set repo variable DOCKERHUB_IMAGE (e.g. docker.io/acme/rolter) and secrets DOCKERHUB_USERNAME / DOCKERHUB_TOKEN; the same tag set is applied to both registries. (Multi-arch images are a separate roadmap item — releases currently ship linux/amd64.)

Production notes

• Put the gateway behind TLS (ingress/load balancer); keep the control plane private.
• Set a strong ROLTER_KEK; provide DB/Redis/ClickHouse URLs via env or a secrets manager.
• Scale gateway horizontally; all replicas hot-reload config from Redis. ClickHouse and Postgres are shared.
• Kubernetes deployments are supported through the rolter Helm chart.

Air-gapped

Running fully offline behind an internal mirror (Nexus/Artifactory/Harbor)? See Air-gapped install & operation.

Kubernetes and Helm

The supported chart is in charts/rolter. It deploys separate gateway and control-plane workloads and services, with health probes, hardened pod defaults, resource requests, an optional HPA, disruption budget, and ingress.

helm upgrade --install rolter ./charts/rolter --namespace rolter --create-namespace \
  --set env.databaseUrl='postgres://rolter:secret@postgres.example/rolter' \
  --set env.redisUrl='redis://redis.example:6379'

PostgreSQL, Redis, and ClickHouse are external by design. Supply provider credentials through Kubernetes Secrets using secretEnv; do not place credentials in config.file or Helm values committed to source control. For GitOps-managed configuration, create a ConfigMap containing rolter.toml and set config.existingConfigMap.

The gateway defaults to two replicas. The control plane defaults to one replica; scale it only after validating the database migration and UI-hosting behavior for your deployment. Configure TLS at the ingress or service-mesh boundary.

llama.cpp (llama-server)

Rolter’s llama_cpp provider preset targets the OpenAI-compatible API exposed by llama-server. It needs no API key by default and works with local CPU or GPU GGUF deployments.

Start llama-server

Choose a GGUF whose license permits your intended use and whose quantization fits available RAM/VRAM. Q4_K_M is a practical starting point for local use; smaller quantizations use less memory at the cost of quality.

With a native llama.cpp build:

llama-server -m /models/model.gguf --host 0.0.0.0 --port 8080

Or with the upstream Docker image:

docker run --rm -p 8080:8080 -v "$PWD/models:/models" \
  ghcr.io/ggml-org/llama.cpp:server \
  -m /models/model.gguf --host 0.0.0.0 --port 8080

Configure Rolter

api_base is the server origin, without /v1. model on the target is the model identifier reported by llama-server; the public route can be a stable alias.

[[providers]]
name = "local-llama"
kind = "llama_cpp"
api_base = "http://127.0.0.1:8080"

[[routes]]
model = "local-chat"
strategy = "round_robin"

[[routes.targets]]
provider = "local-llama"
model = "model.gguf"

Rolter forwards /v1/chat/completions and /v1/completions, including SSE, sampling fields, grammar, and OpenAI response_format. /v1/models lists Rolter’s public route aliases. Routing headers, retries, cooldowns, and active health checks behave like other providers; the default health probe calls the upstream /v1/models endpoint.

llama.cpp-native routes such as /completion, /tokenize, /detokenize, and slot/metrics administration are intentionally not exposed by Rolter’s generic OpenAI API. Call llama-server directly for those endpoints.

Smoke test

With llama-server running and the model id from its /v1/models response:

integration/llama-cpp-smoke.sh http://127.0.0.1:8080 model.gguf

The script starts a temporary Rolter gateway, verifies model listing, non-streaming completion, SSE, and routing headers, then cleans up.

Hugging Face Text Embeddings Inference (TEI)

The tei provider targets TEI’s OpenAI-compatible POST /v1/embeddings endpoint. Self-hosted TEI is keyless by default; api_key or api_key_env can add bearer authentication when TEI sits behind an authenticated proxy.

Run TEI

For a reproducible CPU deployment with a small embedding model:

docker run --rm -p 8080:80 -v "$PWD/data:/data" \
  ghcr.io/huggingface/text-embeddings-inference:cpu-1.9 \
  --model-id sentence-transformers/all-MiniLM-L6-v2

On Apple Silicon, install and run the native server:

brew install text-embeddings-inference
text-embeddings-router \
  --model-id sentence-transformers/all-MiniLM-L6-v2 --port 8080

Configure Rolter

Use the server origin as api_base, without /v1:

[[providers]]
name = "tei-local"
kind = "tei"
api_base = "http://127.0.0.1:8080"

[[routes]]
model = "embed-local"
strategy = "round_robin"

[[routes.targets]]
provider = "tei-local"
model = "sentence-transformers/all-MiniLM-L6-v2"

Rolter preserves OpenAI string, string-array, token-array, and token-array-batch inputs, plus encoding_format, dimensions, user, embedding vectors, usage, and upstream error JSON. Normal routing headers, retries, cooldowns, logging, and health behavior apply. The default active probe uses TEI’s /health route.

Only /v1/embeddings is part of this adapter. TEI-native /embed, /rerank, /embed_sparse, /predict, /tokenize, /health, and /metrics are not exposed through Rolter’s generic OpenAI surface. Call TEI directly for them.

Smoke test

The opt-in Compose test starts TEI, downloads the small model, starts Rolter, and verifies batch embeddings, optional fields, usage, and routing headers:

integration/tei/run.sh

Air-gapped installation & operation

rolter is designed to run in fully air-gapped environments — no public internet at build or run time. The runtime is egress-free by default: it talks only to the backends you configure (upstream providers, Postgres, Redis, ClickHouse) and to nothing else. This page covers how to install rolter behind an internal mirror, what the runtime does and does not reach, and how to verify the deployment offline.

Runtime egress guarantees

rolter makes outbound network calls only to endpoints you configure:

• Upstream providers — the base_url of each configured provider.
• Postgres / Redis / ClickHouse — only when their URLs are set (DATABASE_URL / ROLTER_DATABASE_URL, REDIS_URL, clickhouse_url).
• Control-plane snapshot — only when the gateway is started with --snapshot-url (and Redis pub/sub only with --redis-url).
• OTLP traces — only when an OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is set. With no OTEL_* env, no exporter is built and there is zero tracing egress.

Everything else is self-contained:

• The interactive API reference at /docs embeds the Scalar JS bundle in the binary and sets withDefaultFonts: false, so it never reaches a CDN or fonts.scalar.com. This is asserted by the docs_page_is_self_contained and scalar_bundle_is_embedded tests in crates/rolter-gateway/src/openapi.rs.
• The dashboard SPA is served as static assets by the control plane; it loads no third-party scripts, fonts, or styles.

Caveat — status-page pollers. If a provider sets status_page_url, the gateway periodically fetches that URL as a secondary health signal. Leave status_page_url unset in air-gapped configs (or point it at an internal mirror) so the poller stays inside the enclave.

Install paths through a mirroring proxy

Air-gapped sites usually proxy public registries through an internal mirror (Sonatype Nexus, JFrog Artifactory, Harbor, …). Pick the path that matches how you ship rolter.

Docker image (recommended)

Pull through a registry that proxies GHCR/Docker Hub:

docker pull registry.internal.example/rolter/rolter:latest

Or transfer a fully offline image with docker save / docker load:

# on a connected host
docker pull ghcr.io/rolter-ai/rolter:latest
docker save ghcr.io/rolter-ai/rolter:latest -o rolter.tar

# copy rolter.tar into the enclave, then
docker load -i rolter.tar

PyPI wheel (uv tool install / pip)

Install through a Nexus/Artifactory PyPI proxy:

uv tool install rolter --index-url https://nexus.internal.example/repository/pypi/simple
# or
pip install rolter --index-url https://nexus.internal.example/repository/pypi/simple

Or install a downloaded wheel with no index at all:

uv tool install ./rolter-<version>-py3-none-any.whl
# or
pip install --no-index ./rolter-<version>-py3-none-any.whl

crates.io (cargo install)

Point Cargo at a registry mirror or vendored sources via .cargo/config.toml:

# .cargo/config.toml
[source.crates-io]
replace-with = "internal"

[source.internal]
registry = "sparse+https://nexus.internal.example/repository/cargo/"

For a fully offline build, vendor the dependency sources on a connected host and copy them in:

cargo vendor vendor/                 # connected host, writes a [source] snippet
# copy vendor/ into the enclave, add the printed snippet to .cargo/config.toml
cargo build --workspace --offline

Building from source (cargo + bun)

The Rust build follows the crates.io section above. The UI needs an internal npm mirror for bun install:

# ui/bunfig.toml
[install]
registry = "https://nexus.internal.example/repository/npm/"

# alternatively ui/.npmrc
registry=https://nexus.internal.example/repository/npm/

cd ui && bun install && bun run build
cargo build --workspace --release --offline

Operator checklist

Must be reachable inside the enclave:

• Every configured provider base_url (your internal model servers or a proxied provider endpoint).
• Postgres, Redis, and ClickHouse hosts — only for the features you enable.
• The control plane, if the gateway runs with --snapshot-url.

Must NOT be required:

• Public package registries at run time (only at install/build time, through the mirror).
• CDNs (cdn.jsdelivr.net, fonts.scalar.com, npm/unpkg) — rolter references none.
• Telemetry endpoints — unless you deliberately set OTEL_* to an internal collector.
• Provider status pages — leave status_page_url unset.

Offline smoke test

Verify a running gateway with zero external providers using the built-in fake-llm model (deterministic, no upstream or secrets needed):

rolter gateway --port 4000 &
curl -s http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"fake-llm","messages":[{"role":"user","content":"hello"}]}'

A 200 with a lorem-ipsum completion confirms the gateway serves traffic with no outbound calls. Open http://localhost:4000/docs and confirm the API reference renders with no network requests leaving the host.