rolter documentation

Start here. rolter is a high-performance OpenAI/Anthropic-compatible AI gateway and load balancer (Rust data plane + control plane, shadcn/ui dashboard).

Architecture

• Overview — system shape, crates, data/control plane split
• Load balancing — strategies and the cache-aware design
• Caching — response cache and KV-cache affinity
• Config & hot reload — reload-free updates
• Data model — tenancy, keys, pricing, budgets
• RBAC & auth — roles, virtual keys, SSO/LDAP roadmap
• Security — secret handling, threat model
• Observability — metrics, tracing, logs
• Performance — hot-path principles and targets

Decisions

• ADRs / decision log

API

• OpenAI & Anthropic surface

Development

• Setup
• Testing
• Contributing
• Commit conventions
• Packaging (uv / cargo / docker)

Deployment

• Zero-config quickstart — start with no keys, add providers at runtime
• Configuration reference
• Custom CA bundles — trust private upstream PKI without disabling TLS verification
• Docker

Planning

• Roadmap
• TODO

Architecture overview

rolter is split into a data plane (the hot proxy path) and a control plane (management, RBAC, config writes, UI host), sharing a set of Rust library crates. This separation lets the proxy stay lean and fast while the control plane handles CRUD, auth, and persistence.

Components

flowchart LR
  subgraph clients[Clients]
    C([OpenAI / Anthropic SDKs])
    A([Dashboard user])
  end
  C -->|/v1/*| GW["rolter-gateway<br/>data plane (Axum)"]
  A --> CTL["rolter-control<br/>control plane + UI host"]
  GW -->|forward + stream| UP["Upstreams<br/>OpenAI · Anthropic · vLLM pool"]
  CTL -->|read/write| PG[("PostgreSQL<br/>source of truth")]
  CTL -->|publish change events| RDS[("Redis / Valkey")]
  RDS -->|notify| GW
  GW -->|atomic snapshot swap| GW
  GW -->|async batched writes| CH[("ClickHouse<br/>logs + cost")]

Crates

• rolter-core — config model, domain types, error type, telemetry init. No I/O.
• rolter-balancer — LoadBalancer trait + strategies; pure and unit-tested.
• rolter-proxy — Forwarder over pooled reqwest clients; header injection, model rewrite, streaming, per-provider egress proxy.
• rolter-store — repository traits + in-memory impl; Postgres/Redis/ClickHouse backends land behind features.
• rolter-auth — virtual-key verification, roles, access checks.
• rolter-gateway (bin) — the data plane.
• rolter-control (bin) — the control plane + static UI host.

Data plane (rolter-gateway)

Built on Axum/Hyper/Tower. Holds an ArcSwap<Snapshot> routing table; reads are lock-free. Per request it:

1. parses the JSON body enough to read model and stream
2. authenticates the virtual key (when keys are configured) and checks model access
3. resolves the route, asks the route’s balancer to pick a target, then observes it (cache-aware learning)
4. forwards to the upstream provider with the right auth header, rewriting the model id
5. streams the response body straight back with minimal copying

See data plane details in performance.md.

Control plane (rolter-control)

Hosts the management REST API consumed by the dashboard and serves the built SPA as static assets (ServeDir). It owns all writes to Postgres, enforces RBAC, and publishes a config-change event to Redis so gateways hot-swap their snapshot without a restart (see config-and-hot-reload.md).

Datastores

• PostgreSQL — source of truth for tenancy, RBAC, providers, routes, virtual keys, pricing, budgets.
• Redis / Valkey — response cache, rate-limit counters, cooldown state, and pub/sub for config propagation.
• ClickHouse — high-volume request and cost logs, queried by the dashboard for usage analytics.

Request lifecycle (sequence)

sequenceDiagram
  participant Client
  participant Gateway
  participant Balancer
  participant Upstream
  participant ClickHouse
  Client->>Gateway: POST /v1/chat/completions
  Gateway->>Gateway: auth virtual key + model access
  Gateway->>Balancer: pick(route, ctx)
  Balancer-->>Gateway: target index
  Gateway->>Upstream: forward (Bearer/x-api-key, rewritten model)
  Upstream-->>Gateway: response stream
  Gateway-->>Client: streamed body
  Gateway--)ClickHouse: async usage + cost log

Extensibility

• New balancing strategy: implement rolter_balancer::LoadBalancer, wire into build().
• New provider protocol: extend ProviderKind and rolter-proxy translation.
• New storage backend: implement rolter-store traits behind a cargo feature.
• New modality (audio/image/video): add endpoints + provider adapters; the balancer/auth/logging layers are modality-agnostic.

Load balancing

Each route maps a public model name to one or more upstream targets and a strategy. Strategies implement rolter_balancer::LoadBalancer:

#![allow(unused)]
fn main() {
pub trait LoadBalancer: Send + Sync {
    fn name(&self) -> &'static str;
    fn pick(&self, ctx: &RouteContext, loads: &[u64]) -> Option<usize>;
    fn observe(&self, target: usize, ctx: &RouteContext) {}
}
}

pick returns an index into the route’s targets; observe lets learning strategies (cache-aware) record what a target served. RouteContext carries an optional session_key (from x-session-id) and the request prompt used for affinity scoring.

Strategies (v1)

• round_robin — sequential rotation; predictable, zero state.
• random — uniform random; good for simple homogeneous pools.
• power_of_two — pick the less loaded of two random targets; needs a load snapshot.
• consistent_hash — hash-ring keyed by session_key (falls back to prompt hash); pins a session/user to a target for KV reuse, survives target changes with minimal reshuffle (160 vnodes).
• cache_aware — approximate prefix affinity; see caching.md.
• weighted — smooth weighted round-robin honouring each target’s weight.
• pipeline — composable filter → weighted-score → argmax selection: eligibility filtering drops ineligible targets, then a stack of Scorers (session affinity + static weight + in-flight load + prefix-cache affinity) is combined as a weighted sum and the argmax wins (ties broken randomly). Session affinity pins repeat requests from the same x-session-id to their last-served target (TTL-bounded) for warm-cache reuse. The extension point every future cost/latency/KV-cache scorer plugs into.

Choosing a strategy

Roadmap

The trait is the extension point. Planned strategies:

• precise cache-aware — subscribe to vLLM KV-cache events (ZMQ), index block hashes, score targets by resident-prefix fraction blended with load. Requires vLLM ≥ 0.10 with matching --block-size / hash seed.
• lmcache-aware — query an LMCache controller for real cache occupancy.
• latency-based / cost-based — shipped as fastest (per-target request-latency EWMA read live at pick time) and cheapest (catalog price per target), both with a load tiebreak.
• weighted selection honoring Target.weight.
• health/circuit breaking + cooldowns — skip unhealthy targets; exponential backoff on 429/5xx.

Live per-target load (loads) will be fed from in-flight counters and upstream health so power_of_two and cache_aware balance against real pressure.

Caching

rolter deals with two distinct kinds of caching.

1. KV-cache affinity (load balancing)

The big win for self-hosted fleets: vLLM/SGLang reuse the attention KV cache for shared prompt prefixes (system prompts, few-shot examples, conversation history). But that only helps if the next matching request lands on the same replica. Naive round-robin scatters related requests and destroys cache locality.

The cache_aware strategy keeps, per target, a byte trie of prompts it has served. For an incoming prompt it computes the fraction of leading bytes already present on each target and:

• if the best match ≥ threshold (default 0.5), pins the request to that target (cache hit)
• otherwise spreads to the least-warmed target (or least loaded once load is wired)

This is approximate (no coupling to the engine). The per-target trie is capped at a node ceiling (default 1M nodes) with LRU eviction: inserting past the cap drops the least-recently-inserted prompt, pruning only the nodes that become unreferenced (shared prefixes survive). Each trie exposes an eviction counter for observability.

flowchart TD
  R[incoming prompt] --> S{best prefix match >= threshold?}
  S -- yes --> P[pin to best target<br/>cache hit]
  S -- no --> L[least-warmed / least-loaded target]
  P --> O[observe: insert prompt into target trie]
  L --> O

Precise mode (roadmap)

Subscribe to vLLM KV-cache events over ZMQ, hash blocks the same way vLLM does (--block-size, hash seed), and maintain a global block→target index. Score targets by exact resident-prefix fraction blended with live load. This mirrors llm-d’s precise prefix-cache-aware scheduling and gives the largest, most reliable TTFT/throughput wins on prefix-heavy workloads.

2. Response cache (roadmap)

Optional caching of full responses to cut cost/latency for repeated requests:

• exact: hash of the normalized request → cached response (Redis), short TTL, opt-in per route/key.
• semantic: embed the prompt, match by cosine similarity above a threshold; requires an embeddings provider.

Streaming responses are cached on completion and replayed as a synthetic stream. Cache status is surfaced via response headers (e.g. x-rolter-cache: hit|miss).

Configuration & hot reload

A core requirement: operators change routes, providers, keys, limits and pricing from the UI and have them take effect without restarting the gateway.

Sources of config

1. Bootstrap file (rolter.toml) — used for first run, local dev, and IaC. Maps to rolter_core::GatewayConfig.
2. Database (Postgres) — the runtime source of truth once the control plane is running. The control plane composes a GatewayConfig-equivalent snapshot from normalized tables.

Propagation

sequenceDiagram
  participant UI
  participant Control as rolter-control
  participant PG as PostgreSQL
  participant Redis
  participant GW as rolter-gateway
  UI->>Control: PUT /api/v1/routes/... (RBAC checked)
  Control->>PG: write change in a transaction
  Control->>PG: bump config_version.version
  Control->>Redis: PUBLISH rolter.config {version}
  Redis-->>GW: message {version}
  GW->>Control: GET /internal/snapshot?version=N
  Control-->>GW: full snapshot (JSON)
  GW->>GW: build Snapshot, ArcSwap::store (atomic)

• The gateway keeps the routing table in an ArcSwap<Snapshot>. Swapping is atomic and wait-free for readers — in-flight requests keep using the old snapshot; new requests see the new one.
• Versioning: config_version in Postgres is the monotonic source of truth. The gateway also reconciles on an interval (and at startup) so a missed pub/sub message self-heals.
• Validation: the control plane validates a snapshot (every route target references a known provider, etc.) before bumping the version, so gateways never load a broken config.

Why this design

• Redis pub/sub gives near-instant fan-out to many gateway replicas.
• Postgres versioning makes the system correct even if Redis drops a message.
• ArcSwap keeps the hot path lock-free; no read ever blocks on a config write.

Alternatives considered: Postgres LISTEN/NOTIFY (avoids a Redis dependency but Redis is already needed for cache/rate limits), and pure polling (simplest, higher latency). See ADR.

Data model

PostgreSQL is the source of truth. The initial schema lives in migrations/0001_init.sql; ClickHouse log schema in clickhouse/001_logs.sql.

Tenancy

erDiagram
  orgs ||--o{ teams : has
  teams ||--o{ projects : has
  projects ||--o{ routes : has
  routes ||--o{ route_targets : has
  projects ||--o{ virtual_keys : has
  orgs ||--o{ providers : has
  providers ||--o{ provider_keys : has
  users ||--o{ memberships : has

• Org → Team → Project → Virtual Key is the hierarchy. Budgets and rate limits attach at any scope and combine most-restrictive-wins.
• Providers are owned at the org level and referenced by route targets. Upstream credentials live in provider_keys, envelope-encrypted (see security.md).
• Routes belong to a project and map a public model to route_targets with a strategy.
• Virtual keys belong to a project, store only a hash of the key plus a display prefix, and carry an optional model allow-list.

Cost & limits

• model_prices — USD per million tokens for input/output (+ cached input). Used to compute cost_usd per request, written to ClickHouse.
• budgets — spend caps per scope and period; enforced before forwarding and refreshed from spend aggregates.
• rate_limits — RPM/TPM per scope; counters live in Redis for multi-instance correctness.

Config versioning

config_version holds a single monotonic counter the gateways watch for reload-free updates (config-and-hot-reload.md). audit_log records who changed what.

Mapping to the gateway

The control plane composes the normalized tables into the same shape as rolter_core::GatewayConfig (providers + routes + virtual_keys), which the gateway turns into an immutable Snapshot.

RBAC & authentication

Two distinct auth surfaces:

1. Gateway (data plane) — virtual keys

Clients call /v1/* with a virtual key (Authorization: Bearer <key> or x-api-key). The gateway:

• looks the key up in the current snapshot
• checks the key’s model allow-list (empty = all)
• (roadmap) enforces budgets and RPM/TPM limits for the key’s scope chain

Keys are stored as hashes; the presented key is compared in constant time (rolter_auth::verify_key).

2. Control plane (dashboard) — users + roles

Human users authenticate to the control plane. v1 ships local accounts (argon2id password hashes). RBAC roles:

• admin — full control within scope (manage providers, routes, keys, members, budgets)
• member — create/edit routes and keys within scope
• viewer — read-only (dashboards, logs)

Roles are granted via memberships at an org / team / project scope. Permission checks resolve the most specific membership for the target resource.

flowchart LR
  U[User] -->|member of| Scope[org / team / project]
  Scope -->|role| Caps[admin / member / viewer]
  Caps --> Action{allowed?}

Roadmap

• OAuth2 / OIDC SSO — pluggable IdentityProvider; map IdP groups → roles.
• LDAP — bind + group mapping for enterprise directories.
• JWT service auth and short-lived tokens.
• Audit log surfaced in the UI.
• Optional constant-time map / pepper for virtual-key lookup hardening.

Security

Secret handling

• Upstream provider keys are never stored in plaintext in the database. They are envelope-encrypted with AES-256-GCM: a per-record data key/nonce, wrapped by a master key (KEK) supplied via ROLTER_MASTER_KEY (env/file). Pluggable backends (HashiCorp Vault, cloud KMS) are a roadmap item.
• In the bootstrap file, prefer api_key_env over inline api_key so secrets stay in the environment, not on disk.
• Virtual keys are stored as hashes with a short display prefix; the raw key is shown once at creation.
• Secrets are never logged. The gateway redacts auth headers from traces.

Transport

• Upstream calls use rustls (no OpenSSL). HTTP/2 keep-alive with connection pooling.
• Optional per-provider egress proxy (egress_proxy, HTTP/HTTPS/SOCKS5) for networks where providers aren’t directly reachable.
• Optional global or per-provider custom CA bundles add private PKI roots to outbound upstream clients while retaining public roots, certificate-chain validation, and hostname verification.
• Terminate TLS at the gateway or a fronting proxy/ingress in production.

Wire transparency

• Outbound requests to upstream providers carry no rolter-identifying marks: no User-Agent, no added X-*/Via headers, no metadata injected into the JSON body, no marks in SSE framing. The only headers sent are functionally required ones — content-type, the provider’s auth header, and anthropic-version for Anthropic.
• Responses back to clients likewise gain no rolter-added headers.
• This is a tested guarantee: golden wire tests in rolter-proxy capture the raw outbound request head and fail on any unexpected header (see openai_wire_carries_no_rolter_signature).

Threat model (high level)

• Tenant isolation: virtual keys are scoped to a project; model allow-lists prevent access to unconfigured models; cache keys are namespaced to avoid cross-tenant cache poisoning.
• Abuse: RPM/TPM rate limits and budgets bound spend and load (roadmap enforcement).
• AuthZ: control-plane mutations are RBAC-checked and recorded in audit_log.
• Supply chain: cargo deny/advisory scanning in CI is a roadmap item.

Operational guidance

• Always set a strong ROLTER_MASTER_KEY (e.g. openssl rand -hex 32) and rotate provider keys periodically.
• Run the control plane on a private network; expose only the gateway publicly.
• Back up Postgres; treat the master key as the most sensitive secret.

Observability

Metrics

• The gateway exposes Prometheus metrics at GET /metrics: counters (rolter_requests_total, rolter_upstream_errors_total, rolter_auth_failures_total, reload/log/budget/rate-limit/retry/cooldown/health/breaker/scrape counters), a rolter_config_version gauge, and per-model latency histograms — rolter_request_latency_ms (total) and rolter_request_ttft_ms (time-to-first-token), each labelled {model=...} with the standard _bucket/_sum/_count series. Histograms are observed once per completed request from the log sink, off the response hot path.
• The exporter is hand-rolled (atomic counters + non-cumulative histogram buckets cumulated at render) rather than the metrics facade + global recorder, which does not fit the lock-free arc-swap design where an explicit Arc<Metrics> is threaded through the request path.
• Passive per-target SLA signal: rolter_target_requests_total{provider,target,outcome} (a counter, outcome = ok for 2xx else error) is tallied once per completed request from the log sink — free, derived from real traffic, no extra upstream calls. A per-target error rate / uptime is sum(rate(rolter_target_requests_total{outcome="error"}[5m])) / sum(rate(rolter_target_requests_total[5m])). This is the first slice of provider stability tracking (ROL-123); the ClickHouse provider_health_events table and the dashboard land in later slices. The active prober is guarded: bounded probe concurrency with per-provider jitter, consecutive-failure/-recovery thresholds gating the unhealthy flip (no single-probe flapping), and exponential probe backoff when a probe itself gets a 429.
• Multi-key providers: rolter_key_cooldowns_tripped_total counts api keys parked after a key-level failure (429/401 on a provider with several keys); the request retries in-flight on a sibling key.
• A/B attribution: rolter_variant_requests_total{model,variant} (a counter) tallies requests per chosen variant, so traffic splits are visible in Prometheus/Grafana without querying ClickHouse. Classic single-pool routes (no variant) emit nothing. Observed from the same log-sink funnel (ROL-195, part of ROL-188).
• Roadmap: add per-provider/route labels on the histograms, in-flight gauges, cache-hit ratio, and circuit-breaker state gauges.
• Roadmap: scrape/federate upstream engine metrics from vLLM/SGLang/TGI /metrics and correlate them per target (queue depth, KV-cache usage, running/waiting requests) to feed load- and cache-aware routing and the dashboard.

Tracing & context propagation

• tracing + tracing-subscriber with RUST_LOG filtering; TraceLayer logs each HTTP request.
• Inbound: accept W3C traceparent/tracestate (and b3) from clients and continue the trace; honor x-request-id / x-correlation-id.
• Outbound to engines: inject the active trace context into upstream requests so vLLM/SGLang/TGI spans join the same distributed trace. vLLM and SGLang support OpenTelemetry tracing (e.g. vLLM --otlp-traces-endpoint); point them at the same OTLP collector so engine prefill/decode spans line up with rolter’s request span.
• A per-request request_id is echoed in a response header and stamped on logs, metric exemplars and spans for correlation.

Exporters (OTel-compatible)

rolter emits traces and metrics via OpenTelemetry OTLP (gRPC/HTTP), so any OTel-compatible backend works without code changes — just set an endpoint and headers:

• SigNoz, Grafana Tempo/Mimir, Honeycomb, Datadog (OTLP intake or the OTel Collector datadog exporter).
• Langfuse for LLM-specific observability (prompt/response, token usage and cost as traces), ingested via its OTLP endpoint or SDK.

Recommended topology: rolter → OpenTelemetry Collector → fan-out to the chosen backends. The collector also scrapes the upstream engines’ /metrics and rolter’s /metrics, keeping vendor specifics out of rolter. Configure via env, e.g. OTEL_EXPORTER_OTLP_ENDPOINT, OTEL_EXPORTER_OTLP_HEADERS, OTEL_SERVICE_NAME=rolter-gateway.

Request & cost logs

• Every proxied request is logged to ClickHouse (request_logs): identifiers, model, provider/target, status, token counts, cost_usd, latency, TTFT, cache flag, error.
• Request id / trace continuation: every request carries an x-request-id — the caller’s when supplied, otherwise a generated UUID — which is echoed on the response and stored on the log row for end-to-end correlation. An inbound W3C traceparent or B3 (b3 / x-b3-traceid) header is parsed and its trace id stored in request_logs.trace_id, so gateway logs join the caller’s distributed trace instead of starting a disconnected one.
• Outbound propagation: when the caller sent trace context, it is forwarded verbatim to the chosen upstream (traceparent, tracestate, and the b3 / x-b3-* family) so vLLM/SGLang/TGI continue the same trace. An untraced request adds nothing to the upstream wire — this is the caller’s own context, not a rolter fingerprint, so it preserves wire transparency.
• Writes are async and batched off the hot path so logging never adds request latency.
• The dashboard queries ClickHouse for usage, spend, latency percentiles and error rates, sliced by org/team/project/key/model.

Provider health events

• Every health signal is written to ClickHouse (provider_health_events): target_id, provider, source, outcome, status_code, latency_ms, error_kind, timestamped by ClickHouse on insert.
• source distinguishes where the observation came from: passive (real traffic completing through the request funnel), probe (active liveness sweeps), and the opt-in llm_call / status_page sources.
• outcome is ok / error / timeout; error_kind gives a coarse label (rate_limited, upstream_error, connect_error, timeout).
• Writes reuse the same async, batched, off-hot-path writer and ClickHouse endpoint as request_logs; when no clickhouse_url is configured the sink is a no-op.
• Counters rolter_health_events_written_total and rolter_health_events_dropped_total track the writer, mirroring the request-log counters.
• This event stream feeds uptime %/MTTR rollups and the dashboard health panel.

Stability rollup API

Read-only, window-bounded rollups over provider_health_events, served by the control plane when --clickhouse-url is set (otherwise 503). All accept since/until (RFC3339, default last 7 days); time bounds are passed as ClickHouse query parameters, never interpolated.

• GET /api/v1/health/uptime — per provider/target: event counts, uptime, failure_rate, error_budget_burn and sla_breached against an sla target (query param, fraction in (0,1], default 0.99), and last_event.
• GET /api/v1/health/mttr — per provider/target mean time to recovery (mttr_seconds) and incident count, computed from downtime episodes (a run of non-ok events bounded by ok).
• GET /api/v1/health/timeline?bucket=hour|day|week|month — bucketed ok/error/timeout counts per provider/target for the failure timeline (default bucket hour).

Health

• GET /healthz on both binaries for liveness/readiness probes.

Performance

Goal: beat the reference Python proxy (LiteLLM cites ~8ms P95 added latency at 1k RPS) with a much smaller per-request overhead in Rust.

Hot-path principles

• Lock-free config reads — the routing table is an ArcSwap<Snapshot>; readers never block, even during a hot reload.
• Minimal-copy streaming — upstream responses are piped to the client as a Body::from_stream over the reqwest byte stream; rolter does not buffer whole responses.
• Connection reuse — pooled reqwest clients with HTTP/2 keep-alive and tcp_nodelay; one client per egress-proxy target, cached.
• Cheap auth — virtual-key lookup is an O(1) hash-map hit on the in-memory snapshot.
• Avoid full deserialization — only the fields needed for routing (model, stream) are read; the body is forwarded as raw bytes.
• Logging off the hot path — usage/cost rows are batched and written to ClickHouse asynchronously.
• Release profile — lto = "thin", codegen-units = 1, strip = true.

Things to watch

• The approximate cache-aware trie is per-route in-memory state; bound its size with eviction before it grows large.
• Per-request JSON parse for model is small but measurable; consider a fast path / partial parse for very high RPS.
• Prefer bytes::Bytes (ref-counted) over Vec<u8> copies when rewriting the model field.

Benchmarking (roadmap)

• Add a criterion micro-bench for balancer pick.
• Add an end-to-end load test (e.g. oha/k6) against a mock upstream to measure added latency and max RPS per core.
• Track TTFT and total-latency histograms in Prometheus and watch them in CI perf runs.

Inference engines

The ROL-238 suite checks rolter against OpenAI-compatible engine servers. Output is intentionally meaningless; this validates the HTTP, OpenAI JSON, and SSE contracts rather than model quality.

The default engine, sim, is llm-d-inference-sim: a ~30MB multi-arch vLLM API simulator that needs no model downloads and boots in milliseconds, which makes the suite cheap enough to run as a regular PR check. The real CPU vLLM and SGLang profiles remain for on-demand runs; they use trl-internal-testing/tiny-random-LlamaForCausalLM only for its configuration/tokenizer and initialize random weights with --load-format dummy (with a head_dim=64 override, since the CPU attention kernels reject the model’s native head_dim=4). The CPU vLLM profile uses eager execution to avoid expensive compilation warm-up during CI smoke runs.

It runs on CPU in Docker and therefore works on GitHub-hosted runners. Each engine profile starts two independent dummy upstreams so the gateway exercises a real target pool. Run one engine locally:

just integration-sim
just integration-vllm
just integration-sglang

Each command boots routes for every balancing strategy (round-robin, random, power-of-two, consistent-hash, cache-aware, weighted, pipeline, cheapest, and fastest). It verifies /v1/models, non-streaming chat, and SSE both directly and through rolter, and explicitly confirms round-robin reaches both targets. Logs are kept in artifacts/engines/<engine>/.

Local end-to-end run

just integration-vllm and just integration-sglang start two CPU dummy-weight servers, render the gateway configuration, and run the OpenAI JSON and SSE assertions. The runner cleans up all containers and child processes on exit and preserves the combined engine log plus the gateway log under artifacts/engines/<engine>/.

For manual inspection, start the selected two-server pool and leave it running:

docker compose -f docker/docker-compose.engines.yml --profile vllm up -d
# use profile sglang for ports 30000 and 30001

Render the gateway configuration in another terminal and start rolter:

config=$(mktemp)
sed \
  -e 's/__ROLTER_PORT__/4010/g' \
  -e 's/__ENGINE_1_PORT__/8000/g' \
  -e 's/__ENGINE_2_PORT__/8001/g' \
  integration/engines/rolter-dummy.toml.in >"$config"
cargo run -p rolter-gateway -- --config "$config"

Verify non-streaming JSON and streaming SSE through the gateway:

curl -i http://127.0.0.1:4010/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"dummy-round-robin","messages":[{"role":"user","content":"Reply with one token."}],"max_tokens":1,"temperature":0}'

curl -N http://127.0.0.1:4010/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"dummy-round-robin","stream":true,"messages":[{"role":"user","content":"Reply with one token."}],"max_tokens":1,"temperature":0}'

Exercise all configured strategies; each request must return a non-empty OpenAI choices array:

for model in dummy-round-robin dummy-random dummy-power-of-two \
  dummy-consistent-hash dummy-cache-aware dummy-weighted dummy-pipeline \
  dummy-cheapest dummy-fastest; do
  curl -fsS http://127.0.0.1:4010/v1/chat/completions \
    -H 'content-type: application/json' \
    -d "{\"model\":\"$model\",\"messages\":[{\"role\":\"user\",\"content\":\"ping\"}],\"max_tokens\":1}" \
    | jq -e '.choices | length > 0' >/dev/null
done

Clean up the pool and temporary gateway configuration:

docker compose -f docker/docker-compose.engines.yml --profile vllm down --volumes
rm -f "$config"

For non-gating direct-versus-gateway samples, run just bench-vllm or just bench-sglang. They record non-streaming and streaming p50/p95/p99 latency and streaming first-byte time in JSON. Results only compare runs on the same host, CPU image, engine versions, and host configuration; throughput thresholds are deliberately not merge gates.

The engine integration workflow runs the sim smoke as a regular check on pull requests that touch engine paths. Dispatch it manually with engine=vllm to smoke the real CPU engine (Actions tab, or gh workflow run "engine integration" -f engine=vllm). SGLang remains available through the local just integration-sglang command, but its source-built CPU image is currently too heavy for the shared CI gate. This suite is for compatibility, not a performance gate. When ROL-67 lands, add the equivalent /v1/messages assertion through the gateway.

Architecture Decision Records

Lightweight decision log. Each entry: Status · Context · Decision · Consequences. Supersede rather than rewrite.

ADR-0001 — Rust + Axum/Hyper/Tower for the data plane

Accepted. Need maximum proxy throughput with rich API semantics and SSE streaming. Chose Axum/Hyper/Tower on Tokio over Pingora/Actix for ecosystem fit and ergonomics. Consequence: idiomatic async stack; revisit Pingora only if profiling demands it.

ADR-0002 — Two-binary topology over shared crates

Accepted. Keep the hot proxy path lean and independently scalable from management. rolter-gateway (data plane) and rolter-control (management + UI host) share library crates. Consequence: clear seam; some duplicated wiring.

ADR-0003 — Vite + React + shadcn/ui SPA, Bun toolchain, served by Rust

Accepted. shadcn/ui (Radix + Tailwind) for the dashboard, built with Vite and managed with Bun; output is static assets served by rolter-control (no Node runtime in prod). Consequence: simple prod footprint; Bun used for install/dev/build.

ADR-0004 — Postgres + Redis + ClickHouse

Accepted. Postgres = source of truth (config/RBAC/keys/pricing); Redis = cache + rate-limit counters + config pub/sub; ClickHouse = high-volume request/cost logs. No SQLite. Consequence: three datastores to operate; each fits its job.

ADR-0005 — Org → Team → Project → Virtual Key tenancy

Accepted. Budgets and rate limits attach at any scope, most-restrictive-wins. Consequence: flexible multi-tenancy; enforcement must resolve a scope chain.

ADR-0006 — Local accounts + virtual keys + roles; SSO/LDAP later

Accepted. v1 ships local accounts (argon2id) and roles admin/member/viewer; OAuth2/OIDC and LDAP arrive as pluggable identity providers. Consequence: usable day one without an IdP.

ADR-0007 — Approximate cache-aware balancing behind a pluggable trait

Accepted. v1 uses an approximate per-target prefix trie (no engine coupling) behind LoadBalancer. Precise (KV-events) and lmcache-aware land later without API changes. Consequence: immediate wins; precise mode is additive.

ADR-0008 — Reload-free config: Postgres truth + Redis pub/sub + ArcSwap

Accepted. Control plane writes Postgres, bumps a version, publishes on Redis; gateways fetch and atomically swap an in-memory snapshot, reconciling by version. Consequence: instant fan-out, self-healing, lock-free reads.

ADR-0009 — Envelope encryption for provider secrets

Accepted. Upstream keys are AES-256-GCM envelope-encrypted with a master key from env/file; Vault/KMS backends later. Consequence: no plaintext secrets at rest; master key is the critical secret.

ADR-0010 — Packaging: maturin (uv) + cargo + Docker

Accepted. Ship a maturin-built PyPI wheel bundling the unified rolter launcher (uv tool install rolter), cargo install rolter, and a multi-stage Docker image. The rolter binary dispatches to gateway/control subcommands so one wheel/crate ships the whole system. Consequence: three distribution paths from a single named artifact.

ADR-0011 — API surface v1

Accepted. OpenAI /v1/chat/completions, /v1/completions, /v1/models and Anthropic /v1/messages. Embeddings, images, audio and other modalities follow. Consequence: drop-in for the two dominant client SDKs first.

ADR-0012 — Conventional Commits + CI PR-title lint

Accepted. Commit messages and PR titles follow Conventional Commits; enforced by commitlint, conventional-pre-commit, and a CI PR-title check. Consequence: consistent history, automatable changelogs/releases.

ADR-0013 — OpenTelemetry-based observability with engine propagation

Accepted. Export traces/metrics via OTLP to any compatible backend (SigNoz, Datadog, Grafana, Langfuse, …); propagate W3C trace context to vLLM/SGLang so engine spans join the same trace; federate upstream engine metrics. Consequence: vendor-neutral observability; an OTel Collector is the recommended hub.

ADR-0014 — Extensible API protocol translation

Accepted. Resolve translation by client/upstream protocol pair in rolter-proxy, including incremental SSE, while the gateway retains transport, caching and accounting ownership. Consequence: new provider dialects extend one translation boundary; non-equivalent modalities remain explicit and are never silently dropped.

ADR-0015 — Трансляция OpenAI Responses API

Development. Add OpenAI Responses as a protocol pair for native OpenAI, Chat Completions and Anthropic Messages, while model-less lifecycle operations remain uniformly unsupported until tenant-scoped storage exists.

ADR-0016 — Маршрутизация ресурсов OpenAI Responses по tenant-scoped registry

Development. Pin Responses lifecycle operations to a bounded tenant-scoped process-local record, preserving the original provider credential while making unknown and cross-tenant IDs indistinguishable.

ADR-0017 — Provider/model addressing to disambiguate identical model names

Accepted. First-class provider-slug/model addressing coexists with named routes: a stable, URL-safe provider slug resolves provider-slug/model to a pinned (provider, upstream_model) target (avoiding LiteLLM’s base_url ambiguity). Pinning bypasses cross-provider fan-out but still balances within the provider. Consequence: new immutable slug column + proxy parsing + /v1/models + UI work (see follow-up issues).

API surface

rolter speaks the OpenAI and Anthropic HTTP APIs so existing SDKs work unchanged — point them at the gateway base URL and use a rolter virtual key.

Authentication

• OpenAI-style: Authorization: Bearer <virtual-key>
• Anthropic-style: x-api-key: <virtual-key>

When no virtual keys are configured the gateway runs open (useful for local dev).

Endpoints (v1)

Realtime WebSocket

Connect with the usual gateway bearer key and the public route model as a query parameter:

wss://gateway.example.com/v1/realtime?model=gpt-realtime

rolter authenticates and selects an upstream before accepting the client upgrade, then pins that upstream and its selected provider key for the session. Text, binary audio and WebSocket control frames are relayed in both directions without application-level buffering. If the upstream drops, the client must reconnect; rolter does not fail a live session over to another target because replaying audio or tool events is unsafe.

The WebSocket-first implementation supports the OpenAI Realtime event stream, including session.update, input_audio_buffer.*, response.*, and function-call events. WebRTC/browser ephemeral-token handoff is not exposed by the gateway yet.

Routing

The model field in the body selects a route. The route’s strategy picks a target; rolter rewrites model to the target’s upstream model id and forwards with the provider’s credentials. Session affinity uses x-session-id when present.

When the selected upstream speaks the other chat protocol, rolter translates OpenAI Chat Completions and Anthropic Messages in both directions. Translation includes system/developer instructions, sampling and stop parameters, function tools and tool results, token usage, finish reasons, and live SSE events. Image and document inputs retain URL, base64 media type/data, and file references. Blocks with no equivalent in the target protocol (for example OpenAI input audio sent to an Anthropic Messages upstream) are preserved as opaque content blocks; the target may reject them rather than rolter silently dropping data.

OpenAI Responses

POST /v1/responses is routed by its required model field. Native OpenAI providers receive the request and SSE events unchanged. For Chat Completions or Anthropic Messages upstreams, rolter translates the common text, multimodal, function-tool, tool-result, sampling, and usage fields in both directions and emits Responses-shaped events to the caller. Responses-only features without a wire equivalent (for example background, store, previous_response_id, and provider-specific reasoning controls) are not forwarded to those older surfaces; use a native Responses provider when those features are required.

For native OpenAI providers, rolter records the selected provider, target, upstream model, provider credential fingerprint, and native response ID after a successful creation. GET/DELETE /v1/responses/{id}, cancellation, and input-item retrieval are then pinned to that record. Records are isolated by virtual key, retained for 24 hours by default, bounded to 100,000 entries per gateway process, and removed after a successful delete. Configure these limits with [responses] registry_ttl_secs and registry_max_entries; setting either to 0 disables registration.

The registry is process-local. Multi-replica deployments must keep lifecycle requests sticky to the gateway replica that accepted creation; records do not survive a gateway restart. Route changes do not retarget an existing response. If its provider is removed, its provider kind changes, or its credential is rotated away, the record becomes unavailable. Unknown, expired, deleted, cross-key, and unavailable records all return the same 404 response_not_found error so route ownership is not leaked.

Responses translated through Chat Completions or Anthropic Messages retain an ownership record but expose no lifecycle capabilities, because those upstream contracts do not retain an OpenAI Responses resource. Their lifecycle calls return 501 response_lifecycle_unsupported. Compaction and input-token counting remain unsupported for all providers.

Examples

# openai chat (streaming)
curl -N http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-rolter-dev" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o","stream":true,"messages":[{"role":"user","content":"hi"}]}'

# anthropic messages
curl http://localhost:4000/v1/messages \
  -H "x-api-key: sk-rolter-dev" \
  -H "Content-Type: application/json" \
  -d '{"model":"claude","max_tokens":256,"messages":[{"role":"user","content":"hi"}]}'

# openai embeddings
curl http://localhost:4000/v1/embeddings \
  -H "Authorization: Bearer sk-rolter-dev" \
  -H "Content-Type: application/json" \
  -d '{"model":"text-embedding-3-small","input":["hello","world"]}'

# self-hosted vllm pool via a public model name
curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-rolter-dev" \
  -H "x-session-id: user-123" \
  -H "Content-Type: application/json" \
  -d '{"model":"llama","messages":[{"role":"user","content":"hi"}]}'

Multipart audio (/v1/audio/transcriptions, /v1/audio/translations) forwards the upload verbatim and routes on the model form field; the route target’s upstream model name is not rewritten into the multipart body, and variant routing / per-model param defaults (JSON-only) do not apply.

Development setup

Prerequisites

• Rust (stable) via rustup — the workspace pins the toolchain in rust-toolchain.toml.
• Bun for the UI — curl -fsSL https://bun.sh/install | bash.
• Docker + Compose for Postgres/Redis/ClickHouse.
• uv (optional) for the PyPI-wheel install path and tooling.

Clone & build

git clone https://github.com/ormeilu/rolter.git
cd rolter
cargo build --workspace
cargo nextest run --workspace   # or `cargo test --workspace`; install: cargo install cargo-nextest

Run the gateway (no external services needed)

cp rolter.example.toml rolter.toml
export OPENAI_API_KEY=sk-...        # referenced by api_key_env in the config
cargo run -p rolter-gateway -- --config rolter.toml
# -> http://localhost:4000  (/healthz, /metrics, /v1/*)

Run the control plane + UI

cargo run -p rolter-control          # http://localhost:4001
cd ui && bun install && bun run dev  # http://localhost:3000 (proxies /api -> :4001)

Run the full stack

docker compose -f docker/docker-compose.yml up -d                 # postgres, redis, clickhouse, gateway, control

Handy tasks

just wraps the common commands:

just build | just test | just fmt | just lint
just gateway | just control | just ui-dev | just up

Before committing

cargo fmt --all
cargo clippy --workspace --all-targets -- -D warnings
cargo nextest run --workspace && cargo test --doc --workspace   # or `just test`
# optional hooks: prek install   (conventional commit msg + fmt/clippy)

Testing

Run

Tests run under nextest (the same runner CI uses), plus a separate doc-test pass since nextest does not run doc tests:

cargo nextest run --workspace   # unit + integration tests
cargo test --doc --workspace    # doc tests
cd ui && bun run lint           # ui typecheck

Install the runner once with cargo install cargo-nextest (or see the nextest install docs). just test runs both Rust passes for you. Plain cargo test --workspace still works if you haven’t installed nextest, but CI runs nextest so prefer it locally.

The Ollama Cloud live smoke sends a billed request and is ignored by default:

OLLAMA_API_KEY=... ROLTER_OLLAMA_LIVE_MODEL=gpt-oss:20b \
  cargo test -p rolter-gateway --test ollama_cloud live_smoke -- --ignored

Test grouping is configured in .config/nextest.toml: the Postgres-backed rolter-store/rolter-control suites share one database and reset the schema per test, so they run in a single-threaded group to avoid clobbering each other.

Layout

• Unit tests live next to the code in #[cfg(test)] mod tests. Current coverage: balancer strategies (round-robin cycling, consistent-hash stability, cache-aware affinity, empty targets), the prefix trie, config parsing, model rewrite, auth checks, and the in-memory store.
• Keep the pure crates (rolter-core, rolter-balancer, rolter-auth) fully unit-testable without I/O.

Strategy as the project grows

• Integration tests for the gateway: spin up the Axum app with a mock upstream (wiremock/httpmock) and assert routing, auth, model rewrite, error mapping and streaming passthrough.
• Property tests (proptest) for the balancer: distribution fairness, affinity invariants.
• DB tests for rolter-store Postgres backend behind a feature, using a disposable container.
• Load tests (oha/k6) against a mock upstream to track added latency and max RPS (see performance.md).

Benchmarks

Hot-path micro-benchmarks run under criterion. They live in crates/<crate>/benches/ with a [[bench]] harness = false entry per file, and cover the per-request cost that shows up as pure gateway overhead:

just bench                       # cargo bench --workspace
cargo bench -p rolter-balancer   # just the balancer benches
cargo bench -p rolter-balancer --bench pick   # one bench target

Current coverage (rolter-balancer):

• pick — LoadBalancer::pick for every built-in strategy over a ~24-target pool with a populated RouteContext.
• trie — prefix-trie insert (bounded/unbounded, so LRU eviction is measured) and longest_prefix on a warm trie.

criterion writes HTML reports to target/criterion/. Benches are not run in CI (timings are noisy on shared runners), but cargo clippy --workspace --all-targets -- -D warnings compiles them on every PR, so they cannot silently bit-rot. Use just bench-check (cargo bench --workspace --no-run) to compile them locally without running.

Coverage

Workspace line coverage is measured with cargo llvm-cov:

cargo install cargo-llvm-cov
cargo llvm-cov --workspace --all-features --summary-only   # quick %
cargo llvm-cov --workspace --all-features --html           # browsable report

CI runs coverage in the coverage job of quality.yml and enforces a ratcheting baseline: the committed baseline lives in .github/coverage-baseline.txt, and .github/scripts/coverage-ratchet.sh fails the step if the current percentage drops more than COVERAGE_TOLERANCE points (default 0.5) below it. The job also uploads the lcov.info report as a CI artifact.

Policy (ROL-246):

• New code must not push coverage below baseline − tolerance. If a PR legitimately lowers coverage, edit .github/coverage-baseline.txt in the same PR and explain why.
• When coverage climbs well above the baseline, raise the baseline to lock in the gain (the ratchet only goes up).
• The job is informational (continue-on-error: true) until the baseline is trusted; promote it to blocking by removing that flag on the coverage job.

CI

.github/workflows/ci.yml delegates to the shared quality.yml gate, which runs cargo fmt --check, cargo clippy -D warnings, cargo nextest run --workspace --all-features plus a cargo test --doc pass, the feature matrix, cargo doc (warnings as errors), cargo-deny, gitleaks, the UI lint/build, and a Conventional Commit PR-title check on every push/PR.

Full-stack compose smoke

The compose-smoke job boots the production-shaped Docker Compose topology (Postgres, Redis, ClickHouse, gateway, control) and exercises it end-to-end. Run it locally with the same script CI uses:

bash docker/smoke/smoke.sh

It layers docker/docker-compose.ci.yml over the base compose file: the overlay mounts docker/smoke/rolter.smoke.toml (a keyless open gateway config) so the built-in fake-llm model answers without any provider secret. The script waits for both /healthz endpoints, checks /v1/models and fake-llm chat (non-streaming + SSE) on the gateway and the postgres-backed /internal/snapshot on the control plane, then always dumps compose logs and runs down -v. It is informational (continue-on-error) until the image-build cost and flake profile are trusted (ROL-245).

Contributing

Thanks for helping build rolter.

Workflow

1. Branch from master: feat/<scope>-<short> or fix/<scope>-<short>.
2. Make focused changes; add unit tests next to the code.
3. Run cargo fmt --all, cargo clippy --workspace --all-targets -- -D warnings, and the tests (just test, or cargo nextest run --workspace + cargo test --doc --workspace).
4. Use Conventional Commits for messages and the PR title.
5. Link issues with Closes #N / Refs #N.
6. Open a PR; fill in the template; CI must be green.

Code standards

• Rust 2021; rustfmt defaults; clippy clean with -D warnings.
• thiserror for library errors, anyhow in binaries.
• Keep the data-plane hot path allocation-light; never block on locks (use arc-swap).
• Code comments start lowercase, no trailing punctuation; /// doc comments use normal prose.
• New balancing strategy → implement rolter_balancer::LoadBalancer + wire into build().
• New storage backend → implement rolter_store traits behind a cargo feature.

Agent commits

Automated contributions include the trailer:

Co-Authored-By: Oz <oz-agent@warp.dev>

Don’t

• Don’t commit secrets (use env / the encrypted store).
• Don’t force-push master; force-push is fine on your own feature branches.
• Don’t --amend after pushing shared history.

Commit conventions

rolter uses Conventional Commits for commit messages and PR titles. CI checks PR titles; the conventional-pre-commit hook checks local messages.

Format

<type>(<scope>): <subject>

<body>

<footer>

• type (required): feat, fix, perf, refactor, docs, test, build, ci, chore, revert
• scope (recommended): gateway, balancer, proxy, core, store, auth, control, ui, docs, infra, ci, deps, release
• subject: imperative, lowercase, ≤ 72 chars, no trailing period
• breaking change: add ! after the scope and a BREAKING CHANGE: footer

Examples

feat(balancer): add precise kv-event cache-aware scorer
fix(gateway): stream anthropic sse without buffering
perf(proxy): reuse pooled client per egress proxy
docs(architecture): document reload-free config propagation
refactor(core)!: rename ModelRoute.targets to upstreams

BREAKING CHANGE: config field `targets` is now `upstreams`.

Issues & PRs

• Link issues from the body/footer: Closes #123, Refs #123.
• PR title must be a single valid Conventional Commit line (enforced by CI via amannn/action-semantic-pull-request).
• Squash-merge so the PR title becomes the commit on master; keeps history releasable and changelog-friendly.

Tooling

• .config/commitlint.config.mjs — rules (types, scopes, lowercase subject, 72-char header).
• .pre-commit-config.yaml — conventional-pre-commit (commit-msg) + cargo fmt/cargo clippy.
• Install hooks: prek install (or pre-commit install && pre-commit install --hook-type commit-msg).

Packaging & distribution

rolter ships three ways.

The unified rolter binary dispatches to both planes via subcommands:

rolter gateway --config rolter.toml     # data plane
rolter control --database-url postgres://…   # control plane + UI host

The standalone rolter-gateway / rolter-control binaries remain available.

cargo

cargo install rolter            # unified launcher (from crates.io)
# or from source:
cargo install --path crates/rolter

uv (PyPI wheel via maturin)

The wheel bundles the compiled rolter launcher so Python users can install the CLI with uv. pyproject.toml uses the maturin backend (bindings = "bin", manifest-path = crates/rolter/Cargo.toml).

uv tool install maturin       # one-time
uvx maturin build --release   # build a wheel into target/wheels/
uv tool install rolter        # once published to PyPI

Docker

Multi-stage docker/Dockerfile builds the Rust binaries and the Bun-built UI, then assembles a slim runtime:

docker build -f docker/Dockerfile -t rolter:dev .
docker compose -f docker/docker-compose.yml up -d          # full stack with postgres/redis/clickhouse

Releasing (roadmap)

• Tag-driven CI: build manylinux/macos wheels (cibuildwheel/maturin-action), publish to PyPI; publish crates to crates.io; push multi-arch images to GHCR.
• Conventional Commits enable automated changelog/version bumps (e.g. release-please/semantic-release).

Zero-config quickstart: no keys, add providers at runtime

rolter starts with zero LLM credentials and serves the built-in fake-llm model out of the box. Real providers, models, and upstream API keys are added later — at runtime, over the management API, persisted in Postgres, and picked up by the gateway without a restart.

1. Start with no credentials

rolter easy-up

That’s it. No provider keys, no database, no config file (one is created from the bundled example on first run). The gateway answers immediately:

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer sk-rolter-dev" \
  -H "Content-Type: application/json" \
  -d '{"model": "fake-llm", "messages": [{"role": "user", "content": "hi"}]}'

sk-rolter-dev is the local-dev virtual key from the generated rolter.toml; delete the [[virtual_keys]] section to run open, or replace it before exposing the gateway anywhere.

2. Switch on runtime management (Postgres mode)

Runtime CRUD over providers/models/keys needs the database-backed control plane:

export ROLTER_ADMIN_TOKEN="$(openssl rand -hex 24)"   # protects the management API
export ROLTER_KEK="$(openssl rand -hex 32)"           # encrypts provider keys at rest

rolter easy-up --database-url postgres://user:pass@localhost:5432/rolter

easy-up migrates, seeds a default org/team/project, imports the bootstrap toml, and starts both planes. The gateway port (4000) now also serves the management API: /admin/* proxies to the control plane’s /api/v1/*.

Two deployment secrets matter here:

• ROLTER_ADMIN_TOKEN — bearer token required on the management API and the internal snapshot endpoint. Without it those endpoints are open (fine on localhost; a startup warning reminds you).
• ROLTER_KEK — key-encryption key. Provider API keys submitted over the API are sealed with AES-256-GCM before they reach Postgres; the KEK never leaves the process environment. Set the same value on the control plane and gateway (with easy-up it is one process, so one export). Without a KEK, requests that include an api_key are rejected — there is no plaintext fallback.

3. Add your first real provider — with its key — via curl

BASE=http://localhost:4000/admin

# ids seeded by easy-up
ORG=$(curl -s $BASE/orgs -H "Authorization: Bearer $ROLTER_ADMIN_TOKEN" | jq -r '.[] | select(.name=="default") | .id')
TEAM=$(curl -s $BASE/orgs/$ORG/teams -H "Authorization: Bearer $ROLTER_ADMIN_TOKEN" | jq -r '.[0].id')
PROJECT=$(curl -s $BASE/teams/$TEAM/projects -H "Authorization: Bearer $ROLTER_ADMIN_TOKEN" | jq -r '.[0].id')

# provider + upstream credential (sealed at rest; never returned by the API)
PROVIDER=$(curl -s -X POST $BASE/orgs/$ORG/providers \
  -H "Authorization: Bearer $ROLTER_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"name": "openai", "kind": "openai", "api_base": "https://api.openai.com", "api_key": "sk-..."}' \
  | jq -r .id)

# public model name + target
ROUTE=$(curl -s -X POST $BASE/projects/$PROJECT/routes \
  -H "Authorization: Bearer $ROLTER_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o", "strategy": "round_robin"}' | jq -r .id)

curl -s -X POST $BASE/routes/$ROUTE/targets \
  -H "Authorization: Bearer $ROLTER_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d "{\"provider_id\": \"$PROVIDER\", \"upstream_model\": \"gpt-4o\"}"

Within the snapshot poll interval (instantly with --redis-url) the gateway serves the new model — no restart:

curl http://localhost:4000/v1/models -H "Authorization: Bearer sk-rolter-dev"

4. Rotate or remove a credential

PUT /admin/providers/{id} updates a provider in place. For api_key, api_key_env, and egress_proxy: omit the field to leave it unchanged, send an empty string to clear it, send a value to set/rotate it.

# rotate
curl -X PUT $BASE/providers/$PROVIDER \
  -H "Authorization: Bearer $ROLTER_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"api_key": "sk-new-key"}'

# remove the stored key (falls back to api_key_env, if set)
curl -X PUT $BASE/providers/$PROVIDER \
  -H "Authorization: Bearer $ROLTER_ADMIN_TOKEN" -H "Content-Type: application/json" \
  -d '{"api_key": ""}'

How the pieces fit

• Persistence — providers/routes/keys live in Postgres; credentials in provider_keys, AES-256-GCM-sealed with the ROLTER_KEK-derived key.
• Propagation — every write bumps config_version (a database trigger, transactional with the write). Gateways poll GET /internal/snapshot?version=N and hot-swap their routing snapshot; with Redis configured the control plane also publishes a bump for instant refetch. See Config & hot reload.
• Two surfaces, one API — /admin/* on the gateway is a thin reverse proxy to the control plane’s /api/v1/* (enable on a standalone gateway with --admin-url http://control:4001); pointing tooling at either works identically. Authentication is enforced by the control plane in both cases.
• Config file still wins — anything declared in the bootstrap rolter.toml is a read-only “config model” (LiteLLM-style): the API rejects runtime mutations to it with 409.
• Reads never leak secrets — GET /api/v1/config (the dashboard read) redacts api_key; only the token-guarded snapshot endpoint carries decrypted keys, because the gateway needs them to call upstreams.

Configuration reference

The gateway boots from a TOML file (--config, default rolter.toml); see rolter.example.toml. At runtime, the control plane is the source of truth and applies changes without a restart (config-and-hot-reload.md).

Schema

[server]

• host (string, default 0.0.0.0)
• port (u16, default 4000)
• metrics_path (string, default /metrics) — path the Prometheus metrics endpoint is served on; change it to avoid colliding with an upstream app or sidecar that already owns /metrics. Must be rooted (/…) and must not collide with a built-in route (/healthz, /v1/*).

[tls]

• ca_bundles (string[], default []) — PEM CA-bundle files added to the normal public-root trust store for outbound upstream TLS. ROLTER_CA_BUNDLE replaces this global list with a single deployment-local path. Files are checked for missing, unreadable, empty, and malformed content while config is loaded.

[[providers]]

• name (string, unique) — referenced by route targets
• kind (openai | anthropic | openai_compatible | ollama | ollama_cloud | llama_cpp | openrouter | tei | azure_openai | bedrock | vertex)
• api_base (string) — base URL, no trailing slash
• api_key (string, optional) — prefer api_key_env
• api_key_env (string, optional) — environment variable to read the key from
• role_profile (openai | system_only | anthropic, optional) — explicit instruction-role semantics. The default is openai for kind = "openai", anthropic for kind = "anthropic", and conservative system_only for every OpenAI-compatible kind. system_only converts leading developer messages to system in place; it rejects a system or developer message after a user/assistant/tool turn with role_capability_unsupported rather than silently changing it.
• model_role_profiles (table, optional) — upstream-model-specific role_profile overrides. Use this only for a custom template whose developer-role support is explicitly known; rolter never probes a vLLM template at runtime.

Role-capability profiles

openai_compatible describes the HTTP surface only. vLLM, in particular, renders roles using the selected model’s chat template, so an endpoint’s role support must not be inferred from its /v1 API. The default system_only profile is suitable for Qwen-style templates that do not define developer. Set role_profile = "openai" or a model_role_profiles entry only after confirming that the deployed template supports distinct developer messages.

Anthropic targets collect leading OpenAI developer and system messages into ordered top-level system blocks. Instruction messages placed after a conversation turn are rejected for anthropic and system_only profiles; rolter returns an OpenAI-style 400 with code role_capability_unsupported instead of dropping or reclassifying them.

Ollama: local daemon vs Cloud

Use ollama for a local/self-hosted daemon such as http://localhost:11434 (no authentication). Use ollama_cloud for direct programmatic Cloud access. Cloud requires api_key_env (normally OLLAMA_API_KEY); inline keys and key pools are rejected. Configure api_base = "https://ollama.com"; rolter uses the OpenAI-compatible /v1/chat/completions and /v1/models endpoints with bearer authentication. Ollama’s native /api/* endpoints are distinct.

[[providers]]
name = "ollama-cloud"
kind = "ollama_cloud"
api_base = "https://ollama.com"
api_key_env = "OLLAMA_API_KEY"

Azure OpenAI, Amazon Bedrock, and Vertex AI

These providers use their current OpenAI-compatible APIs. Set api_base to the provider’s OpenAI-compatible prefix and use an environment-sourced credential:

[[providers]]
name = "azure"
kind = "azure_openai"
api_base = "https://RESOURCE.openai.azure.com/openai/v1"
api_key_env = "AZURE_OPENAI_API_KEY"

[[providers]]
name = "bedrock"
kind = "bedrock"
api_base = "https://bedrock-runtime.us-east-1.amazonaws.com/v1"
api_key_env = "AWS_BEARER_TOKEN_BEDROCK"

[[providers]]
name = "vertex"
kind = "vertex"
api_base = "https://aiplatform.googleapis.com/v1/projects/PROJECT/locations/global/endpoints/openapi"
api_key_env = "VERTEX_ACCESS_TOKEN"

Azure credentials are sent in the api-key header. Bedrock and Vertex credentials are sent as bearer tokens. The default active-health probes use Azure’s model list, Bedrock ListFoundationModels, and Vertex’s publisher model list, respectively; none invokes a model.

• [[providers.api_keys]] (optional) — multiple weighted API keys for one provider; when present it takes precedence over the single api_key/api_key_env pair. Providers cap throughput per key, so rotating across keys multiplies effective RPM/TPM
  • key (string, optional) — inline key value; prefer env
  • env (string, optional) — environment variable to read the key from
  • weight (u32, default 1) — relative selection weight
• api_key_env (string, optional) — env var to read the key from
• egress_proxy (string, optional) — HTTP/HTTPS/SOCKS5 outbound proxy
• ca_bundles (string[], optional) — provider-specific replacement for global [tls].ca_bundles; [] explicitly selects public roots only
• also_track_via_llm_call (bool, default false) — when set, active health checks send a real max_tokens = 1 completion to this provider instead of the free /v1/models liveness probe, so a healthy result proves end-to-end inference. This burns a few tokens on every sweep (interval_secs); leave it off unless you need inference-level health. Recorded as source = llm_call in provider_health_events.
• llm_probe_model (string, optional) — the upstream model id the also_track_via_llm_call completion targets (e.g. gpt-4o-mini). Required when the flag is on; without it (or an api key) the checker logs a warning and falls back to the free probe.
• status_page_url (string, optional) — statuspage.io-style status.json URL (e.g. https://status.anthropic.com/api/v2/status.json). When set, a slow background poll records the provider’s public status as a secondary status_page health signal — it surfaces in provider_health_events, the dashboard and rolter_status_page_degraded_total, but never marks the provider unhealthy or affects routing on its own. Parse/transport failures are logged and skipped.

See Custom CA bundles for rotation behavior and Docker/Kubernetes mount examples.

[[routes]]

• model (string) — public model name clients request
• strategy (round_robin | random | power_of_two | consistent_hash | cache_aware | weighted | pipeline, default round_robin)
• [[routes.targets]]
  • provider (string) — a provider name
  • model (string, optional) — upstream model id; defaults to the requested model
  • weight (u32, default 1)
• [routes.params] (table, optional) — admin default inference params injected into the request body (e.g. temperature, max_tokens, stop). Provider-agnostic: keys are whatever the upstream accepts. An unset param passes through untouched.
• [routes.param_policy] — whether callers may override the params defaults
  • mode (allow | deny, default allow) — baseline override policy
  • allow (string[], default []) — params callers may override when mode = "deny"
  • deny (string[], default []) — params callers may not override when mode = "allow"
  • when an override is denied and the caller sends the param anyway, the admin default silently wins
• [[routes.variants]] (optional) — weighted variants for A/B, canary, and key-split traffic. When present, the route ignores the top-level targets pool: a request samples one variant by weight (the primary) and, on failure, falls over to the remaining variants in declared order. Within a variant the route’s strategy picks which target leads; the remaining targets follow in declared order as the deterministic fallback tail.
  • name (string) — variant identifier, attributed in request logs (the variant column)
  • weight (u32, default 1) — relative traffic share for the primary draw
  • [[routes.variants.targets]] — same shape as [[routes.targets]]
  • [routes.variants.params] (table, optional) — variant-scoped param defaults, layered over [routes.params] (the variant wins) under the route’s param_policy

[[virtual_keys]]

• key (string) — the bearer token clients present
• name (string, optional)
• models (string[], default []) — allow-list; empty = all

[logging]

• clickhouse_url (string, optional)

[health]

• enabled (bool, default false) — master switch for active upstream probing
• interval_secs (u64, default 10) — seconds between probe sweeps
• timeout_secs (u64, default 2) — per-probe timeout
• path (string, default /) — probe path; the default resolves to each provider kind’s free liveness endpoint (normally /v1/models, or the provider-native Azure, Bedrock, or Vertex model-list endpoint)
• probe_concurrency (usize, default 2) — max probes in flight at once during a sweep, so probing never stampedes upstreams
• consecutive_failure_threshold (u32, default 3) — consecutive probe failures before a provider is marked unhealthy
• recovery_success_threshold (u32, default 2) — consecutive successes before an unhealthy provider recovers
• status_page_interval_secs (u64, default 60) — seconds between provider status-page polls; only providers with a status_page_url are polled, and the poller runs even when enabled = false
• probes are jittered across the first quarter of the interval (per-provider stable offset), and a 429 on the probe itself pauses that provider’s probing with exponential backoff (1, 2, 4, 8 sweeps) without marking it unhealthy

[realtime]

Guardrails for persistent /v1/realtime WebSocket sessions. All limits are per gateway process; set a value to 0 to disable that limit.

• max_connections (u64, default 1000) — concurrent sessions admitted by this gateway instance
• max_session_secs (u64, default 3600) — hard session-duration limit
• idle_timeout_secs (u64, default 300) — closes a session when neither side sends a frame

Environment variables

• ROLTER_CONFIG, ROLTER_HOST, ROLTER_PORT — gateway
• ROLTER_CONTROL_HOST, ROLTER_CONTROL_PORT, ROLTER_UI_DIR — control plane
• ROLTER_MASTER_KEY — AES-256-GCM KEK for provider-secret encryption
• DATABASE_URL, REDIS_URL, CLICKHOUSE_URL — datastores
• RUST_LOG — tracing filter (e.g. info, rolter_gateway=debug)
• provider key vars referenced by api_key_env (e.g. OPENAI_API_KEY)

CLI flags override env, which override file values.

Custom CA bundles

Rolter can add private CA certificates to the normal public-root trust store for outbound HTTPS connections to upstream providers. Certificate-chain and hostname verification remain enabled; this feature does not affect inbound TLS or configure mTLS client certificates.

Minimal air-gapped configuration

Mount a PEM file containing one or more CA certificates, then use either the environment variable:

ROLTER_CA_BUNDLE=/etc/rolter/ca/private-root.pem rolter-gateway --config /app/rolter.toml

or the matching global TOML field:

[tls]
ca_bundles = ["/etc/rolter/ca/root.pem", "/etc/rolter/ca/intermediate.pem"]

[[providers]]
name = "private-vllm"
kind = "openai_compatible"
api_base = "https://llm.internal.example"

ROLTER_CA_BUNDLE replaces the global TOML list. A provider can replace the global selection independently:

[[providers]]
name = "isolated-cluster"
kind = "openai_compatible"
api_base = "https://llm.cluster.internal"
ca_bundles = ["/etc/rolter/ca/cluster-root.pem"]

Other providers keep using the global private roots plus the built-in public roots. Set a provider’s ca_bundles = [] to use public roots only.

Docker Compose

Mount the bundle read-only and pass its in-container path:

services:
  gateway:
    environment:
      ROLTER_CA_BUNDLE: /etc/rolter/ca/private-root.pem
    volumes:
      - ./pki/private-root.pem:/etc/rolter/ca/private-root.pem:ro

Kubernetes

Store the public CA certificate in a ConfigMap or Secret and mount it read-only:

apiVersion: v1
kind: ConfigMap
metadata:
  name: rolter-upstream-ca
data:
  private-root.pem: |
    -----BEGIN CERTIFICATE-----
    ...
    -----END CERTIFICATE-----
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: rolter-gateway
spec:
  template:
    spec:
      containers:
        - name: gateway
          image: rolter:latest
          env:
            - name: ROLTER_CA_BUNDLE
              value: /etc/rolter/ca/private-root.pem
          volumeMounts:
            - name: upstream-ca
              mountPath: /etc/rolter/ca
              readOnly: true
      volumes:
        - name: upstream-ca
          configMap:
            name: rolter-upstream-ca

Validation and rotation

Startup fails with the bundle path and an actionable error when a configured file is missing, unreadable, contains no certificates, or has malformed PEM. Snapshot updates are rejected under the same conditions.

HTTP clients capture trust roots when their connection pool is created. After replacing a mounted certificate, publish or fetch a new configuration snapshot—even if the path is unchanged—to clear configured pools and rebuild them from the new bundle. With static bootstrap configuration, restart the gateway. Existing in-flight connections finish with their original trust configuration; subsequent connections use the rotated bundle.

Self-hosted Ollama

rolter supports a local or privately hosted Ollama daemon through Ollama’s OpenAI-compatible API. This provider does not require an API key.

Native setup

Install Ollama, start the daemon, and pull a small smoke-test model:

ollama serve
ollama pull qwen2.5:0.5b

Configure the daemon origin, without /v1 (rolter appends endpoint paths):

[[providers]]
name = "ollama-local"
kind = "ollama"
api_base = "http://localhost:11434"

[[routes]]
model = "local-qwen"
strategy = "round_robin"
[[routes.targets]]
provider = "ollama-local"
model = "qwen2.5:0.5b"

Start rolter and exercise model discovery, chat, legacy completions, embeddings, and streaming:

curl http://localhost:4000/v1/models
curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"local-qwen","messages":[{"role":"user","content":"hello"}]}'
curl http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"local-qwen","stream":true,"messages":[{"role":"user","content":"hello"}]}'
curl http://localhost:4000/v1/completions \
  -H 'content-type: application/json' \
  -d '{"model":"local-qwen","prompt":"hello"}'
curl http://localhost:4000/v1/embeddings \
  -H 'content-type: application/json' \
  -d '{"model":"local-qwen","input":"hello"}'

/v1/models lists rolter’s configured public route names, so the example returns local-qwen; it does not expose unrelated models installed in Ollama.

Docker setup

Containers must address Ollama by its Compose service name:

services:
  ollama:
    image: ollama/ollama:0.9.6
    volumes:
      - ollama-data:/root/.ollama

Use api_base = "http://ollama:11434" in the gateway container’s config. The opt-in smoke suite under integration/ollama/ provides a complete reproducible Compose setup and pulls qwen2.5:0.5b automatically.

Compatibility and known gaps

rolter passes OpenAI request JSON and response bodies through unchanged (apart from the configured model-name rewrite), preserving retry, cooldown, health, logging, error mapping, routing, and SSE semantics. Ollama currently documents chat and legacy completions, streaming, JSON mode (response_format), tools, vision message content, seed, and usage fields. The gateway also passes stream_options through, though Ollama may ignore unsupported options.

Support depends on the installed Ollama release and model: tool calling and vision require capable models, JSON schemas are not guaranteed to be obeyed by every model, and some OpenAI fields are accepted but ignored. Ollama’s OpenAI-compatible embeddings endpoint accepts models with embedding support; for production, route it to a dedicated embedding model. Ollama’s native /api/* endpoints and Ollama Cloud authentication are outside this provider’s scope.

OpenRouter

rolter’s openrouter provider targets OpenRouter’s OpenAI-compatible API while keeping OpenRouter model identifiers and routing controls intact.

Configuration

Create an API key in OpenRouter and expose it only through the environment:

export OPENROUTER_API_KEY='...'

[[providers]]
name = "openrouter"
kind = "openrouter"
api_base = "https://openrouter.ai/api/v1"
api_key_env = "OPENROUTER_API_KEY"

[[routes]]
model = "router-chat"
strategy = "round_robin"
[[routes.targets]]
provider = "openrouter"
model = "anthropic/claude-sonnet-4"

The public rolter model (router-chat) is rewritten only to the target model override. OpenRouter identifiers such as anthropic/claude-sonnet-4, including their provider prefix and optional variants, are otherwise forwarded verbatim. rolter fallback chooses another configured target after a retryable failure; OpenRouter’s own provider request object then controls routing among upstreams inside the selected OpenRouter target.

For example, this body preserves OpenRouter’s provider ordering and fallback policy:

{
  "model": "router-chat",
  "messages": [{"role": "user", "content": "hello"}],
  "provider": {
    "order": ["Anthropic", "Google"],
    "allow_fallbacks": true,
    "data_collection": "deny"
  }
}

Chat completions, SSE chunks, usage/cost fields, response metadata, and OpenRouter error JSON pass through without normalization. rolter still applies its normal authentication, route policy, retries, cooldowns, health checks, request logging, and routing headers. /v1/models lists configured rolter route aliases rather than exposing every model in OpenRouter’s catalog.

Attribution headers

OpenRouter recommends HTTP-Referer and X-Title for application attribution; they are not required for authentication. rolter omits them by default so it does not disclose deployment identity. Set either explicitly when desired:

export OPENROUTER_HTTP_REFERER='https://example.com'
export OPENROUTER_X_TITLE='Example gateway'

These values are forwarded only by openrouter providers. Never put API keys, user identifiers, or private internal hostnames in attribution headers.

Live smoke

The ignored live test makes a billable request and therefore requires both a credential and an explicitly selected model:

OPENROUTER_API_KEY=... ROLTER_OPENROUTER_LIVE_MODEL=openai/gpt-4.1-mini \
  cargo test -p rolter-gateway --test openrouter live_openrouter_smoke -- --ignored

Docker deployment

Compose (full stack)

docker/docker-compose.yml brings up Postgres, Redis, ClickHouse, and the rolter gateway + control services.

cp .env.example .env            # set OPENAI_API_KEY etc.
docker compose -f docker/docker-compose.yml up -d
docker compose -f docker/docker-compose.yml logs -f gateway

• Gateway: http://localhost:4000
• Control + UI: http://localhost:4001
• Postgres 5432, Redis 6379, ClickHouse 8123/9000

DB schemas auto-apply on first start (migrations/ → Postgres initdb, clickhouse/ → ClickHouse initdb).

Image

The multi-stage docker/Dockerfile produces a slim Debian runtime with both binaries and the built UI at /app/ui/dist.

docker build -f docker/Dockerfile -t rolter:dev .
docker run --rm -p 4000:4000 \
  -e OPENAI_API_KEY=$OPENAI_API_KEY \
  -v "$PWD/rolter.toml:/app/rolter.toml" \
  rolter:dev

Override the entrypoint to run the control plane:

docker run --rm -p 4001:4001 rolter:dev rolter-control

Published images

Release tags publish an image to GHCR (and, when configured, Docker Hub) under the same repo name and tags. Each release is tagged with its version and latest:

docker pull ghcr.io/<owner>/rolter:latest
docker pull ghcr.io/<owner>/rolter:0.0.4

Publishing is fail-closed and opt-in, mirroring the PyPI flow. The publish-docker job in .github/workflows/release.yml runs only when:

• repo variable DOCKER_PUBLISH_ENABLED = true, and
• the verify + external-check gates pass for the tagged commit.

GHCR always publishes via the built-in GITHUB_TOKEN. To also push to Docker Hub, set repo variable DOCKERHUB_IMAGE (e.g. docker.io/acme/rolter) and secrets DOCKERHUB_USERNAME / DOCKERHUB_TOKEN; the same tag set is applied to both registries. (Multi-arch images are a separate roadmap item — releases currently ship linux/amd64.)

Production notes

• Put the gateway behind TLS (ingress/load balancer); keep the control plane private.
• Set a strong ROLTER_MASTER_KEY; provide DB/Redis/ClickHouse URLs via env or a secrets manager.
• Scale gateway horizontally; all replicas hot-reload config from Redis. ClickHouse and Postgres are shared.
• Kubernetes manifests/Helm chart are a roadmap item.

Air-gapped

Running fully offline behind an internal mirror (Nexus/Artifactory/Harbor)? See Air-gapped install & operation.

llama.cpp (llama-server)

Rolter’s llama_cpp provider preset targets the OpenAI-compatible API exposed by llama-server. It needs no API key by default and works with local CPU or GPU GGUF deployments.

Start llama-server

Choose a GGUF whose license permits your intended use and whose quantization fits available RAM/VRAM. Q4_K_M is a practical starting point for local use; smaller quantizations use less memory at the cost of quality.

With a native llama.cpp build:

llama-server -m /models/model.gguf --host 0.0.0.0 --port 8080

Or with the upstream Docker image:

docker run --rm -p 8080:8080 -v "$PWD/models:/models" \
  ghcr.io/ggml-org/llama.cpp:server \
  -m /models/model.gguf --host 0.0.0.0 --port 8080

Configure Rolter

api_base is the server origin, without /v1. model on the target is the model identifier reported by llama-server; the public route can be a stable alias.

[[providers]]
name = "local-llama"
kind = "llama_cpp"
api_base = "http://127.0.0.1:8080"

[[routes]]
model = "local-chat"
strategy = "round_robin"

[[routes.targets]]
provider = "local-llama"
model = "model.gguf"

Rolter forwards /v1/chat/completions and /v1/completions, including SSE, sampling fields, grammar, and OpenAI response_format. /v1/models lists Rolter’s public route aliases. Routing headers, retries, cooldowns, and active health checks behave like other providers; the default health probe calls the upstream /v1/models endpoint.

llama.cpp-native routes such as /completion, /tokenize, /detokenize, and slot/metrics administration are intentionally not exposed by Rolter’s generic OpenAI API. Call llama-server directly for those endpoints.

Smoke test

With llama-server running and the model id from its /v1/models response:

integration/llama-cpp-smoke.sh http://127.0.0.1:8080 model.gguf

The script starts a temporary Rolter gateway, verifies model listing, non-streaming completion, SSE, and routing headers, then cleans up.

Hugging Face Text Embeddings Inference (TEI)

The tei provider targets TEI’s OpenAI-compatible POST /v1/embeddings endpoint. Self-hosted TEI is keyless by default; api_key or api_key_env can add bearer authentication when TEI sits behind an authenticated proxy.

Run TEI

For a reproducible CPU deployment with a small embedding model:

docker run --rm -p 8080:80 -v "$PWD/data:/data" \
  ghcr.io/huggingface/text-embeddings-inference:cpu-1.9 \
  --model-id sentence-transformers/all-MiniLM-L6-v2

On Apple Silicon, install and run the native server:

brew install text-embeddings-inference
text-embeddings-router \
  --model-id sentence-transformers/all-MiniLM-L6-v2 --port 8080

Configure Rolter

Use the server origin as api_base, without /v1:

[[providers]]
name = "tei-local"
kind = "tei"
api_base = "http://127.0.0.1:8080"

[[routes]]
model = "embed-local"
strategy = "round_robin"

[[routes.targets]]
provider = "tei-local"
model = "sentence-transformers/all-MiniLM-L6-v2"

Rolter preserves OpenAI string, string-array, token-array, and token-array-batch inputs, plus encoding_format, dimensions, user, embedding vectors, usage, and upstream error JSON. Normal routing headers, retries, cooldowns, logging, and health behavior apply. The default active probe uses TEI’s /health route.

Only /v1/embeddings is part of this adapter. TEI-native /embed, /rerank, /embed_sparse, /predict, /tokenize, /health, and /metrics are not exposed through Rolter’s generic OpenAI surface. Call TEI directly for them.

Smoke test

The opt-in Compose test starts TEI, downloads the small model, starts Rolter, and verifies batch embeddings, optional fields, usage, and routing headers:

integration/tei/run.sh

Air-gapped installation & operation

rolter is designed to run in fully air-gapped environments — no public internet at build or run time. The runtime is egress-free by default: it talks only to the backends you configure (upstream providers, Postgres, Redis, ClickHouse) and to nothing else. This page covers how to install rolter behind an internal mirror, what the runtime does and does not reach, and how to verify the deployment offline.

Runtime egress guarantees

rolter makes outbound network calls only to endpoints you configure:

• Upstream providers — the base_url of each configured provider.
• Postgres / Redis / ClickHouse — only when their URLs are set (DATABASE_URL / ROLTER_DATABASE_URL, REDIS_URL, clickhouse_url).
• Control-plane snapshot — only when the gateway is started with --snapshot-url (and Redis pub/sub only with --redis-url).
• OTLP traces — only when an OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is set. With no OTEL_* env, no exporter is built and there is zero tracing egress.

Everything else is self-contained:

• The interactive API reference at /docs embeds the Scalar JS bundle in the binary and sets withDefaultFonts: false, so it never reaches a CDN or fonts.scalar.com. This is asserted by the docs_page_is_self_contained and scalar_bundle_is_embedded tests in crates/rolter-gateway/src/openapi.rs.
• The dashboard SPA is served as static assets by the control plane; it loads no third-party scripts, fonts, or styles.

Caveat — status-page pollers. If a provider sets status_page_url, the gateway periodically fetches that URL as a secondary health signal. Leave status_page_url unset in air-gapped configs (or point it at an internal mirror) so the poller stays inside the enclave.

Install paths through a mirroring proxy

Air-gapped sites usually proxy public registries through an internal mirror (Sonatype Nexus, JFrog Artifactory, Harbor, …). Pick the path that matches how you ship rolter.

Docker image (recommended)

Pull through a registry that proxies GHCR/Docker Hub:

docker pull registry.internal.example/rolter/rolter:latest

Or transfer a fully offline image with docker save / docker load:

# on a connected host
docker pull ghcr.io/ormeilu/rolter:latest
docker save ghcr.io/ormeilu/rolter:latest -o rolter.tar

# copy rolter.tar into the enclave, then
docker load -i rolter.tar

PyPI wheel (uv tool install / pip)

Install through a Nexus/Artifactory PyPI proxy:

uv tool install rolter --index-url https://nexus.internal.example/repository/pypi/simple
# or
pip install rolter --index-url https://nexus.internal.example/repository/pypi/simple

Or install a downloaded wheel with no index at all:

uv tool install ./rolter-<version>-py3-none-any.whl
# or
pip install --no-index ./rolter-<version>-py3-none-any.whl

crates.io (cargo install)

Point Cargo at a registry mirror or vendored sources via .cargo/config.toml:

# .cargo/config.toml
[source.crates-io]
replace-with = "internal"

[source.internal]
registry = "sparse+https://nexus.internal.example/repository/cargo/"

For a fully offline build, vendor the dependency sources on a connected host and copy them in:

cargo vendor vendor/                 # connected host, writes a [source] snippet
# copy vendor/ into the enclave, add the printed snippet to .cargo/config.toml
cargo build --workspace --offline

Building from source (cargo + bun)

The Rust build follows the crates.io section above. The UI needs an internal npm mirror for bun install:

# ui/bunfig.toml
[install]
registry = "https://nexus.internal.example/repository/npm/"

# alternatively ui/.npmrc
registry=https://nexus.internal.example/repository/npm/

cd ui && bun install && bun run build
cargo build --workspace --release --offline

Operator checklist

Must be reachable inside the enclave:

• Every configured provider base_url (your internal model servers or a proxied provider endpoint).
• Postgres, Redis, and ClickHouse hosts — only for the features you enable.
• The control plane, if the gateway runs with --snapshot-url.

Must NOT be required:

• Public package registries at run time (only at install/build time, through the mirror).
• CDNs (cdn.jsdelivr.net, fonts.scalar.com, npm/unpkg) — rolter references none.
• Telemetry endpoints — unless you deliberately set OTEL_* to an internal collector.
• Provider status pages — leave status_page_url unset.

Offline smoke test

Verify a running gateway with zero external providers using the built-in fake-llm model (deterministic, no upstream or secrets needed):

rolter gateway --port 4000 &
curl -s http://localhost:4000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"fake-llm","messages":[{"role":"user","content":"hello"}]}'

A 200 with a lorem-ipsum completion confirms the gateway serves traffic with no outbound calls. Open http://localhost:4000/docs and confirm the API reference renders with no network requests leaving the host.