Architecture Decision Records

Lightweight decision log. Each entry: Status · Context · Decision · Consequences. Supersede rather than rewrite.

A standalone record opens with its title and a single metadata line — no table:

# Title of the decision

**Status:** Accepted · **Date:** 6 Aug 2026 · **Issues:** [#812](https://github.com/rolter-ai/rolter/issues/812)
**Supersedes:** ADR-0007 for exact vLLM modes
**Relates:** ADR-0022 (config-vs-DB tiering)

## Context

Status, Date and Issues are the line; Supersedes and Relates get their own line and are omitted when there is nothing to say. Nothing else belongs there — authorship and dates are what git is for, and a field whose value is either constant across every record or a — (unassigned) placeholder is noise that makes the actual decision harder to find. Records are English only.

ADR-0001 — Rust + Axum/Hyper/Tower for the data plane

Accepted. Need maximum proxy throughput with rich API semantics and SSE streaming. Chose Axum/Hyper/Tower on Tokio over Pingora/Actix for ecosystem fit and ergonomics. Consequence: idiomatic async stack; revisit Pingora only if profiling demands it.

ADR-0002 — Two-binary topology over shared crates

Accepted. Keep the hot proxy path lean and independently scalable from management. rolter-gateway (data plane) and rolter-control (management + UI host) share library crates. Consequence: clear seam; some duplicated wiring.

ADR-0003 — Vite + React + shadcn/ui SPA, Bun toolchain, served by Rust

Accepted. shadcn/ui (Radix + Tailwind) for the dashboard, built with Vite and managed with Bun; output is static assets served by rolter-control (no Node runtime in prod). Consequence: simple prod footprint; Bun used for install/dev/build.

ADR-0004 — Postgres + Redis + ClickHouse

Accepted. Postgres = source of truth (config/RBAC/keys/pricing); Redis = cache + rate-limit counters + config pub/sub; ClickHouse = high-volume request/cost logs. No SQLite. Consequence: three datastores to operate; each fits its job.

ADR-0005 — Org → Team → Project → Virtual Key tenancy

Accepted. Budgets and rate limits attach at any scope, most-restrictive-wins. Consequence: flexible multi-tenancy; enforcement must resolve a scope chain.

ADR-0006 — Local accounts + virtual keys + roles; SSO/LDAP later

Accepted. v1 ships local accounts (argon2id) and roles admin/member/viewer; OAuth2/OIDC and LDAP arrive as pluggable identity providers. Consequence: usable day one without an IdP.

ADR-0007 — Approximate cache-aware balancing behind a pluggable trait

Accepted. v1 uses an approximate per-target prefix trie (no engine coupling) behind LoadBalancer. Precise (KV-events) and lmcache-aware land later without API changes. Consequence: immediate wins; precise mode is additive.

ADR-0008 — Reload-free config: Postgres truth + Redis pub/sub + ArcSwap

Accepted. Control plane writes Postgres, bumps a version, publishes on Redis; gateways fetch and atomically swap an in-memory snapshot, reconciling by version. Consequence: instant fan-out, self-healing, lock-free reads.

ADR-0009 — Envelope encryption for provider secrets

Accepted. Upstream keys are AES-256-GCM envelope-encrypted with a master key from env/file; Vault/KMS backends later. Consequence: no plaintext secrets at rest; master key is the critical secret.

ADR-0010 — Packaging: maturin (uv) + cargo + Docker

Accepted. Ship a maturin-built PyPI wheel bundling the unified rolter launcher (uv tool install rolter), cargo install rolter, and a multi-stage Docker image. The rolter binary dispatches to gateway/control subcommands so one wheel/crate ships the whole system. Consequence: three distribution paths from a single named artifact.

ADR-0011 — API surface v1

Accepted. OpenAI /v1/chat/completions, /v1/completions, /v1/models and Anthropic /v1/messages. Embeddings, images, audio and other modalities follow. Consequence: drop-in for the two dominant client SDKs first.

ADR-0012 — Conventional Commits + CI PR-title lint

Accepted. Commit messages and PR titles follow Conventional Commits; enforced by commitlint, conventional-pre-commit, and a CI PR-title check. Consequence: consistent history, automatable changelogs/releases.

ADR-0013 — OpenTelemetry-based observability with engine propagation

Accepted. Export traces/metrics via OTLP to any compatible backend (SigNoz, Datadog, Grafana, Langfuse, …); propagate W3C trace context to vLLM/SGLang so engine spans join the same trace; federate upstream engine metrics. Consequence: vendor-neutral observability; an OTel Collector is the recommended hub.

ADR-0014 — Extensible API protocol translation

Accepted. Resolve translation by client/upstream protocol pair in rolter-proxy, including incremental SSE, while the gateway retains transport, caching and accounting ownership. Consequence: new provider dialects extend one translation boundary; non-equivalent modalities remain explicit and are never silently dropped.

ADR-0015 — OpenAI Responses API translation

Development. Add OpenAI Responses as a protocol pair for native OpenAI, Chat Completions and Anthropic Messages, while model-less lifecycle operations remain uniformly unsupported until tenant-scoped storage exists.

ADR-0016 — Routing OpenAI Responses resources through a tenant-scoped registry

Development. Pin Responses lifecycle operations to a bounded tenant-scoped process-local record, preserving the original provider credential while making unknown and cross-tenant IDs indistinguishable.

ADR-0017 — Provider/model addressing to disambiguate identical model names

Accepted. First-class provider-slug/model addressing coexists with named routes: a stable, URL-safe provider slug resolves provider-slug/model to a pinned (provider, upstream_model) target (avoiding LiteLLM’s base_url ambiguity). Pinning bypasses cross-provider fan-out but still balances within the provider. Slugs are org-scoped and collision-safe: deterministic reported migration backfill with -N de-dup, 409+suggestion on runtime create-conflict (never silent suffixing), soft-delete-preferred reclaim. Addendum: provider groups unify fleets (e.g. ten vLLM instances) under one group slug in the same namespace — vllm-cluster/qwen3 fans out across members with a chosen balancing strategy and passthrough model names, so no route-per-model. Consequence: new immutable slug column + proxy parsing + /v1/models + UI work + provider-groups entity (see follow-up issues).

ADR-0018 — Config mutated via granular CRUD, not whole-config replace

Accepted. The normalized Postgres store (ADR-0004, source of truth) is mutated only through the scoped CRUD API — providers, routes, targets, virtual keys — each write bumping the config version and hot-reloading gateways via the snapshot poll (ADR-0008). PostgresConfigStore::save is deliberately read-only: there is no whole-config “apply” endpoint, because a full replace would fight the normalized model and clobber concurrent edits. The dashboard Config page is a read-only effective-config viewer; the /gw reverse-proxy that fronts the gateway for the Playground is unrelated to config writes. Consequence: one live-edit path (CRUD) with instant fan-out; a raw whole-config editor would need a deliberate transactional diff/apply (guarding config-owned entries, preserving key hashes) and is out of scope (closed #494).

ADR-0019 — Per-provider egress proxy pools

Accepted. Rotate across a provider-local proxy pool, fail over only connection/tunnel failures, and quarantine repeatedly failing members. Authenticated URLs are resolved exclusively from whole-value environment references. Consequence: resilient egress without leaking credentials; health state remains process-local.

ADR-0020 — Bounded semantic response caching in Redis

Accepted. Run semantic lookup only after an exact miss, embed through an explicitly configured provider, and scan a bounded recent Redis window. Consequence: similarity reuse without another datastore; embedding and cache failures fail open and candidate search remains deliberately bounded.

ADR-0021 — External cache telemetry for routing

Accepted. Extend ADR-0007 with opt-in vLLM KV-event and LMCache occupancy scorers whose network I/O runs in background tasks. Consequence: exact/capacity-aware routing stays allocation-light and fail-open; stale or untrusted telemetry becomes neutral and least-load selection remains available.

ADR-0022 — Uniform config-vs-DB tiering for models, providers, and provider groups

Accepted. Give models, providers, and provider groups the same two-tier config wrapper over the DB (CRUD) tier: readonly entries are immutable and config-owned; default entries are seeded into the default project once at startup and then owned/editable via API/UI; pure DB rows come from CRUD. readonly wins resolution; a default colliding with a readonly key is a load-time error. Top-level [[routes]]/[[providers]]/[[provider_groups]] stay as deprecated readonly aliases for back-compat. Consequence: providers gain a seed-then-edit story and provider groups gain a full DB lifecycle (new tables/repos/seed/CRUD, tracked as follow-up PRs); the model tiering pattern is reused rather than reinvented per entity.

ADR-0023 — Propagating access-profile model policy to the data plane

Accepted. Resolve each virtual key owner’s merged access-profile policy when the snapshot is built, carry it on the key record, and enforce it on the gateway’s model and route selection; the key’s own allow-list and the owner’s policy must both permit a model. Rejects key-mint-time resolution because a later policy edit would never reach issued keys, so a revocation would silently fail to revoke. Adds bump_config_version() triggers to exactly the four tables that feed the resolution, changing migration 0058’s premise rather than its rule. Consequence: the policy /rbac/effective reports is the policy enforced, and a deployment with no access profiles is unaffected.

ADR-0024 — Dashboard UX telemetry as a structural-only event stream

Accepted. Collect dashboard usability events into a ui_events ClickHouse table beside request_logs rather than adding a product-analytics vendor, and make the schema itself the privacy guarantee: every column is a key, an enum, a duration or an id, so a form value or prompt has nowhere to land. Attribution is server-side; the ingest endpoint is guarded by CurrentUser with no capability row, because a create capability granted to any authenticated caller would break the RBAC invariant that a viewer writes nothing. Consequence: usability data with no new vendor, egress or retention policy, joinable to gateway traffic on trace_id; the cost is that any question needing a value rather than a key is unanswerable by design.

ADR-0025 — Events as logs, not span events

Accepted. OpenTelemetry is deprecating the Span Events API, so log-based events become rolter’s event model: no new explicit add_event/record_exception, and #809 (OTLP log export) sequences before #808 (GenAI conventions) because those conventions are specified as log-based events and cannot be emitted until logs are exported at all. A grep finds zero uses of the deprecated API, but tracing-opentelemetry turns every tracing event fired inside a span into a span event, so rolter is in practice an exclusive implicit user of it — which makes the migration a dependency upgrade in one bridge rather than a sweep of 57 call sites. Consequence: tracing stays; #809’s trace_id/span_id correlation becomes a blocking acceptance criterion rather than a nicety; new event types have no correct home until it lands, and should wait rather than add a span event that must later be removed.

ADR-0026 — Client control over telemetry: a kill switch now, collector-routed tenant destinations later

Accepted. Splits #812 in two. ROLTER_TELEMETRY_ENABLED ships now as one explicit off for every signal — traces, metrics, the dashboard’s browser tracing, and logs once #809 lands — that can only subtract, defaults to enabled so no existing deployment changes, and leaves export on for an unrecognized value so a typo cannot silently blind a deployment. It is environment-only despite the issue asking for a config key: telemetry::init installs the subscriber before any config file is read, so a config-file switch could not gate trace export at all. Per-tenant destinations are deferred and, when built, belong in the OpenTelemetry Collector’s routing processor keyed on an org resource attribute rather than in-process exporters — that keeps one egress path in the gateway and keeps tenant-supplied URLs, an SSRF surface, out of the data plane entirely. Consequence: operators get a reviewable “no telemetry leaves here” today; per-tenant routing later requires running a collector, which is already the recommended topology.

ADR-0027 — End-to-end test harness: Python/uv project driving a black-box stack

Accepted. The e2e suite is a Python driver on the latest CPython managed by uv, exercising the real HTTP APIs of a docker-compose stack (postgres, redis, clickhouse, control, gateway, and N llm-d-inference-sim fake-vLLM engines) with no in-process shortcuts, so the tests see exactly what an operator or tenant sees. Numbered after ADR-0026 despite its earlier date because ADR numbers are append-only and never reused. Consequence: the suite needs a container runtime rather than cargo test alone; in exchange it covers the wiring between the two binaries and the three datastores that unit tests cannot reach.

ADR-0028 — Disaggregated prefill/decode routing belongs to the engine, not the gateway

Accepted. rolter will not implement P/D disaggregation; a disaggregated fleet is one upstream and its own coordinator owns phase selection and the KV handoff. Engines move KV tensors through connector-selected transports such as NIXL/UCX. Some can coordinate that transfer through engine-specific HTTP fields, but rolter would still have to own compatible-worker topology, version-specific metadata, two upstream calls and per-request handoff state — exactly the lifecycle coupling ADR-0014 prevents. The distinguishing test is whether rolter can consume an input through a stable, engine-independent contract without joining the engine’s request lifecycle. Consequence: disaggregated fleets work with rolter today for any engine, with no code and no mid-request-handoff failure modes; revisit if a stable phase-placement contract emerges on the OpenAI or Anthropic surface.