Caching

rolter deals with two distinct kinds of caching.

1. KV-cache affinity (load balancing)

The big win for self-hosted fleets: vLLM/SGLang reuse the attention KV cache for shared prompt prefixes (system prompts, few-shot examples, conversation history). But that only helps if the next matching request lands on the same replica. Naive round-robin scatters related requests and destroys cache locality.

The cache_aware strategy keeps, per target, a byte trie of prompts it has served. For an incoming prompt it computes the fraction of leading bytes already present on each target and:

• if the best match ≥ threshold (default 0.5), pins the request to that target (cache hit) — unless that target is overloaded (below)
• otherwise spreads to the least-loaded target, or the least-warmed one when no load is known

The “prompt” is the conversation, not the request body (#1851). prompt_affinity::affinity reads each turn’s role and text in the order the model sees them — messages for chat, system plus messages for Anthropic, instructions plus input for Responses, prompt for completions — and the raw body is only the fallback for a shape it cannot read. Before #1851 the trie was fed the raw JSON, and the envelope every request to a model opens with ({"model":"…","messages":[{"role":…) cleared the threshold by itself, so unrelated requests all pinned to whichever replica served first.

Only the first 32 KiB of that text is copied into RouteContext::prompt, cut on a character boundary, since prefix affinity is decided by the leading bytes: the trie walk stays bounded and a long prompt never costs a second full copy of itself. The whole prompt is still measured and hashed as it is read, and travels as RouteContext::prompt_len and RouteContext::prompt_digest (xxh3). Both cover what the text leaves out. Tool calls and results (Anthropic tool_use / tool_result, OpenAI tool_calls, Responses function_call / function_call_output) are prefilled like any other token and are often most of an agent’s prompt, so they count towards the length and the digest. Images, audio and files count towards the digest only, since their encoded size says little about their token cost. Neither is copied: a non-string value is serialized straight into the hasher. prompt_affinity::route_context builds the context for every JSON request, so the proxy and the tests hand the balancer the same thing. Each prompt reader takes the part it needs:

• cache_aware and the pipeline’s prefix scorer read the bounded text.
• consistent_hash without a session keys on the digest, so requests that share a long system prompt and differ after it, or differ only in an attached image or tool result, still spread over the ring. Hashing the bounded text would send all of them to one target.
• The latency predictor’s token estimate is prompt_len / 4, so a long prompt keeps growing in size past the cut instead of stopping at 8192 tokens, and a tool-heavy agent turn is sized by its tool results too.

The load guard. Affinity alone is winner-take-all: the replica that served a shared system prompt first holds the longest match for every request that shares it. After SGLang’s cache-aware router, CacheAware::pick keeps the affinity only while the warm target’s in-flight count stays within both balance margins of the least-loaded target — more than BALANCE_ABS_THRESHOLD (2) requests ahead and more than BALANCE_REL_THRESHOLD (1.5×) its load is an imbalance, and the request goes to the least-loaded target instead. The absolute margin lets a warm replica take a short queue for its cache; the relative one keeps a busy pool from treating a gap of a request or two as an imbalance. With no load known there is nothing to balance against, and affinity stands.

Every comparison runs over the targets the gateway can use for the attempt. pick_untried passes its eligibility (not tried yet, not parked on a cooldown, healthy, breaker not open, allowed to the caller’s key) through LoadBalancer::pick_eligible, and CacheAware restricts the affinity match, the least-loaded minimum and the fallbacks to those targets. A replica that takes no traffic drains to an empty queue, and counting its zero made every warm replica with three requests in flight look overloaded, which sent the spill to the dead replica and from there to target 0 by index. When nothing is eligible the pick weighs every target, as a plain pick does, and the gateway fails open as before. The variant path passes the same rules to each variant’s balancer (variant_target_eligible). The eligibility is asked once per target and kept in a word for routes of up to 64 targets, so a pick allocates nothing. Each answer still takes the cooldown and breaker locks, and power_of_two and the pipeline stacks still pick blind to it; both are #1942.

This is approximate (no coupling to the engine). The per-target trie is capped at a node ceiling (default 1M nodes) with LRU eviction: inserting past the cap drops the least-recently-inserted prompt, pruning only the nodes that become unreferenced (shared prefixes survive). Each trie exposes an eviction counter for observability.

flowchart TD
  R[incoming prompt] --> S{best prefix match >= threshold?}
  S -- yes --> B{best target past both balance margins?}
  B -- no --> P[pin to best target<br/>cache hit]
  B -- yes --> L
  S -- no --> L[least-loaded / least-warmed target]
  P --> O[observe: insert prompt into target trie]
  L --> O

Precise vLLM mode

Set a provider’s [providers.kv_events] block and choose strategy = "precise_cache_aware". rolter supports the vLLM V1 msgpack KVEventBatch protocol documented for vLLM 0.22: a three-frame ZMQ publication (topic, big-endian sequence, payload) containing tagged BlockStored, BlockRemoved, and AllBlocksCleared events. BlockStored.token_ids and block_size derive stable local prefix identities; external block hashes allow exact removals. State is capped by max_blocks per provider.

Exact scoring needs the token ids produced by the same tokenizer as vLLM. Send them in x-rolter-vllm-token-ids as comma-separated unsigned integers. Without the header, with stale/malformed events, or after a sequence gap, the scorer is neutral and least-load routing takes over. A sequence gap clears the local index and precise scoring remains disabled until an AllBlocksCleared event establishes a clean boundary.

[[providers]]
name = "vllm-a"
kind = "openai_compatible"
api_base = "http://vllm-a:8000"

[providers.kv_events]
endpoint = "tcp://vllm-a:5557"
topic = "kv-events"
max_blocks = 1000000
stale_secs = 30

vLLM must enable KV events with the ZMQ publisher and matching topic. Metrics expose consumed/malformed events, stream failures, decision count, and per-provider freshness.

LMCache-aware mode

Set [providers.lmcache] and use strategy = "lmcache_aware". The supported controller signal is an HTTP 200 JSON object:

{"occupancy": 0.42, "cache_available": true}

occupancy is clamped to [0,1]; available targets score 1 - occupancy, unavailable targets score zero. Polling happens in the background. Failed, malformed, or stale signals are neutral, so existing least-load routing continues.

[providers.lmcache]
endpoint = "http://lmcache-a:9000/v1/occupancy"
refresh_secs = 2
stale_secs = 10

2. Response cache

Optional caching of full responses to cut cost/latency for repeated requests:

• exact: hash of the normalized request → cached response (Redis), short TTL, opt-in per route/key.
• semantic: after an exact miss, embed the normalized conversation text through a configured provider and compare cosine similarity against a bounded recent-entry window in Redis. The route controls the threshold and candidate cap. Embedding, Redis, and decode failures fail open to normal routing. Only compatible requests are ever compared — see Semantic compatibility.

Streaming responses are cached on completion and replayed as a synthetic stream. Cache status is surfaced via response headers (e.g. x-rolter-cache: hit|miss).

[cache]
enabled = true

[routes.cache]
enabled = true

[routes.cache.semantic]
provider = "openai"
model = "text-embedding-3-small"
threshold = 0.92
max_candidates = 256

Post-response policy on cached responses

Enabling the cache must never change which policy a completion passes through (#1477). A buffered, successful body is finalised by exactly one function, DeliveryPolicy::apply in crates/rolter-gateway/src/handlers.rs. It runs the built-in output guardrails, then the post_response plugin chain, then the PII sanitizer’s response leg, and returns either the body or the refusal to send instead. Every delivery path calls it:

The entry is written before the policy runs, so it is always the upstream’s own bytes. A transform is re-applied per delivery. A rule or plugin installed by a hot reload reaches entries cached before it. A PII restore uses the ticket of the request being answered and is never persisted. A refusal is returned as Delivery::Refuse with the reason for the log row. On the live and cacheable-miss paths it goes through withheld_response, so the upstream call is still billed. A refused hit writes a zero-cost withheld row (see Billed but withheld). Streamed responses skip all three stages exactly as they do uncached (the request path has already refused the stream or had it opted out), so a cached SSE replay gets the same treatment as a live stream.

crates/rolter-gateway/src/cache_policy_tests.rs holds the HTTP regression suite. It covers cache off, the first miss, and exact and semantic hits, with blocking, transforming and unreachable (fail-closed and fail-open) plugins, a hot reload that installs a plugin over existing entries, and per-request PII restoration with a check that nothing restored reaches the store.

Semantic compatibility

A semantic hit replays a reply produced for a different request, so similar wording is not enough on its own: two requests can read alike and still expect incompatible replies (#1476). crates/rolter-gateway/src/semantic.rs splits every request into two parts before any similarity is computed:

• the text that is embedded: the user and assistant turns (with an OpenAI name), or the prompt of a legacy completion;
• a partition: a sha-256 over the canonical, key-sorted form of every other field of the post-injection body, including stream, stream_options, tools, tool_choice, response_format, sampling and length parameters, the model, and the dialect. OpenAI system/developer messages (with their position) and the Anthropic top-level system are partition fields too, so instructions are matched exactly rather than by similarity. "Answer in German" and "Answer in Japanese" embed almost identically, so this is the only safe way to compare them.

The partition is part of the Redis index key, so the candidate scan never sees an entry from another partition. It is an allowlist of what may vary, not a list of what must match. Only the conversation text, user and metadata are left out (attribution that does not change the reply), and stream: false is folded into an absent stream. A field the gateway does not know splits the cache, which costs hit rate but never correctness. As a last check before replay, a hit whose stored content type disagrees with the caller’s stream flag is treated as a miss.

Requests whose meaning the text would not capture bypass semantic lookup before the embedding call, and so cost no embedding spend. They still use the exact cache.

Offering tools in the request (tools, tool_choice) is supported. The schema is a partition field, so a reply is only replayed to a request that offered exactly the same tools.

The index key carries a layout version (<namespace>:semantic:v2). Entries written before #1476 had no partition and live under the old <namespace>:semantic key, so after an upgrade they are never read again and expire on their TTL. Bump SEMANTIC_LAYOUT_VERSION whenever the partition rules change.

The HTTP regression suite is crates/rolter-gateway/src/semantic/http_tests.rs. It serves the gateway in-process against a stub upstream whose embeddings are identical for every input, with the in-memory cache backend (ResponseCache::in_memory, test builds only), so it needs neither Redis nor a paid model. Because similarity alone would match any two requests there, each miss it asserts is the partition or the bypass at work, and each hit shows the partition does not over-split.