Realtime metering

A /v1/realtime session is a WebSocket that can stay open for an hour and run hundreds of model responses. Until #1396 it was admitted against the process-local [realtime] caps and nothing else: no budget, no rate limit, no request-log row. This page states how a session is admitted, how it is metered and what happens when its budget runs out.

The code is in crates/rolter-gateway/src/realtime.rs (admission and the relay loop) and crates/rolter-gateway/src/realtime_metering.rs (the meter).

Admission

The upgrade request goes through the same checks as an HTTP request, keyed on the same scope chain (key, project, team, org, plus the business unit and customer the key is attributed to). They run before any upstream is dialled:

Opening a session is one request against rpm. The process-local cap is checked first so that a session this gateway could not hold anyway never costs the key a slot. The tokens a session’s turns use reach tpm as they are metered, so a key that spent its token window over a socket is refused its next session, or its next HTTP request, like any other usage.

A full tpm window does not close a live session. Rate limits refuse new work, and the HTTP path does not cut off a response in flight either.

The metering unit: the response turn

A session is metered in the unit the upstream bills and reports. Every response.done server event carries the usage of the response it ends, with input_tokens, output_tokens, total_tokens and input_token_details.cached_tokens, and that is what gets priced against the route model’s [[model_prices]] row.

The alternatives were weighed and rejected:

• Metering on close leaves a session that never closes cleanly unaccounted until it is killed, and a session that ends because the network dropped may never be accounted at all.
• Metering wall-clock time bills something no provider charges for, and cannot be reconciled with a provider invoice.
• Estimating from audio bytes duplicates the provider’s own tokenizer badly, while the provider reports the exact count on every turn.

Reading the event stream cheaply

The relay hands every upstream text frame to a TurnTracker. Almost every frame is a delta carrying base64 audio, often tens of kilobytes, so parsing each one would put a JSON decode of the whole audio stream on the relay’s path. The tracker reads the event type from the leading bytes of the object instead (the first key, or the second behind event_id, which are the two places the Realtime API writes it) and fully parses only response.created and response.done, which are small. A frame whose type sits anywhere else falls back to a substring scan: slower, but still metered correctly.

response.created opens a turn and starts its clock, and response.done closes it. Time to first token is stamped by the first response.*.delta that names the turn in its response_id, or by the first one naming none. Other deltas do not count: conversation.item.input_audio_transcription.delta streams the user’s own words, often well before the model answers. A done whose created was never seen is still billed, with zero latency.

The periodic flush

A closed turn goes to the session’s meter task over a bounded channel. The relay never waits on Redis or ClickHouse; it carries live audio.

Every [realtime] usage_flush_secs (default 15), and once more when the session ends, the meter flushes:

1. one request_logs row per turn, attributed like an HTTP row: org, team, project, key, business unit, customer, model, provider, target and variant. Its request_id is the upgrade request’s id with the turn’s ordinal appended (<id>:1, <id>:2, …), so every row is unique and one session’s rows share a prefix and a trace_id
2. the window’s cost added to every applicable budget counter, in one write
3. the window’s tokens added to every applicable tpm window
4. a re-read of the session’s budgets

Rows are per turn rather than per window so that latency, time to first token and token counts in analytics describe one model response, as they do for HTTP. The flush window only batches the counter writes and bounds how late enforcement can be.

usage_flush_secs = 0 flushes after every turn instead of on a timer. The meter still ticks once a second in that mode, but only to re-read the budgets while the session is quiet (see the next section), so 0 is the tightest setting on both counts. There is no setting that turns metering off.

The snapshot is read per flush, so a price, budget or limit edited mid-session applies from the next flush on.

A budget that runs out mid-session

The meter re-reads the budgets on every tick, not only after this session spent something: a budget is shared by its whole scope chain, so another session or plain HTTP traffic may be what used it up. A tick is one usage_flush_secs, or one second when that is 0. The re-read costs nothing when no budget applies to the session or Redis is not configured.

When a budget is spent the session is closed. The client first receives a Realtime error event whose error object is the HTTP refusal’s:

{
  "type": "error",
  "event_id": "event_rolter_…",
  "error": {
    "type": "insufficient_quota",
    "code": "insufficient_quota",
    "message": "budget exceeded for Org 'acme' (limit $500.00)",
    "param": null
  }
}

It then receives a close frame with code 1008 (policy violation), and the upstream leg is closed. rolter_budget_blocks_total counts the close.

Closing is the conservative choice of the three the issue named:

• Degrading has nothing to degrade to. The session is pinned to one upstream and model for its life, and switching mid-session is exactly the failover the relay refuses to do because replaying audio or tool events is unsafe.
• Allowing the session to completion turns a budget into a suggestion for an hour at a time, which is the bug #1396 was filed about.

The overshoot is bounded: at most the spend of one flush window, plus whatever turn was in flight when the budget ran out. An operator who needs a tighter bound lowers usage_flush_secs.

Responses cut short

A response still in flight when the session ends never reports usage, but the upstream generated part of it and will bill it. The last flush writes a row for it with usage_unknown = 1, zero tokens, and the reason the session ended as its status:

This follows Client disconnects and Billed but withheld: spend that cannot be counted is marked as unknown, never recorded as free. A row cut short by the client also counts on rolter_client_disconnects_total.

These rows do not count against the target’s health. A client leaving, a session limit or a spent budget is not the upstream’s failure, and when the upstream leg did break, the relay has already attributed that to the target once for the whole session.

A turn whose response.done says status: "failed" is logged with status 502. Its usage is billed if the upstream reported any.

Shutdown

A session outlives the HTTP request that opened it. axum’s graceful shutdown stops tracking a connection once it upgrades to a WebSocket, so on its own it would let the process exit under live sessions. The runtime would then cancel each relay and meter wherever it stood, and every turn still waiting for the next flush would lose its row, its budget charge and its tpm tokens. With the default usage_flush_secs that is up to 15 seconds of every session, on every rollout.

So the gateway drains realtime sessions itself (Sessions in realtime.rs, AppState::drain_realtime_sessions):

1. Every relay and meter task is tracked from admission on, including the moment between the upgrade response and the completed handshake.
2. When SIGTERM or Ctrl-C arrives, every session is told to close at once, in parallel with axum’s drain of plain HTTP requests. Each client receives a 1001 (going away) close, the code a WebSocket server sends when it stops, so an SDK reconnects rather than treating the session as refused. The upstream leg is closed too.
3. New upgrades are refused with 503 from that moment.
4. Each meter runs its last flush: rows for finished turns, usage_unknown rows with status 503 for responses in flight, budget and tpm writes.
5. Once the HTTP drain is done, the process waits up to 10 seconds for the tracked tasks to finish, then exits. The sessions were closed when the signal arrived, so the wait only covers a meter still waiting on Redis, and it fits well inside the 30 seconds an orchestrator usually allows before SIGKILL. A session still open after it is logged as a warning.

The budget and tpm writes are awaited by the meter, so they land before the process exits. The request-log rows are handed to the shared ClickHouse writer, which is not drained at shutdown yet, so rows from the last [logging] flush_ms can still be lost. That gap is not specific to realtime (#1924).

Failure modes

• Redis down. Admission and recording fail open, as on the HTTP path: the session is admitted, turns are still logged to ClickHouse, and spend is not counted until Redis is back. See Redis connections.
• The meter falls behind. A turn that finds the meter’s queue full is dropped and counted on rolter_usage_records_dropped_total, the policy the HTTP usage sink already has (#1051). The queue holds 1024 turns and each flush is bounded by the Redis timeouts, so reaching it needs a thousand responses inside one stalled flush.
• The relay task dies. The meter sees its channel close, flushes what it was already handed and stops.
• The process shuts down. Covered by Shutdown. A SIGKILL, or a drain that outlives its 10 seconds, still loses whatever the meter had not written.

What is not metered

These are tracked rather than silently missing:

• Guardrails and plugins do not run on realtime events (#1880). This is why realtime still carries its stability marker: the marker’s note named guardrails alongside metering, and a subsystem graduates in the pull request that closes the last gap its note names.
• A revoked or expired key does not end a live session (#1881).
• Audio and text tokens are priced at the same rate, because a price row has one input and one output rate (#1882).
• Input-audio transcription runs a second model whose usage arrives on conversation.item.input_audio_transcription.completed, which the meter does not read yet (#1883).

Tests

crates/rolter-gateway/tests/realtime_metering.rs drives real sessions against a mock Realtime upstream: a completed turn is logged with its usage and cost, a session that stays open is accounted on the flush timer, a response cut short is logged as unknown, turns waiting on the timer are flushed when the session ends, an unpriced model is refused under block, a spent budget refuses a new session, a session is closed when its budget runs out, a budget spent by other traffic closes an idle session on the flush timer and, with usage_flush_secs = 0, on the one-second budget tick, usage is charged while the session continues, rpm applies per key rather than per process, and turn tokens fill the key’s tpm window. The Redis-backed ones read ROLTER_TEST_REDIS_URL and skip, or skip their Redis assertions, without it.

Shutdown has three: an in-process drain flushes every live session before it returns and refuses new ones with 503, a response in flight at shutdown is logged as unknown, and a real rolter-gateway child sent SIGTERM closes its session with 1001 and exits 0 only after the meter has flushed. With Redis, the first and the third route the gateway through a relay that holds every reply back by 200ms once shutdown starts. A flush charges a budget with INCRBYFLOAT and sends its EXPIRE only after that reply, so a budget key that carries its expiry proves the meter ran to the end of its flush instead of merely starting it. Sessions itself is unit-tested for holding the drain while a meter or an upgrade is still running.

The tracker’s frame handling, including which deltas count as a first token, is unit-tested in realtime_metering.rs itself.