> ## Documentation Index > Fetch the complete documentation index at: https://docs.finesse.flamapp.ai/llms.txt > Use this file to discover all available pages before exploring further. # Realtime (WebSocket) > Bidirectional, low-latency synthesis over a WebSocket connection. For interactive, realtime use cases the Voice API exposes a WebSocket endpoint that bridges your connection to the synthesis engine. Authenticate by passing your API key as the `api_key` query parameter (browsers and most WS clients can't set custom headers during the handshake). ``` GET wss://fi.production.flamapis.com/flam-voice-svc/api/v1/public/stream?api_key=flv_... ``` A valid key upgrades the connection (`101 Switching Protocols`); an invalid key fails the handshake with `401`. On the JSON control frames **you** send, the service replaces `voice_id` with your enrolled engine voice for that id — so you always reference **your own Flam voice ids** (the 24-char ids from [List voices](/voice-api/list-voices)), never the engine's. Everything else is forwarded verbatim in both directions. ## Session model One connection is one **session**; a session can run up to **4 concurrent contexts** (independent synthesis streams), each keyed by a client-chosen `context_id`. You start a context, optionally keep appending text to it (ideal for LLM output arriving in chunks), and finalize it when the turn is done. Audio for each context streams back as it's synthesized. * **You send** — JSON text frames. The `type` field dispatches the message (see below). * **You receive** — JSON text frames (status/control) interleaved with **binary frames** (raw audio). * **Limits** — text frames over **1 MB** close the socket (`1009`); a session idle for **600 s** is closed by the server (`session_ended`, reason `idle`); per-key connection caps close with `4429`. ## Messages you send ### `task_start` — begin a context | Field | Required | Default | Notes | | - | - | - | - | | `type` | yes | — | `"task_start"` | | `context_id` | yes | — | client-unique per session; duplicate → `context_already_exists` | | `voice_id` | yes | — | **your Flam voice id**; unknown/unenrolled → `voice_not_found` | | `text` | no | `""` | text to synthesize; ≥ 10,000 chars → `text_too_long`; empty with `finalize: true` → `text_required` | | `finalize` | no | `false` | `true` = no more text for this context; flush and finish | | `language` | no | `"en"` | invalid → `invalid_language` | | `output_format` | no | `f32le` | one of the [audio formats](#audio-formats); invalid is coerced to the default | | `sample_rate` | no | `24000` | native rate is 24000; only upsampling above it | | `num_step` | no | `32` | 1–32; quality vs latency | | `expressiveness` | no | — | 0–1; overrides `guidance_scale` | | `guidance_scale` | no | `2.0` | 0–10 | | `length_scale` | no | `1.0` | (0, 5]; speaking-rate control | | `idle_flush_ms` | no | `800` | clamped 200–5000; auto-flush a pending sentence after this much silence from you | ### `task_continue` — append text to an open context ```json theme={null} { "type": "task_continue", "context_id": "ctx-1", "text": "And another sentence.", "finalize": false } ``` Send as many as you like; set `finalize: true` on the last one to flush and finish the context. ### `task_cancel` — abort a context ```json theme={null} { "type": "task_cancel", "context_id": "ctx-1" } ``` ### `session_end` — graceful close ```json theme={null} { "type": "session_end" } ``` The server replies with `session_ended` and closes the socket. ## Messages you receive `{context_id, sample_rate, audio_format}` — the **effective** rate and format for the context. Use these (not what you requested) to configure playback. `{context_id, sentence_index, chunk_index, audio_duration_s, byte_length}` — **immediately followed by one binary frame** of raw PCM (or opus) bytes. Pair them in order: each `audio_chunk` announces the binary frame that follows it. `{context_id, sentence_index, text, audio_duration_s}` — one sentence fully synthesized. `{context_id, num_sentences, total_audio_duration_s, profile, trace_id}` — the context is complete. `profile` carries timing detail (`ttfb_ms`, `generate_ms`, `overall_rtf`, …). `{context_id, sentences_completed, sentences_dropped, trace_id}` — reply to `task_cancel`. `{context_id, sentences_completed, code, trace_id}` — terminal failure for one context. `{context_id?, code, message, fatal}` — validation or billing errors. When `fatal` is `true` the session is closing; otherwise only the offending message was rejected. `{reason}` — `idle` or `client_request`. The socket closes after this frame. ## Audio formats Binary frames carry **raw PCM** (no container) in the context's `output_format`: | Format | Bytes/sample | Notes | | - | - | - | | `f32le` | 4 | **default**; float32 in \[-1, 1) | | `s16le` | 2 | best choice for web playback and WAV wrapping | | `s24le` / `s32le` / `f64le` | 3 / 4 / 8 | niche pipelines | | `opus` | — | 48 kHz, 20 ms frames; can stall under heavy chunking — prefer PCM on WS | Audio is **mono** at the effective `sample_rate` from `task_started` (native 24000 Hz). Wrap or play it client-side; nothing on this path returns WAV. ## Billing WebSocket synthesis is metered exactly like [`/speak`](/voice-api/synthesize-speech) and [SSE streaming](/voice-api/stream-speech): **you're billed per character of `text`** on every `task_start` / `task_continue` frame, against your plan's monthly character allowance. * The handshake checks your plan **before** upgrading: streaming not included in your plan → `403`; character quota exhausted → `402` (when billing enforcement is active). The handshake response carries `X-Billing-Remaining` with your remaining characters. * If your workspace runs out of credits **mid-session** (strict enforcement), the text frame that exceeded the balance is not forwarded — you receive a fatal error frame and the session closes: ```json theme={null} { "type": "error", "code": "insufficient_credits", "message": "You're out of credits - top up to continue.", "fatal": true } ``` Usage appears in your dashboard analytics and on the monthly invoice like any other synthesis. ## Close codes | Code | Meaning | | - | - | | `4401` | unauthorized (bad/missing key at the engine layer) | | `4429` | per-key connection cap exceeded | | `1009` | text frame over 1 MB | ## End-to-end example ```javascript JavaScript (browser) theme={null} const url = "wss://fi.production.flamapis.com/flam-voice-svc/api/v1/public/stream" + `?api_key=${process.env.FLAM_API_KEY}`; const ws = new WebSocket(url); ws.binaryType = "arraybuffer"; // Gapless playback: schedule each PCM chunk on a running Web Audio playhead. const audioCtx = new AudioContext(); let playhead = 0; let sampleRate = 24000; ws.addEventListener("open", () => { ws.send(JSON.stringify({ type: "task_start", context_id: "ctx-1", voice_id: "665f1a2b3c4d5e6f7a8b9c0d", // your Flam voice id text: "Hello from Finesse, streaming in realtime.", finalize: true, // one-shot turn; omit to keep appending output_format: "s16le", sample_rate: 24000, })); }); ws.addEventListener("message", (event) => { if (typeof event.data === "string") { const msg = JSON.parse(event.data); if (msg.type === "task_started") sampleRate = msg.sample_rate; if (msg.type === "task_done") ws.close(); if (msg.type === "task_failed" || (msg.type === "error" && msg.fatal)) { console.error("synthesis failed", msg); ws.close(); } return; } // Binary frame: raw s16le mono PCM for the most recent audio_chunk. const i16 = new Int16Array(event.data); const f32 = Float32Array.from(i16, (s) => s / 32768); const buf = audioCtx.createBuffer(1, f32.length, sampleRate); buf.getChannelData(0).set(f32); const src = audioCtx.createBufferSource(); src.buffer = buf; src.connect(audioCtx.destination); playhead = Math.max(playhead, audioCtx.currentTime + 0.05); src.start(playhead); playhead += buf.duration; }); ``` ```javascript JavaScript (LLM turn, multi-frame) theme={null} // Start the context with the first tokens, keep appending as the LLM streams, // finalize on the last chunk. Sentences synthesize as soon as they're complete. ws.send(JSON.stringify({ type: "task_start", context_id: "turn-42", voice_id: "665f1a2b3c4d5e6f7a8b9c0d", text: "The quick answer is yes.", })); ws.send(JSON.stringify({ type: "task_continue", context_id: "turn-42", text: " The longer answer involves a few tradeoffs worth knowing.", })); ws.send(JSON.stringify({ type: "task_continue", context_id: "turn-42", text: " Ask me about any of them.", finalize: true, })); ``` ```python Python theme={null} import asyncio, json, os import websockets # pip install websockets async def main(): url = ( "wss://fi.production.flamapis.com/flam-voice-svc/api/v1/public/stream" + "?api_key=" + os.environ["FLAM_API_KEY"] ) async with websockets.connect(url) as ws: await ws.send(json.dumps({ "type": "task_start", "context_id": "ctx-1", "voice_id": "665f1a2b3c4d5e6f7a8b9c0d", # your Flam voice id "text": "Hello from Finesse, streaming in realtime.", "finalize": True, "output_format": "s16le", "sample_rate": 24000, })) pcm = bytearray() async for frame in ws: if isinstance(frame, bytes): pcm.extend(frame) # raw s16le mono PCM continue msg = json.loads(frame) if msg.get("type") == "task_done": break if msg.get("type") == "task_failed" or (msg.get("type") == "error" and msg.get("fatal")): raise RuntimeError(msg) print(f"received {len(pcm)} bytes of PCM") asyncio.run(main()) ``` The first connection after a long idle period may be slow while the synthesis engine warms up. For latency-sensitive apps, open the socket ahead of the first turn and keep it alive. Need a one-way, fully self-described stream (e.g. for server-side playback)? [Stream speech (SSE)](/voice-api/stream-speech) uses a fixed `start` / `chunk` / `done` / `error` event shape. Use the WebSocket when you need a bidirectional, turn-based session. ## Prompt Copy this into your coding agent (Claude Code, Cursor, Codex…) to implement this endpoint. ```text theme={null} You are implementing a typed client for one Finesse Voice API endpoint: "Realtime (WebSocket)". Everything you need is in this prompt. Do not invent anything that is not specified here. WHAT TO BUILD - A RealtimeSession client (name it per the project's conventions) for bidirectional, low-latency synthesis over one WebSocket: connect() open the socket startContext(options) send task_start; returns a context handle context.append(text, finalize?) send task_continue (finalize: true on the last text) context.cancel() send task_cancel end() send session_end and wait for session_ended plus typed events per context (started, audio, sentence done, done, cancelled, failed, error) via an async iterator or callbacks, whichever fits the stack. - Use the stack's standard WebSocket client: the global WebSocket in browsers and Node 22+, the "ws" package on older Node, "websockets" in Python. Receive binary frames as ArrayBuffer/bytes. - Detect the stack first: look for package.json, tsconfig.json, pyproject.toml, requirements.txt, go.mod, Gemfile, composer.json and similar. Use the project's language, HTTP client, module layout, error-handling conventions and test runner. If there is no project yet or the stack is unclear, use TypeScript on Node 18+ with the built-in fetch and no new dependencies. - If the project already has a Finesse API client module, add this to it (reuse its auth, base URL, error type and retry helper) instead of creating a parallel one. - Keep the base URL (https://fi.production.flamapis.com) in one constant or config value. ENDPOINT AND AUTH - URL: wss://fi.production.flamapis.com/flam-voice-svc/api/v1/public/stream?api_key= - The API key goes in the api_key QUERY PARAMETER, because browsers and most WebSocket clients cannot set headers during the handshake. URL-encode it. - Read the key from the environment variable FLAM_API_KEY (or the project's secrets mechanism). Never hard-code, log or commit it, and redact it from any logged URL. - Keep this client server-side. The key can synthesize with every voice in the workspace, so it must not be shipped in a public browser bundle. If the app needs browser playback, relay audio from your server, or stop and ask me how the key should reach the browser. - A valid key upgrades the connection (101 Switching Protocols). An invalid key fails the handshake with HTTP 401. - The handshake also checks the plan BEFORE upgrading: 403 if streaming is not included in the plan, 402 if the character quota is exhausted (when billing enforcement is active). The handshake response carries X-Billing-Remaining (remaining characters). Surface these as distinct connect errors; do not auto-reconnect on 401, 402 or 403. VOICE IDS - Always send your own Flam voice ids (the 24-character hex ids from GET https://fi.production.flamapis.com/flam-voice-svc/api/v1/public/voices, whose voices are in data.voices). The service replaces voice_id with the enrolled engine voice on the frames you send; everything else is forwarded verbatim. - An unknown or unenrolled voice returns an error frame with code "voice_not_found". SESSION MODEL - One connection is one session. A session can run up to 4 CONCURRENT contexts (independent synthesis streams), each keyed by a client-chosen context_id that is unique within the session. Enforce the limit of 4 open contexts client-side and generate unique ids (e.g. a counter or UUID). - Start a context, optionally keep appending text (ideal for LLM output arriving in chunks), and finalize it when the turn is done. Audio for each context streams back as it is synthesized. - You send JSON text frames; the "type" field dispatches the message. - You receive JSON text frames (status/control) interleaved with binary frames (raw audio). MESSAGES YOU SEND task_start (begin a context): field required default notes type yes - "task_start" context_id yes - unique per session; a duplicate gives context_already_exists voice_id yes - your Flam voice id; unknown/unenrolled gives voice_not_found text no "" text to synthesize; 10,000 chars or more gives text_too_long; empty text with finalize: true gives text_required finalize no false true = no more text for this context; flush and finish language no "en" invalid gives invalid_language output_format no f32le one of the audio formats below; invalid is coerced to default sample_rate no 24000 native rate is 24000; only upsampling above it num_step no 32 1-32; quality vs latency expressiveness no - 0-1; overrides guidance_scale guidance_scale no 2.0 0-10 length_scale no 1.0 greater than 0, up to 5; speaking-rate control idle_flush_ms no 800 clamped to 200-5000; auto-flush a pending sentence after this much silence from you { "type": "task_start", "context_id": "ctx-1", "voice_id": "665f1a2b3c4d5e6f7a8b9c0d", "text": "Hello from Finesse, streaming in realtime.", "finalize": true, "output_format": "s16le", "sample_rate": 24000 } task_continue (append text to an open context; send as many as you like, and set finalize: true on the last one to flush and finish): { "type": "task_continue", "context_id": "ctx-1", "text": "And another sentence.", "finalize": false } task_cancel (abort a context): { "type": "task_cancel", "context_id": "ctx-1" } session_end (graceful close; the server replies with session_ended and closes the socket): { "type": "session_end" } Validate locally before sending: required fields present, context_id unique and at most 4 open, task_start text under 10,000 characters, num_step 1-32, expressiveness 0-1, guidance_scale 0-10, length_scale in (0, 5], and every text frame well under 1 MB. MESSAGES YOU RECEIVE (JSON text frames, each with a "type") - task_started: {context_id, sample_rate, audio_format}. The EFFECTIVE rate and format for the context. Use these, not what you requested, to configure playback. - audio_chunk: {context_id, sentence_index, chunk_index, audio_duration_s, byte_length}, IMMEDIATELY FOLLOWED BY ONE BINARY FRAME of raw PCM (or opus) bytes. Pair them in order: each audio_chunk announces the binary frame that follows it. Route that binary frame to the announced context_id, and check its length against byte_length (log a warning on mismatch). - sentence_done: {context_id, sentence_index, text, audio_duration_s}. One sentence is fully synthesized. - task_done: {context_id, num_sentences, total_audio_duration_s, profile, trace_id}. The context is complete; profile carries timing detail (ttfb_ms, generate_ms, overall_rtf, and more; type it as an open record). - task_cancelled: {context_id, sentences_completed, sentences_dropped, trace_id}. Reply to task_cancel. - task_failed: {context_id, sentences_completed, code, trace_id}. Terminal failure for one context. - error: {context_id?, code, message, fatal}. Validation or billing errors. When fatal is true the session is closing; otherwise only the offending message was rejected (fail just that context if context_id is set). - session_ended: {reason}, where reason is "idle" or "client_request". The socket closes after it. Documented error codes: context_already_exists, voice_not_found, text_too_long, text_required, invalid_language, insufficient_credits. Keep code typed as string; unknown codes must not crash. Ignore unknown message types. AUDIO FORMATS (binary frames carry raw PCM, no container, in the context's output_format) format bytes/sample notes f32le 4 default; float32 in [-1, 1) s16le 2 best choice for web playback and WAV wrapping s24le / s32le / f64le 3 / 4 / 8 niche pipelines opus - 48 kHz, 20 ms frames; can stall under heavy chunking; prefer PCM on WebSocket Audio is MONO at the effective sample_rate from task_started (native 24000 Hz). Nothing on this path returns WAV: provide a small helper that wraps mono s16le PCM in a WAV header when needed. Default the client to output_format "s16le" unless the caller chooses otherwise, and state it. LIMITS AND CLOSE CODES - Text frames over 1 MB close the socket with code 1009. - A session idle for 600 s is closed by the server (session_ended with reason "idle"). Surface that as a normal end, not an error. - Close codes: 4401 unauthorized (bad or missing key at the engine layer): do not reconnect 4429 per-key connection cap exceeded: back off before reconnecting and reuse one socket for many contexts instead of opening more connections 1009 text frame over 1 MB: a client bug; do not reconnect with the same payload - On any close, fail every still-open context with a typed error carrying the close code/reason. - Reconnect policy: never reconnect automatically on 401/402/403/4401. For other abnormal closes, reconnect only if the caller opted in, with exponential backoff, and never replay contexts that already sent text (replaying bills the text again). BILLING - Billed per character of text on every task_start and task_continue frame, against the plan's monthly character allowance, exactly like the /speak and SSE endpoints. - If the workspace runs out of credits mid-session (strict enforcement), the frame that exceeded the balance is not forwarded; you receive a fatal error frame and the session closes: { "type": "error", "code": "insufficient_credits", "message": "You're out of credits - top up to continue.", "fatal": true } Map it to a distinct out-of-credits error. LATENCY - The first connection after a long idle period may be slow while the synthesis engine warms up. Allow a generous, configurable connect/first-audio timeout, and for latency-sensitive apps connect ahead of the first turn and keep the socket alive (reuse it across turns). ACCEPTANCE CRITERIA - Connects with the key in the api_key query parameter (never logged) and surfaces 401/402/403 handshake failures distinctly. - Sends only the documented message types and fields; enforces at most 4 open contexts and unique context ids. - Pairs every audio_chunk with the next binary frame and delivers the bytes to the right context, using the effective sample_rate/audio_format from task_started. - Handles task_done, task_cancelled, task_failed, non-fatal and fatal error frames, session_ended, and close codes 4401/4429/1009 as described. - Supports multi-frame turns (task_start, then task_continue ..., finalize: true), e.g. feeding LLM output. TEST PLAN - Unit tests against a local fake WebSocket server: single-shot turn (task_start with finalize: true, s16le) collects PCM and resolves on task_done; audio_chunk/binary pairing with two interleaved contexts; a fifth concurrent context is rejected locally; non-fatal voice_not_found fails only that context; fatal insufficient_credits fails all contexts and closes; task_cancel resolves with task_cancelled; close 4429 and 4401 map to the right errors without reconnect loops; session_end resolves on session_ended. - Smoke script: - Get a ready voice id from FLAM_VOICE_ID, or by calling GET https://fi.production.flamapis.com/flam-voice-svc/api/v1/public/voices (same X-API-Key header; voices are in data.voices) and picking the first one whose status is "ready". - Connect, send one task_start with finalize: true, output_format "s16le", sample_rate 24000 and the text "Hello from Finesse, streaming in realtime.", collect PCM until task_done, wrap it as a mono WAV at the task_started sample_rate, write realtime.wav, and print time to first audio chunk, total bytes and task_done.profile. - Then run a second context as a multi-frame turn: task_start with "The quick answer is yes.", task_continue " The longer answer involves a few tradeoffs worth knowing.", and task_continue " Ask me about any of them." with finalize: true. Send session_end and exit 0 on session_ended; exit non-zero on any error. AMBIGUITIES TO ASK ABOUT, NOT GUESS - Whether a finalize-only task_continue with empty text is accepted (the spec only says empty text with finalize: true is rejected on task_start). Until confirmed, finalize by setting finalize: true on the last task_continue that carries text. - Any per-context or per-session total text limit beyond the documented per-frame limits. RULES - Implement only what this spec describes. Do not invent parameters, fields, headers, endpoints or error codes. If something is ambiguous, missing, or conflicts with the existing codebase, stop and ask me before guessing. - Match the project's existing style and conventions; do not refactor unrelated code. ``` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.