You are implementing a typed client for one Finesse Voice API endpoint: "Realtime (WebSocket)".
Everything you need is in this prompt. Do not invent anything that is not specified here.
WHAT TO BUILD
- A RealtimeSession client (name it per the project's conventions) for bidirectional, low-latency
synthesis over one WebSocket:
connect() open the socket
startContext(options) send task_start; returns a context handle
context.append(text, finalize?) send task_continue (finalize: true on the last text)
context.cancel() send task_cancel
end() send session_end and wait for session_ended
plus typed events per context (started, audio, sentence done, done, cancelled, failed, error)
via an async iterator or callbacks, whichever fits the stack.
- Use the stack's standard WebSocket client: the global WebSocket in browsers and Node 22+, the
"ws" package on older Node, "websockets" in Python. Receive binary frames as ArrayBuffer/bytes.
- Detect the stack first: look for package.json, tsconfig.json, pyproject.toml,
requirements.txt, go.mod, Gemfile, composer.json and similar. Use the project's language, HTTP
client, module layout, error-handling conventions and test runner. If there is no project yet or
the stack is unclear, use TypeScript on Node 18+ with the built-in fetch and no new dependencies.
- If the project already has a Finesse API client module, add this to it (reuse its auth, base
URL, error type and retry helper) instead of creating a parallel one.
- Keep the base URL (https://fi.production.flamapis.com) in one constant or config value.
ENDPOINT AND AUTH
- URL: wss://fi.production.flamapis.com/flam-voice-svc/api/v1/public/stream?api_key=<key>
- The API key goes in the api_key QUERY PARAMETER, because browsers and most WebSocket clients
cannot set headers during the handshake. URL-encode it.
- Read the key from the environment variable FLAM_API_KEY (or the project's secrets mechanism).
Never hard-code, log or commit it, and redact it from any logged URL.
- Keep this client server-side. The key can synthesize with every voice in the workspace, so it
must not be shipped in a public browser bundle. If the app needs browser playback, relay audio
from your server, or stop and ask me how the key should reach the browser.
- A valid key upgrades the connection (101 Switching Protocols). An invalid key fails the
handshake with HTTP 401.
- The handshake also checks the plan BEFORE upgrading: 403 if streaming is not included in the
plan, 402 if the character quota is exhausted (when billing enforcement is active). The
handshake response carries X-Billing-Remaining (remaining characters). Surface these as distinct
connect errors; do not auto-reconnect on 401, 402 or 403.
VOICE IDS
- Always send your own Flam voice ids (the 24-character hex ids from
GET https://fi.production.flamapis.com/flam-voice-svc/api/v1/public/voices, whose voices are in data.voices). The service replaces voice_id with
the enrolled engine voice on the frames you send; everything else is forwarded verbatim.
- An unknown or unenrolled voice returns an error frame with code "voice_not_found".
SESSION MODEL
- One connection is one session. A session can run up to 4 CONCURRENT contexts (independent
synthesis streams), each keyed by a client-chosen context_id that is unique within the session.
Enforce the limit of 4 open contexts client-side and generate unique ids (e.g. a counter or UUID).
- Start a context, optionally keep appending text (ideal for LLM output arriving in chunks), and
finalize it when the turn is done. Audio for each context streams back as it is synthesized.
- You send JSON text frames; the "type" field dispatches the message.
- You receive JSON text frames (status/control) interleaved with binary frames (raw audio).
MESSAGES YOU SEND
task_start (begin a context):
field required default notes
type yes - "task_start"
context_id yes - unique per session; a duplicate gives context_already_exists
voice_id yes - your Flam voice id; unknown/unenrolled gives voice_not_found
text no "" text to synthesize; 10,000 chars or more gives text_too_long;
empty text with finalize: true gives text_required
finalize no false true = no more text for this context; flush and finish
language no "en" invalid gives invalid_language
output_format no f32le one of the audio formats below; invalid is coerced to default
sample_rate no 24000 native rate is 24000; only upsampling above it
num_step no 32 1-32; quality vs latency
expressiveness no - 0-1; overrides guidance_scale
guidance_scale no 2.0 0-10
length_scale no 1.0 greater than 0, up to 5; speaking-rate control
idle_flush_ms no 800 clamped to 200-5000; auto-flush a pending sentence after this
much silence from you
{
"type": "task_start",
"context_id": "ctx-1",
"voice_id": "665f1a2b3c4d5e6f7a8b9c0d",
"text": "Hello from Finesse, streaming in realtime.",
"finalize": true,
"output_format": "s16le",
"sample_rate": 24000
}
task_continue (append text to an open context; send as many as you like, and set finalize: true
on the last one to flush and finish):
{ "type": "task_continue", "context_id": "ctx-1", "text": "And another sentence.", "finalize": false }
task_cancel (abort a context):
{ "type": "task_cancel", "context_id": "ctx-1" }
session_end (graceful close; the server replies with session_ended and closes the socket):
{ "type": "session_end" }
Validate locally before sending: required fields present, context_id unique and at most 4 open,
task_start text under 10,000 characters, num_step 1-32, expressiveness 0-1, guidance_scale 0-10,
length_scale in (0, 5], and every text frame well under 1 MB.
MESSAGES YOU RECEIVE (JSON text frames, each with a "type")
- task_started: {context_id, sample_rate, audio_format}. The EFFECTIVE rate and format for the
context. Use these, not what you requested, to configure playback.
- audio_chunk: {context_id, sentence_index, chunk_index, audio_duration_s, byte_length},
IMMEDIATELY FOLLOWED BY ONE BINARY FRAME of raw PCM (or opus) bytes. Pair them in order: each
audio_chunk announces the binary frame that follows it. Route that binary frame to the announced
context_id, and check its length against byte_length (log a warning on mismatch).
- sentence_done: {context_id, sentence_index, text, audio_duration_s}. One sentence is fully
synthesized.
- task_done: {context_id, num_sentences, total_audio_duration_s, profile, trace_id}. The context
is complete; profile carries timing detail (ttfb_ms, generate_ms, overall_rtf, and more; type it
as an open record).
- task_cancelled: {context_id, sentences_completed, sentences_dropped, trace_id}. Reply to
task_cancel.
- task_failed: {context_id, sentences_completed, code, trace_id}. Terminal failure for one
context.
- error: {context_id?, code, message, fatal}. Validation or billing errors. When fatal is true the
session is closing; otherwise only the offending message was rejected (fail just that context
if context_id is set).
- session_ended: {reason}, where reason is "idle" or "client_request". The socket closes after it.
Documented error codes: context_already_exists, voice_not_found, text_too_long, text_required,
invalid_language, insufficient_credits. Keep code typed as string; unknown codes must not crash.
Ignore unknown message types.
AUDIO FORMATS (binary frames carry raw PCM, no container, in the context's output_format)
format bytes/sample notes
f32le 4 default; float32 in [-1, 1)
s16le 2 best choice for web playback and WAV wrapping
s24le / s32le / f64le 3 / 4 / 8 niche pipelines
opus - 48 kHz, 20 ms frames; can stall under heavy chunking; prefer
PCM on WebSocket
Audio is MONO at the effective sample_rate from task_started (native 24000 Hz). Nothing on this
path returns WAV: provide a small helper that wraps mono s16le PCM in a WAV header when needed.
Default the client to output_format "s16le" unless the caller chooses otherwise, and state it.
LIMITS AND CLOSE CODES
- Text frames over 1 MB close the socket with code 1009.
- A session idle for 600 s is closed by the server (session_ended with reason "idle"). Surface
that as a normal end, not an error.
- Close codes:
4401 unauthorized (bad or missing key at the engine layer): do not reconnect
4429 per-key connection cap exceeded: back off before reconnecting and reuse one socket for
many contexts instead of opening more connections
1009 text frame over 1 MB: a client bug; do not reconnect with the same payload
- On any close, fail every still-open context with a typed error carrying the close code/reason.
- Reconnect policy: never reconnect automatically on 401/402/403/4401. For other abnormal closes,
reconnect only if the caller opted in, with exponential backoff, and never replay contexts that
already sent text (replaying bills the text again).
BILLING
- Billed per character of text on every task_start and task_continue frame, against the plan's
monthly character allowance, exactly like the /speak and SSE endpoints.
- If the workspace runs out of credits mid-session (strict enforcement), the frame that exceeded
the balance is not forwarded; you receive a fatal error frame and the session closes:
{ "type": "error", "code": "insufficient_credits", "message": "You're out of credits - top up to continue.", "fatal": true }
Map it to a distinct out-of-credits error.
LATENCY
- The first connection after a long idle period may be slow while the synthesis engine warms up.
Allow a generous, configurable connect/first-audio timeout, and for latency-sensitive apps
connect ahead of the first turn and keep the socket alive (reuse it across turns).
ACCEPTANCE CRITERIA
- Connects with the key in the api_key query parameter (never logged) and surfaces 401/402/403
handshake failures distinctly.
- Sends only the documented message types and fields; enforces at most 4 open contexts and unique
context ids.
- Pairs every audio_chunk with the next binary frame and delivers the bytes to the right context,
using the effective sample_rate/audio_format from task_started.
- Handles task_done, task_cancelled, task_failed, non-fatal and fatal error frames, session_ended,
and close codes 4401/4429/1009 as described.
- Supports multi-frame turns (task_start, then task_continue ..., finalize: true), e.g. feeding LLM
output.
TEST PLAN
- Unit tests against a local fake WebSocket server: single-shot turn (task_start with finalize:
true, s16le) collects PCM and resolves on task_done; audio_chunk/binary pairing with two
interleaved contexts; a fifth concurrent context is rejected locally; non-fatal voice_not_found
fails only that context; fatal insufficient_credits fails all contexts and closes; task_cancel
resolves with task_cancelled; close 4429 and 4401 map to the right errors without reconnect loops;
session_end resolves on session_ended.
- Smoke script:
- Get a ready voice id from FLAM_VOICE_ID, or by calling GET https://fi.production.flamapis.com/flam-voice-svc/api/v1/public/voices (same
X-API-Key header; voices are in data.voices) and picking the first one whose status is "ready".
- Connect, send one task_start with finalize: true, output_format "s16le", sample_rate 24000 and
the text "Hello from Finesse, streaming in realtime.", collect PCM until task_done, wrap it as
a mono WAV at the task_started sample_rate, write realtime.wav, and print time to first audio
chunk, total bytes and task_done.profile.
- Then run a second context as a multi-frame turn: task_start with "The quick answer is yes.",
task_continue " The longer answer involves a few tradeoffs worth knowing.", and task_continue
" Ask me about any of them." with finalize: true. Send session_end and exit 0 on
session_ended; exit non-zero on any error.
AMBIGUITIES TO ASK ABOUT, NOT GUESS
- Whether a finalize-only task_continue with empty text is accepted (the spec only says empty text
with finalize: true is rejected on task_start). Until confirmed, finalize by setting
finalize: true on the last task_continue that carries text.
- Any per-context or per-session total text limit beyond the documented per-frame limits.
RULES
- Implement only what this spec describes. Do not invent parameters, fields, headers, endpoints or
error codes. If something is ambiguous, missing, or conflicts with the existing codebase, stop
and ask me before guessing.
- Match the project's existing style and conventions; do not refactor unrelated code.
