Realtime

Realtime is for voice agents and anything conversational. One session is one WebSocket. You send text as it is produced and get audio back as fast as the engine makes it. Each turn of speech runs in a context, and a session can run several contexts at once.

Two contexts on one WebSocket session: text goes out as task_start and task_continue; audio comes back per context; cancelling a context drops the audio still in flight

Say something

const rt = await client.realtime.connect();

for await (const event of rt.say({ voice_id: voiceId, text: "Hi! How can I help you today?" })) {
  if (event.type === "audio") speaker.write(event.pcm); // s16le, 24 kHz, mono
}

await rt.close();

Stream text from an LLM

Send text a token or a sentence at a time, then finish() the turn:

const ctx = rt.context({ voice_id: voiceId });

const playing = (async () => {
  for await (const pcm of ctx.audio()) speaker.write(pcm);
})();

for await (const token of llm.stream(prompt)) await ctx.send(token);
await ctx.finish();
await playing;

Interruptions (barge-in)

When the user starts talking, cancel the turn. Audio still on its way for that context is dropped, and iterating the context ends with a cancelled event.

await ctx.cancel();

Events

Iterating a context yields, in order:

`type`	When
`"task_started"`	The turn began. Has the `sample_rate` and `audio_format` of its audio.
`"audio"`	A chunk of audio: `pcm`, `sentence_index`, `chunk_index`.
`"sentence_done"`	A sentence finished: `text`, `audio_duration_s`.
`"task_done"`	The turn finished (last event): `total_audio_duration_s`.
`"task_cancelled"`	The turn was cancelled (last event).
`"unknown"`	An event this SDK version doesn’t know yet: `event_type`, `data`.

A turn that fails throws RealtimeError (for example with code voice_not_found).

Limits and reconnects

A session runs up to 4 contexts at once, and holds one of your plan’s concurrent-synthesis slots until it closes. Opening one more than the plan allows throws RateLimitError with code concurrency_limited (wait retryAfter seconds), so reuse a session.
If the connection drops, running contexts throw RealtimeError. Check its retryable flag. The next context reconnects automatically; turn this off with connect({ autoReconnect: false }).
Running out of credits ends the session with a RealtimeError whose code is insufficient_credits.
Text is billed per character as it is sent. A session reserves credits in blocks of up to 1,000 characters and returns what it doesn’t use when it ends.

To use realtime from a browser or device, connect with a short-lived token. The wire protocol behind the SDK is described in Realtime WebSocket.

TypeScript SDK

Reference

Say something

Stream text from an LLM

Interruptions (barge-in)

Events

Limits and reconnects

​Say something

​Stream text from an LLM

​Interruptions (barge-in)

​Events

​Limits and reconnects

Say something

Stream text from an LLM

Interruptions (barge-in)

Events

Limits and reconnects