Realtime Overview

Realtime means the audio is still arriving when transcription or diarization begins. A streamed response doesn't make a complete uploaded file realtime.

1. Choose by source lifecycle

TTS can stream generated output, but it isn't the live audio-input lifecycle defined here. See the concrete transport guides for TTS behavior.

2. Choose the live transport

Use the SDK transport when the runtime supports Socket.IO. Use direct HTTP when the trusted runtime can't use Socket.IO and can implement the exact framing and response contract itself.

Both live transports require x-api-key and PCM16 little-endian, 16 kHz, mono audio. Start them from a trusted backend. Each input frame contains 16 raw UUID bytes, a flags byte, a language byte, then PCM. The concrete transport guides below show the exact layout.

3. Create one state record per stream

Before sending audio, create a fresh UUID and one app-owned state record for that stream. Track:

• the stream id and an app-assigned arrival counter;
• one replaceable provisional transcription;
• committed final-event words in observed arrival order;
• whether speech-final and stream-final signals arrived;
• closed and active diarization segments; and
• the first structured error plus the overall deadline.

Keep this state independent from the socket or HTTP reader. A transport close must not erase committed text, and a late provisional response must not replace newer or finalized state.

4. Reconcile ASR arrival order and finality

The response contains seq, but the current public Realtime contract does not define ordering or uniqueness semantics for it. Assign a local arrival number, check both final flags on every response, and apply these mutations:

When both final flags are true on one response, commit it once and record both facts. Never infer either flag from a quiet interval, a closed connection, or a completed close() call.

Build SRT or WebVTT only from finalized word timing. RealtimeSubtitles deduplicates by id:seq, so it can collapse distinct final events while the server supplies non-distinct seq values. For this contract, collect final words in observed arrival order and render them with Subtitles.

5. Reconcile diarization as an evolving timeline

For each diarization_result or HTTP diarization record:

1. Preserve and deduplicate final_segments; these closed segments don't change.
2. Replace the prior active_segments collection with the latest one; these segments can evolve or become final.
3. Treat is_final: true as the last response for that diarization stream.
4. Reconcile the latest speaker timeline with finalized word timing using an explicit overlap rule.

Speaker labels represent relative turns in this stream, not real-world identity.

6. End with separate deadlines and explicit cleanup

Use separate finite deadlines for connection, the whole session, each send or read, and the final-result wait. Then end the stream in this order:

1. Stop the audio producer so it can't enqueue more PCM.
2. Send exactly one documented final frame.
3. Wait only until the final-result deadline for the relevant final signal.
4. Record whether termination was complete, timed out, or failed.
5. Release the transport in a finally block or async context.

For realtime ASR in SDK 0.18.0, JavaScript stream.close(timeoutSeconds) and Python stream.close(timeout_seconds=...) send the final frame and wait for protocol-level is_final, a routed error, or the supplied timeout. They return when the wait expires instead of raising a timeout error. That return isn't proof that is_final arrived; inspect the state recorded by the callbacks. is_speech_final is only an utterance boundary and does not release the close wait. The diarization helper likewise returns its best-known timeline if the final wait expires.

After stream-level termination, JavaScript must still call disconnect() in finally. Python must still exit the client async context. For direct HTTP, cancel an expired request, and close its response reader.

7. Treat disconnect outcome as ambiguous

The public contracts define neither session resumption nor idempotent audio frame replay. They don't guarantee what server-side state survives a dropped Socket.IO connection or interrupted HTTP request. Recover across a new stream boundary:

1. Stop feeding the old stream and close its transport.
2. Preserve committed results, discard unresolved provisional text, and mark the uncertain audio interval.
3. Retry only if the structured error and app policy allow it and the overall deadline remains. Bound backoff, jitter, attempts, and elapsed time.
4. Reconnect with a fresh UUID and send a new start frame.
5. Keep the new stream's results separate until the app explicitly joins the two committed timelines.

Don't reuse the old UUID or replay frames under an assumption of server-side deduplication. If the app retained audio from the uncertain interval, process it through an explicit recovery path instead of silently splicing it into the new live stream. See Errors and Rate Limits.

8. Continue with a transport, reference, and recipe

Choose one transport, run its smallest representative live stream, verify the right final signal, and exercise timeout and disconnect paths before production.