Realtime ASR

Realtime here means PCM audio is still arriving while recognition or diarization runs. Use Fast transcription for one already-complete, bounded, latency-sensitive conversational or AI-agent unit. Use Batch for long or large complete meetings, podcasts, recordings, or archives.

This page describes the client-side Socket.IO event contract, so operation actions are relative to the client. JavaScript and Python SDK 0.18.0 wrap audio_stream and the dedicated diarization_stream; the ASR bit-2 diarization tee is a direct-event contract. speaker_id is obsolete and returns METHOD_NOT_ALLOWED.

Audio Format

Audio data is expected to be formatted as:

• PCM16 little endian
• 16 kHz
• mono

Audio payloads must contain an even number of bytes.

Limits

Four bounds apply to a realtime session. All are inclusive: exactly at the limit succeeds, and only strictly over it fails.

Per event: 16 MiB. One audio_stream or diarization_stream event must not exceed 16777216 bytes, counting the 18-byte header. Exceeding it emits PAYLOAD_TOO_LARGE with data.bound of realtime_asr_frame_bytes or realtime_diarization_frame_bytes, then closes the connection. This matches the cap the equivalent HTTP routes already enforced.

Per session, total audio content: 14400 seconds (4 hours). Accumulated accepted audio, which is a different quantity from how long the session has been open. Exceeding it emits AUDIO_DURATION_EXCEEDED with data.bound: session_audio_duration and retires the session; start a new one.

Per session, audio rate: four times real time. Audio may arrive at up to 128000 bytes per second, with a burst allowance of 16 MiB - one maximum-size frame. A client streaming genuinely in real time uses a quarter of its allowance and can never trip this; a client catching up after a network stall drains its backlog at three times real time. Exceeding it emits SESSION_BYTE_RATE_EXCEEDED with data.bound: session_audio_rate_burst and retry_after_seconds (this transport has no Retry-After header, so the wait travels in the frame), and DOES NOT close the connection: the identical payload succeeds once credit refills. No credit or quota is consumed by a rejection.

The bucket's capacity always covers the largest frame a surface accepts, so a maximum-size frame is never refused by the rate bound on a NEWLY STARTED session. On Socket.IO the bucket belongs to the CONNECTION and is shared by every audio event on it, so a maximum-size frame sent on a connection that has already streamed audio can be throttled; honour retry_after_seconds and resend it unchanged.

The per-session accumulated-byte, wall-clock-duration and idle bounds are unchanged.

Credit and billing errors

A realtime ASR stream reserves a renewable credit lease at start and renews it as it runs. Two funding outcomes are distinct and are reported with the same codes on every transport and over HTTP:

• CREDITS_EXHAUSTED — the account is out of credit. HTTP 402; retryable: false (an immediate replay cannot restore funding).
• BILLING_AUTHORIZATION_UNAVAILABLE — the billing authority could not be reached or gave an undecidable answer, so the request fails closed rather than serving unpaid work. HTTP 503; retryable: true (retry after a wait).

These may arrive at stream START (the reserve was refused) or MID-STREAM (a lease renewal was denied while audio was still arriving).

Mid-stream is terminal for the stream. When a renewal is denied the server stops accepting new billable audio, ALWAYS sends the terminal error event FIRST, then closes. Already-accepted audio up to that point is billed; nothing after the terminal event is accepted. A start-time reserve refusal is NOT terminal — the socket multiplexes other operations, so that error is emitted without closing.

Close codes. On the raw-WebSocket OpenAI-compatible endpoint the close frame carries a private-use (RFC 6455 §7.4.2, 4000–4999) code encoding 4000 + HTTP status: 4402 for CREDITS_EXHAUSTED, 4503 for BILLING_AUTHORIZATION_UNAVAILABLE. This Socket.IO transport carries no application-level close code — Socket.IO runs over, but is not, raw WebSocket and does not surface an RFC 6455 code to the application layer, so on Socket.IO the terminal JSON error event is the authoritative signal and the disconnect that follows carries no 4000-range code. Treat the error event, not a close code, as the contract on this transport.

Frame validation

Every frame is validated before any model work. A frame is rejected when it is shorter than its 18-byte header, when its stream UUID is all zeroes, when it sets a flag bit outside the three defined below, when its language byte is not one of 0, 1, 2 or 255, when its audio is empty, or when its audio length is odd. A truncated frame is answered, never silently accepted. All of these are client-fixable input faults and none is reported as a server error.

Realtime ASR buffering

Recommended to send 100ms worth of samples (1600 samples).

• Realtime ASR performs inference on 1600 samples.
• When fewer than 1600 samples are sent, Realtime ASR buffers up to 1600 samples before inference.
• When more than 1600 samples are sent, Realtime ASR segments the audio into 1600-sample chunks.

These buffering rules do not describe the dedicated diarization_stream, which forwards each received audio chunk to the diarization service.

Multiplexing

Multiple streams can share one connection. The request UUID occupies the first 16 bytes of every frame and is echoed as response id. Reuse it for one stream and use a fresh UUID for every new stream.

Language Options (STT)

Connection

Host: wss://api.voice.humain.com

Use the API_URL and API_KEY provisioned for the target environment. API_PATH is optional and defaults to /socket.io, the single path that serves every subsystem; set it only to override for a self-hosted or proxied deployment. This is Socket.IO over WebSocket, not raw WebSocket. Transport: WebSocket only — set transports: ["websocket"]. Polling is not supported.

Example (JavaScript):

const socket = io(process.env.API_URL, {
  path: process.env.API_PATH ?? "/socket.io",
  transports: ["websocket"],
  extraHeaders: { "x-api-key": process.env.API_KEY, "Origin": process.env.API_URL }
});

Example (Python — python-socketio):

async def main() -> None:
    await sio.connect(
        os.environ["API_URL"],
        headers={"x-api-key": os.environ["API_KEY"], "Origin": os.environ["API_URL"]},
        socketio_path=os.environ.get("API_PATH", "/socket.io"),
        transports=["websocket"],
    )

asyncio.run(main())

Authentication: x-api-key (header httpApiKey).

Events

Messages

Audio stream packet

audio_stream · Client → Server

Raw binary buffer containing stream metadata and audio bytes.

Content type: application/octet-stream

Raw unencoded binary buffer. Do not send JSON or base64.

Byte layout (offsets in bytes):

• 0..15: transcription_id (UUID, 16 bytes)
• 16: flags (uint8)
  • bit 0: is_start (start of stream)
  • bit 1: is_final (end of stream)
  • bit 2: diarization_enabled (tee audio to diarization service; results arrive as diarization_result events)
  • bits 3-7: reserved (0)
• 17: language_id (uint8)
  • 0 = Arabic
  • 1 = English
  • 2 = Codeswitch
  • 255 = Auto
• 18..end: audio bytes (PCM16 LE, 16 kHz, mono; even byte length)

Recommended chunk: 1600 samples (100ms).

A one-frame stream may set both start and final bits (flags=3). Bit 2 is a direct-event ASR feature; SDK 0.18.0 does not expose it.

Examples

Start of stream, English

12ab34cd56ef789012ab34cd56ef7890 01 01 <pcm16-bytes>

Mid stream, Arabic

12ab34cd56ef789012ab34cd56ef7890 00 00 <pcm16-bytes>

Final stream frame

12ab34cd56ef789012ab34cd56ef7890 02 01 <pcm16-bytes>

Speaker ID packet

speaker_id · Client → Server

Obsolete; requests return METHOD_NOT_ALLOWED.

Content type: application/octet-stream

Obsolete raw binary speaker ID buffer. The server no longer performs speaker identification and emits a non-retryable METHOD_NOT_ALLOWED error instead of forwarding audio to a SpeakerID backend.

Byte layout (offsets in bytes):

• 0..15: transcription_id (UUID, 16 bytes)
• 16..end: audio bytes (PCM16 LE, 16 kHz, mono; even byte length)

Examples

Speaker identification request

12ab34cd56ef789012ab34cd56ef7890 <audio-bytes>

Transcription result

transcription_result · Server → Client

Content type: application/json

WordSegment

Examples

Non-final result

{
  "id": "7f51f2c2-e7bc-41c8-a850-f848df2ddfc8",
  "seq": 0,
  "transcription": "hello wor",
  "words": [
    {
      "start_time": 0,
      "end_time": 0.45,
      "word": "hello"
    }
  ],
  "is_final": false,
  "is_speech_final": false
}

Final result

{
  "id": "7f51f2c2-e7bc-41c8-a850-f848df2ddfc8",
  "seq": 0,
  "transcription": "hello world",
  "words": [
    {
      "start_time": 0,
      "end_time": 0.45,
      "word": "hello"
    },
    {
      "start_time": 0.46,
      "end_time": 0.9,
      "word": "world"
    }
  ],
  "is_final": true,
  "is_speech_final": true
}

Speaker ID result

speaker_id_result · Server → Client

Obsolete; this message is no longer emitted.

Content type: application/json

Examples

speaker

{
  "id": "7f51f2c2-e7bc-41c8-a850-f848df2ddfc8",
  "speaker": "speaker-1"
}

Diarization audio stream packet

diarization_stream · Client → Server

Raw binary buffer for speaker diarization. It uses the same 18-byte envelope as audio_stream, with dedicated flag semantics. The diarization model is resolved by the server.

Content type: application/octet-stream

Raw unencoded binary buffer for the dedicated diarization_stream.

Byte layout (offsets in bytes):

• 0..15: stream UUID (16 bytes; reuse for every frame in this stream)
• 16: flags (uint8)
  • bit 0: is_start (start of stream)
  • bit 1: is_final (end of stream)
  • bits 2-7: reserved (send as 0)
• 17: framing-only language slot (ignored by diarization; send 0)
• 18..end: audio bytes (PCM16 LE, 16 kHz, mono; even byte length)

A one-frame stream may set both start and final bits (flags=3).

Examples

Start of diarization stream

12ab34cd56ef789012ab34cd56ef7890 01 01 <pcm16-bytes>

Final chunk of diarization stream

12ab34cd56ef789012ab34cd56ef7890 02 00 <pcm16-bytes>

Diarization result

diarization_result · Server → Client

Content type: application/json

SpeakerSegment

Examples

Incremental result with finalized and active segments

{
  "id": "7f51f2c2-e7bc-41c8-a850-f848df2ddfc8",
  "final_segments": [
    {
      "start_time": 0,
      "end_time": 1.5,
      "speaker": "SPEAKER_01"
    },
    {
      "start_time": 1.5,
      "end_time": 3,
      "speaker": "SPEAKER_02"
    }
  ],
  "active_segments": [
    {
      "start_time": 3,
      "end_time": 4.2,
      "speaker": "SPEAKER_01"
    }
  ],
  "is_final": false
}

Final result — all segments finalized

{
  "id": "7f51f2c2-e7bc-41c8-a850-f848df2ddfc8",
  "final_segments": [
    {
      "start_time": 3,
      "end_time": 4.5,
      "speaker": "SPEAKER_01"
    }
  ],
  "active_segments": [],
  "is_final": true
}

Error message

error · Server → Client

Content type: application/json

Examples

Invalid data type

{
  "code": "VALIDATION_INVALID_FORMAT",
  "message": "Invalid data type",
  "retryable": false,
  "timestamp": "2025-05-07T10:00:00.000Z"
}

The account already has as many concurrent realtime ASR streams in flight as its plan allows, counted across every server instance. The limit is per BILLABLE ACCOUNT, so several API keys belonging to one account share one allowance. Retryable: it usually clears within seconds. The connection is NOT closed - other operations already admitted on this socket keep running.

{
  "id": "7f51f2c2-e7bc-41c8-a850-f848df2ddfc8",
  "code": "CONCURRENCY_LIMIT_EXCEEDED",
  "message": "too many concurrent operations for this account",
  "retryable": true,
  "timestamp": "2026-01-15T10:30:00Z",
  "retry_after_seconds": 5,
  "data": {
    "limit": 8,
    "observed": 8,
    "unit": "operations",
    "bound": "account_concurrency_realtime_asr"
  }
}

The account ran out of credit while a realtime ASR stream was still running. This is TERMINAL: it is the last event on the stream and the server disconnects immediately after it. Audio accepted before this point is billed; nothing after it is accepted. NOT retryable — an immediate replay cannot restore funding. On this Socket.IO transport the disconnect carries no application-level close code, so this error event is the authoritative signal (the raw-WebSocket endpoint additionally closes with code 4402).

{
  "id": "7f51f2c2-e7bc-41c8-a850-f848df2ddfc8",
  "code": "CREDITS_EXHAUSTED",
  "message": "insufficient credits",
  "retryable": false,
  "timestamp": "2026-01-15T10:30:00Z"
}

A lease renewal could not be authorized mid-stream because the billing authority was unreachable or gave an undecidable answer, so the stream fails closed rather than serving unpaid work. TERMINAL (last event, then disconnect). Retryable after a wait — start a new stream. On the raw-WebSocket endpoint the close code is 4503; on Socket.IO this event is the authoritative signal.

{
  "id": "7f51f2c2-e7bc-41c8-a850-f848df2ddfc8",
  "code": "BILLING_AUTHORIZATION_UNAVAILABLE",
  "message": "billing authorization unavailable",
  "retryable": true,
  "timestamp": "2026-01-15T10:30:00Z"
}