The Streaming Event Shape of the OpenAI Realtime API

The Realtime API is not a streamed HTTP response with a different content type. It is a bidirectional session over a WebSocket in which both sides send discrete JSON events at any time, and almost everything that is hard about building on it comes from that word bidirectional.

The envelope

The connection is a WebSocket to wss://api.openai.com/v1/realtime with the model as a query parameter and your key in an Authorization header. There is also a WebRTC transport intended for browsers, which carries the same events over a data channel while the audio itself rides the media track — the event vocabulary below is identical on both. What you may put on the wire as audio is a separate question, answered by the accepted input audio formats.

Every message in either direction is a JSON object with two fields in common:

{
  "event_id": "event_1920",
  "type": "input_audio_buffer.append",
  ...type-specific fields
}

type is a dotted string naming the event. event_id is server-assigned on inbound events; on outbound events you may supply your own, and it will be echoed on any error the server raises about that event. Supplying it is the only way to correlate a failure with the specific message that caused it, and it is worth doing from the first line of code rather than added later during a debugging session.

This is a different contract from the server-sent-event chunks of a streamed chat completion, where one request produces one ordered stream that ends. Here the socket outlives any individual turn, events for different turns can be in flight together, and nothing guarantees you are only ever assembling one thing at a time.

Two structural nouns run through all of it. A conversation is an ordered list of items — a user audio turn, an assistant turn, a function call, a function result. A response is one model turn being generated, and it is a separate object with its own lifecycle. Items outlive responses; cancelling a response does not remove the items it already produced, which is the source of most state bugs in Realtime clients.

Events you send

The client vocabulary is small — nine types in the beta surface — and they divide into three jobs.

• session.update — set or change session configuration: instructions, voice, the input and output audio formats, the turn-detection mode, the tool list, temperature. Valid at any time, including mid-session, and this is how you change the tools available part-way through a call.
• input_audio_buffer.append — push base64-encoded audio into the input buffer. Sent continuously while the microphone is open. input_audio_buffer.commit ends a user turn; input_audio_buffer.clear discards what is buffered. In server voice-activity-detection mode the server commits for you and you send neither.
• conversation.item.create — insert an item directly, which is how you supply text input, prior context, or a function_call_output after running a tool. conversation.item.truncate shortens an assistant audio item to a point in time, and conversation.item.delete removes one.
• response.create — ask the model to produce a turn now, optionally with per-response overrides. response.cancel stops one in progress. Under server turn detection, response.create is usually implicit.

Events you receive

The server vocabulary is much larger, because it narrates every state change. It is easier to hold as families than as a list.

Session and conversation lifecycle

session.created arrives immediately on connect and carries the effective configuration, including defaults you did not set — read it rather than assuming. session.updated confirms each session.update. conversation.created arrives once.

Input audio buffer

input_audio_buffer.speech_started and input_audio_buffer.speech_stopped are the server’s voice-activity detector firing; speech_started is your cue that the user has begun talking and is the event interruption handling hangs off. input_audio_buffer.committed announces that a user audio item has been closed, and input_audio_buffer.cleared confirms a clear.

Items

conversation.item.created fires for every item from either side. Transcription of user audio arrives separately and later, as conversation.item.input_audio_transcription.completed or its .failed counterpart — separately because transcription runs alongside the model rather than gating it, which means the transcript of a user turn can arrive after the assistant has already started answering it. Any UI that displays both must tolerate that ordering.

Response lifecycle

This is the nested part, and the nesting is strict: a response contains output items, an output item contains content parts, and a content part emits deltas.

response.created
  response.output_item.added
    response.content_part.added
      response.audio.delta            ← repeated, base64 PCM16 audio
      response.audio_transcript.delta ← repeated, text of what is being said
      response.text.delta             ← repeated, for text modality
      response.function_call_arguments.delta  ← repeated, JSON fragments
    response.content_part.done
  response.output_item.done
response.done                          ← carries status and usage

Each .delta family has a matching .done carrying the assembled value, so you may accumulate deltas yourself for responsiveness and then reconcile against the .done — which is the right pattern for function-call arguments in particular, since a JSON fragment stream is not parseable until it is complete. response.done carries the final status (including cancelled or failed) and the token usage for the turn, broken down by modality. Alongside all this, rate_limits.updated reports your remaining budget, and error can arrive at any point without ending the session.

One turn, event by event

Here is a single exchange with a tool call, annotated. Arrows show direction; the shape is the documented sequence rather than a captured log.

→  session.update              set voice, tools, server VAD
←  session.updated             effective config echoed back

→  input_audio_buffer.append   ×N, ~100ms of audio per event
←  input_audio_buffer.speech_started   VAD heard the user begin
←  input_audio_buffer.speech_stopped   VAD heard silence
←  input_audio_buffer.committed        user item closed
←  conversation.item.created           the user's audio item

←  response.created                    model turn begins
←  response.output_item.added          item is a function_call
←  response.function_call_arguments.delta  ×N, JSON fragments
←  response.function_call_arguments.done   complete arguments string
←  response.output_item.done
←  response.done                       status: completed

// you now run the tool and hand the result back
→  conversation.item.create      type: function_call_output, with call_id
→  response.create               ask for the spoken answer

←  response.created
←  response.output_item.added
←  response.content_part.added
←  response.audio_transcript.delta ×N   text, for display
←  response.audio.delta           ×N    base64 PCM16, for playback
←  response.audio.done
←  response.content_part.done
←  response.output_item.done
←  response.done                  usage for the turn

←  conversation.item.input_audio_transcription.completed
   // the user's transcript, arriving after the answer

Two things in that trace are worth pausing on. The tool call takes a complete response lifecycle of its own and ends with response.done — the model finished its turn by asking for the function, and the spoken answer is a second response that you have to request. And the user’s transcript really does land last, after the assistant has already spoken.

Interruption, which is the hard part

In a text stream, the user cannot talk over the model. In a voice session they do it constantly, and handling it correctly is three coordinated actions rather than one.

1. On input_audio_buffer.speech_started while a response is active, stop local audio playback immediately. This is the only step the user perceives, and it must not wait for a server round trip.
2. Send response.cancel so the model stops generating rather than continuing to produce audio nobody will hear and tokens you will be billed for.
3. Send conversation.item.truncate with the number of milliseconds of that assistant item the user actually heard. This is the step that is easy to skip and expensive to skip: without it, the conversation history contains the full assistant turn, so the model believes it said things the user never heard, and it will refer back to them. The client is the only party that knows how much audio reached the speaker, so only the client can supply this number.

Two smaller things follow from the event shape and cause disproportionate trouble. The audio deltas are base64-encoded PCM16 at the sample rate the session was configured for, which means the event stream is genuinely large — base64 adds a third on top of raw PCM, and raw PCM is not compressed to begin with. A handler that logs every event verbatim will fill a disk during a demo, so log the type and the length rather than the payload. And because the deltas are audio frames rather than text, they must be played in arrival order into a buffer you control; letting a player start on the first delta without any jitter buffer produces the gaps and clicks that are usually misdiagnosed as a network problem.

The second is that the session carries its own configuration state and you did not necessarily set all of it. session.created arrives with the effective values for everything — voice, formats, turn detection thresholds, tools — including defaults you never specified. Reading that event and asserting on the fields you care about is a two-line check that catches the entire class of bug where a session.update was sent before the socket was ready, or was rejected, and the session quietly ran on defaults for the rest of the call.

Everything else about a Realtime client — buffering, playback, reconnection — is ordinary engineering. That truncation offset is the piece with no analogue in the request-response world, and it is the reason a voice agent that works in testing can feel subtly broken in a real conversation.