The Gemini Live API: Streaming Session Shape for Real-Time Audio and Video

The Live API is a different protocol from the rest of the Gemini API, not a flag on it. It is a bidirectional WebSocket where audio flows continuously in both directions, the model can be interrupted mid-sentence, and the session has state that the stateless generateContent endpoint does not have.

How it differs from streamGenerateContent

streamGenerateContent is a normal HTTP request whose response arrives in pieces. One request in, one response out, delivered incrementally. You cannot send anything after the request has begun.

The Live API is a persistent connection. After the handshake, either side can send at any time. That single change is what makes natural conversation possible — the user can talk over the model, the model hears it, and it stops. Everything else about the protocol follows from supporting that.

• Transport is a WebSocket rather than a chunked HTTP response.
• State lives on the server for the duration of the session. You do not resend the conversation on every turn.
• Audio is native in both directions, not a transcription pipeline bolted on either end. The model hears the audio itself rather than a transcript of it, and speaks with prosody.
• Turns are negotiated rather than implied by request boundaries, which is why there are explicit messages for turn completion and interruption.

Opening the session

Connect to the BidiGenerateContent WebSocket endpoint and send exactly one setup message before anything else. Nothing else is valid until the server acknowledges it. Documented in Google’s Live API guide.

wss://generativelanguage.googleapis.com/ws/\
google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent?key=API_KEY

// client -> server, first message on the socket
{
  "setup": {
    "model": "models/gemini-live-2.5-flash-preview",
    "generationConfig": {
      "responseModalities": ["AUDIO"],
      "speechConfig": {
        "voiceConfig": {"prebuiltVoiceConfig": {"voiceName": "Puck"}}
      }
    },
    "systemInstruction": {
      "parts": [{"text": "You are a terse booking assistant. Confirm before acting."}]
    },
    "tools": [{"functionDeclarations": [ /* ... */ ]}]
  }
}

// server -> client
{ "setupComplete": {} }

responseModalities is the field to get right first. It takes a single modality — AUDIO or TEXT — and you cannot request both spoken and written output in the same session. If you want a transcript alongside speech, enable output transcription in the setup config rather than asking for two modalities.

What the client sends

After setup, every client message is one of a small union. Which one you use is the main design decision in a Live API integration.

• realtimeInput — continuous, unsolicited media. Microphone audio chunks and video frames go here. The server handles turn detection itself: it decides from the audio when the user has stopped speaking. This is the message for live conversation.
• clientContent — structured conversational turns, with explicit turns and a turnComplete boolean. This is the message for typed text, for seeding a session with history, and for any case where you control turn boundaries rather than delegating them.
• toolResponse — results for function calls the model requested, the same functionResponse shape as the batch API.

// streaming microphone audio: 16-bit PCM, 16 kHz, little-endian, base64
{
  "realtimeInput": {
    "audio": {"mimeType": "audio/pcm;rate=16000", "data": "<base64 chunk>"}
  }
}

// a typed message with an explicit turn boundary
{
  "clientContent": {
    "turns": [{"role": "user", "parts": [{"text": "Move it to Thursday instead."}]}],
    "turnComplete": true
  }
}

The input audio format is not negotiable and getting it wrong produces a session that connects and then does nothing: raw 16-bit PCM, 16 kHz, mono, little-endian. Output audio comes back at a different sample rate — 24 kHz — so a naive playback path that reuses the input rate will play the model back at the wrong speed. That mismatch is the most common first bug.

What the server sends

• setupComplete — the handshake acknowledgement. Send nothing before it.
• serverContent — the model’s output. Carries modelTurn with the audio or text parts, and the boolean flags interrupted, turnComplete and generationComplete. Also carries inputTranscription and outputTranscription when those are enabled.
• toolCall — the model wants a function executed. Answer with toolResponse.
• toolCallCancellation — a previously requested call is no longer wanted, because the user interrupted. Cancel the work if you can; do not send a response for a cancelled id.
• goAway — the connection is about to be closed by the server, with a time remaining. This is your cue to reconnect gracefully rather than to discover the socket is dead.
• sessionResumptionUpdate — a handle you can store and present on reconnect to continue the same conversation state.

The three completion flags on serverContent are not synonyms and mixing them up produces subtly wrong UI. generationComplete means the model has finished generating; turnComplete means the turn is over and it is the user’s go; interrupted means generation was abandoned because the user spoke.

Interruption and turn boundaries

Interruption is the feature that justifies the protocol. With voice activity detection enabled, the server detects the user speaking while the model is producing audio, stops generation, and sends a serverContent with interrupted: true.

Your client has one obligation at that moment, and it is easy to miss: discard the audio you have buffered but not yet played. You are typically several hundred milliseconds ahead of the speaker. If you keep playing, the user hears the model continue talking after it has already stopped generating, and the conversation feels broken in a way that is hard to diagnose from the server side.

If you would rather manage turns yourself, voice activity detection can be disabled in the setup configuration, at which point you signal the start and end of user speech explicitly. That is the right choice for push-to-talk interfaces and for noisy environments where automatic detection triggers on background sound.

Session limits and what to plan for

• Sessions have a maximum duration, documented separately for audio-only and audio-plus-video sessions. A long-running assistant must handle the session ending and resume.
• The context window fills. A session accumulates everything said in it. Context window compression, enabled in the setup config, is the documented mechanism for extending a session past that point.
• Reconnection needs a plan. Store the handle from sessionResumptionUpdate, act on goAway rather than waiting for the socket to close, and reconnect before the deadline.
• API keys do not belong in a browser. The WebSocket URL carries the key as a query parameter, so a direct client-side connection exposes it. Proxy the socket through your own server, or use the ephemeral token mechanism Google documents for client-side connections.