Native Audio Input in the Gemini API

Gemini takes audio as a first-class input part, not as text you transcribed first. That distinction changes what you can ask, what it costs, and where it breaks—and none of those are obvious from an API surface that looks like file upload.

What it accepts

Google’s audio understanding documentation lists the supported MIME types:

• audio/wav
• audio/mp3
• audio/aiff
• audio/aac
• audio/ogg (Vorbis)
• audio/flac

The MIME type you declare must match the bytes. A request labelling an M4A file as audio/mp3 is rejected on decode rather than gracefully sniffed, and this is the most common first-attempt failure because container and codec names are not the same thing in casual usage. If you are accepting user uploads, normalise to a known format before sending rather than trusting a filename extension.

Two things happen to your audio on the way in, and both are documented rather than incidental. It is downsampled to 16 kHz, and multiple channels are combined into one. So a 96 kHz studio master costs exactly what a 16 kHz voice memo of the same length costs, and stereo separation—the interviewer on the left channel, the subject on the right—is destroyed before the model sees it. If channel separation is your speaker-identification strategy, it does not survive the upload.

The token cost

Audio is charged at a documented 32 tokens per second of duration. Not per word, not per megabyte—per second of wall clock, including silence.

tokens = duration_in_seconds x 32

One minute        60 x 32   =      1,920 tokens
Ten minutes      600 x 32   =     19,200 tokens
One hour       3,600 x 32   =    115,200 tokens
Nine hours    32,400 x 32   =  1,036,800 tokens

The bitrate independence is the practically useful part. Re-encoding a recording to a smaller file does not reduce its token cost by a single token; the only thing that reduces cost is removing time. For long recordings that means trimming silence and dead air before upload, and for meeting audio it can mean a great deal.

The comparison with video is instructive because it is the same 32-tokens-per-second audio charge with frames added on top: an hour of audio is around 115,000 tokens, while an hour of video at default resolution is an order of magnitude more. The video arithmetic is worth reading if you have a choice about which you send—for a recorded meeting where nothing on screen matters, sending audio alone is not a compromise, it is a tenfold saving.

Sending audio two ways

Small files can go inline, base64-encoded, in the request body:

{
  "contents": [
    {
      "role": "user",
      "parts": [
        { "inline_data": { "mime_type": "audio/mp3", "data": "SUQzBAAAAAA..." } },
        { "text": "Summarise the decisions taken, with a timestamp for each." }
      ]
    }
  ]
}

Inline data counts against a total request size limit—20 MB in the documented case—and base64 inflates bytes by about a third, so the practical inline ceiling is lower than it sounds. Anything larger, or anything you will ask more than one question about, goes through the File API and is referenced by URI:

{
  "contents": [
    {
      "role": "user",
      "parts": [
        { "file_data": { "mime_type": "audio/mp3", "file_uri": "https://generativelanguage.googleapis.com/v1beta/files/abc123" } },
        { "text": "Who raised the objection about the timeline, and roughly when?" }
      ]
    }
  ]
}

Uploaded files are held for a limited period—48 hours in the documented case—and then deleted. Re-uploading an hour-long recording because the handle expired is free in tokens but not in time, so store the URI with its expiry rather than discovering it.

What this is not: a transcription pipeline

A speech-to-text service converts audio to text and discards everything that was not words. Sending that text to a model means the model reasons over a lossy representation. Native audio input skips the conversion, and the difference shows up in the questions that become answerable.

You can ask about things a transcript does not contain: whether a speaker sounded hesitant before agreeing, whether there was applause, whether the recording has background noise that would explain a misheard word, how many distinct voices appear. You can ask for a summary directly, without a transcription step that introduces its own errors and then has them reasoned over as though they were facts. And you can point at moments—“what is said around 14:20” —because timestamps are available to the model.

A prompt that exploits this looks different from a transcription request:

For each agenda item discussed in this recording, give:
  - the item
  - the decision, or "no decision" if it was deferred
  - the approximate timestamp in MM:SS
  - whether any participant expressed reservations, and what about

Do not transcribe. If something is inaudible, say so rather than guessing.

The last line matters. A model given audio it cannot make out will produce a plausible reconstruction unless told not to, and a confidently wrong quotation is worse than a gap.

Splitting a long recording

The nine-and-a-half-hour ceiling sounds generous until you notice that an hour of audio is already around 115,000 tokens and that every follow-up question re-sends all of it. Long recordings usually want splitting for cost reasons long before they hit any limit.

Two approaches, with different failure modes.

1. Cache once, ask many times. If the recording is fixed and the questions vary, put it behind an explicit context cache and reference it. An hour of audio comfortably clears any documented minimum, and the per-question cost drops to the cached input rate. This is the right shape for “here is the board meeting, now interrogate it”.
2. Chunk with overlap, then combine. If the recording is too long for the window or you want parallelism, split it into segments with a minute or two of overlap at each boundary, process each independently, and merge. The overlap exists because a decision discussed across a boundary is otherwise present in neither segment in full.

Chunking has a specific cost that is worth naming rather than discovering: each segment is analysed without the others, so anything requiring the whole recording—who spoke most, whether a point raised at minute 12 was ever resolved, how the tone changed— cannot be answered from any single segment and has to be reconstructed in a second pass over the segment summaries. That second pass is cheap, because summaries are text, but it has to exist.

Timestamps need adjusting too. A model given the second segment reports positions relative to that segment’s start, not to the original recording, so store each segment’s offset and add it back. Asking the model to use absolute times because you mentioned the offset in the prompt works unreliably; doing the addition in your own code works every time.

Where it stops

• It is not a diarisation system. The model can often distinguish speakers and will label them, but it has no speaker registry and no acoustic identity model. “Speaker 1” in one response and in another may not be the same person. If you need reliable attribution, that is a different tool.
• It is not verbatim. Asking for a full transcript works, and long recordings will paraphrase, compress or skip. Timings are approximate. For a legal or medical transcript, use a transcription service and use the model on its output.
• Silence is billed. Thirty minutes of a two-hour recording being dead air costs 57,600 tokens of nothing. Trim first.
• This is not the Live API. Everything on this page is audio as an input part in a request-response call. Real-time bidirectional audio is a separate interface with a session model of its own—see the Live API session.