Music and Sound Generation Models

Three minutes of CD-quality stereo audio is about sixteen million numbers, arranged in a sequence where an error lasting one thousandth of a second is audible. That is the problem every approach on this page is trying to make smaller, and which representation it picks decides everything else about it.

The short answer

Audio generation works by not generating audio. Every practical system maps the waveform into a much smaller representation, generates in that representation, and maps back with a decoder. The three representations in use are time-frequency images, discrete tokens from a learned codec, and continuous latents — and the third is the same trick as latent diffusion for images, applied to a one-dimensional signal.

Why audio was harder, in numbers

Raw audio, CD quality:
    44,100 samples per second per channel × 2 channels = 88,200 values/s

    30 seconds  =  2,646,000 values
    3 minutes   = 15,876,000 values

Compare a 1024×1024 RGB image:
    1024 × 1024 × 3 = 3,145,728 values

So three minutes of audio is about 5× the raw values of a megapixel
image — and, unlike an image, it is a single sequence in which order is
the content.

The raw count is only half of it. Three further properties make audio less forgiving than a picture of the same size:

• Errors are audible at a much finer scale. A wrong pixel is invisible. A discontinuity lasting a millisecond is a click that everyone hears, and periodic artefacts at any level become a tone.
• Phase matters and is hard to represent. Two signals with identical frequency content and different phase can sound completely different. Most of the intuitive representations throw phase away.
• Structure spans the whole sequence. A chorus must return ninety seconds later in the same key at the same tempo. That is a dependency across millions of samples, which is far longer than any dependency an image model has to maintain.

Spectrograms, and the phase problem

The oldest workable approach converts audio into a time-frequency image and reuses image machinery on it. A short-time Fourier transform produces a 2D array of magnitudes: time along one axis, frequency along the other. A mel scale compresses the frequency axis to match human hearing.

30 seconds of audio as a mel spectrogram:

    hop length 256 samples at 44.1 kHz  →  172 frames per second
    30 s × 172                          →  5,166 time steps
    80 mel bands

    5,166 × 80 = 413,280 values

    Down from 2,646,000 raw values: a 6.4× reduction, and now it is a
    2D array that a convolutional or transformer image model can eat.

The catch is in what the transform discarded. A magnitude spectrogram keeps how much energy is at each frequency and throws away the phase. Reconstructing a waveform therefore requires inventing phase, and the classical iterative algorithms for doing so produce the characteristic metallic, watery quality that early neural audio was known for.

The fix was neural vocoders: networks trained to produce a waveform from a spectrogram directly, learning plausible phase rather than estimating it. That works well and it also means the vocoder becomes the quality ceiling for the whole system, in exactly the way the autoencoder is the ceiling for a latent image model.

Neural codecs turn audio into tokens

The change that made audio look like a language modelling problem was the neural audio codec: an autoencoder whose bottleneck is discrete. The encoder produces a vector per frame, and residual vector quantisation snaps it to entries in a stack of learned codebooks — each codebook correcting the error left by the one before.

A codec with a 50 Hz frame rate and 8 codebooks of 1,024 entries:

    tokens per second = 50 frames × 8 codebooks = 400 tokens/s
    bits per second   = 400 × log2(1024) = 400 × 10 = 4,000 bits/s = 4 kbps

    30 seconds  =    12,000 tokens
    3 minutes   =    72,000 tokens

Versus 2,646,000 raw values for the same 30 seconds: a 220× reduction,
into a form a transformer can model autoregressively.

At 12,000 tokens for thirty seconds, generation becomes next-token prediction over a vocabulary of 1,024 — structurally the same problem as text. The residual structure creates its own difficulty: the first codebook carries the coarse content and later ones carry fine detail, so the eight tokens for one frame are not interchangeable and the model has to be arranged to predict them in a sensible order. Different systems interleave them differently, and that ordering is one of the main design decisions in this family.

The trade against the alternatives is the usual one for discrete representations: a hard ceiling set by the codec’s bitrate, against the ability to use every technique developed for autoregressive sequence models, including conditioning on text in exactly the way a language model does.

Continuous latents and diffusion

The third approach skips quantisation. Train a continuous autoencoder that compresses the waveform along the time axis, and run a diffusion or flow model in that latent space — the identical structure to image latent diffusion, with one spatial axis instead of two.

A continuous audio autoencoder at 1,000× temporal compression:

    44,100 samples/s  →  ~44 latent frames/s, each a vector of width 64

    30 seconds  =  1,320 latent positions × 64 channels
    3 minutes   =  7,920 latent positions × 64 channels

Attention over 7,920 positions is entirely ordinary for a transformer,
which is why this is the direction long-form generation went.

The advantages mirror the image case: the whole sequence is generated jointly rather than left to right, so global structure is available to the model at every step; there is no quantisation ceiling; and generation time is set by step count rather than by output length. The disadvantage also mirrors it: the autoencoder’s reconstruction quality is a floor nothing downstream can beat, and heavier temporal compression trades transient sharpness — drum hits, consonants, string attacks — for cheaper generation.

The long-range structure problem

Even with a good representation, musical form is a distinct difficulty and it is worth separating from audio quality.

• Repetition must be exact enough and varied enough. A returning chorus is not a copy and not a new section. Nothing in a next-token or denoising objective represents “this is the same section as before, transposed”.
• Tempo and key are global constraints on local decisions. Like spelling in an image, they are properties that no local sample decision is penalised for violating, only their accumulation.
• Long context is expensive in exactly the way video is. Attention over a full track is quadratic in its length, so systems either use a heavily compressed representation, a windowed attention pattern, or a hierarchy that generates structure first and detail second.
• Evaluation has no quick glance. Judging a three-minute output requires listening to three minutes, which makes iteration slow and automated metrics unusually load-bearing — and the automated metrics for musical quality are not well established.

Different problems wearing the same name

“AI audio generation” covers several tasks with very different difficulty profiles, and conflating them is the most common error in discussion of the field.

The pipeline reasoning transfers to speech as well; the latency and cost side of that is in the text-to-speech guide, and native audio input models are covered in speech-to-text versus native audio.