Audio Embedding Models Like CLAP, Explained

CLAP is the audio analogue of CLIP: two encoders trained so that a clip and its description land near each other in one vector space. The consequence is that a text query can search a sound library that has no labels, and the interesting part is how badly and how well that generalises.

The training objective, precisely

Take a batch of N audio-text pairs. Encode each clip with an audio encoder and each caption with a text encoder, project both into a shared space of the same dimensionality, and L2-normalise. Compute the N × N matrix of cosine similarities between every clip and every caption, scale it by a learned temperature, and apply cross-entropy along both axes so that the diagonal — the true pairs — is the correct answer for both “which caption goes with this audio” and “which audio goes with this caption”. That symmetric InfoNCE loss is the whole training signal.

Two things follow immediately. Every non-diagonal entry is treated as a negative, so the batch size is a hyperparameter of the objective rather than of the optimiser — a larger batch is a harder and more informative contrastive problem. And the model never learns what a sound is; it learns which of the captions in front of it fits best. Absolute similarity values therefore carry very little meaning, while rankings carry a lot. This is the single most useful fact about deploying one.

What it was trained on

There are two distinct lines of work under the same name, and confusing them causes real trouble when someone quotes a number.

Microsoft Research published “CLAP: Learning Audio Concepts From Natural Language Supervision” (Elizalde, Deshmukh, Al Ismail and Wang, June 2022), trained on 128,000 audio-text pairs and evaluated on 16 downstream tasks across 8 domains.

LAION published “Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation” (Wu, Chen, Zhang, Hui, Nezhurina, Berg-Kirkpatrick and Dubnov, November 2022), which released LAION-Audio-630K: 633,526 audio-text pairs collected from several sources. Its later checkpoints add AudioSet and music and speech corpora on top, which the project’s repository describes as roughly 4 million samples in total. When someone says “CLAP” without qualification in an open-source context, this is usually the one they mean.

The provenance detail that matters most is in the LAION paper’s own title: keyword-to-caption augmentation. AudioSet, published by Google’s Machine Perception group in 2017, is 2,084,320 human-labelled ten-second YouTube clips over an ontology of 632 audio event classes — but they are labels, not sentences. To use them for contrastive training the labels are turned into captions programmatically. So a large fraction of the text side of this model’s training was machine-generated from a fixed tag vocabulary.

That explains a behaviour people find puzzling. The text tower is far better at ontology-shaped phrases than at free description. “A dog barking” works well because it is close to the caption template built from a tag. “A large dog barking twice, then a car door” works much less well, because nothing in the training text distribution described audio that way.

The two towers and the length problem

On the audio side, the LAION work compares PANN-style convolutional encoders against HTSAT, a transformer operating on a mel spectrogram, and the released checkpoints are built on HTSAT. The repository specifies that audio must be loaded at a 48 kHz sample rate. The text side is a standard transformer encoder; the paper compares several candidates rather than asserting one, and the practical consequence is only that the text tower inherits the tokeniser and context limits of whatever it was built from.

Length is the awkward part. The encoder is trained on a fixed window — ten seconds is the usual figure for this family — while real audio is any length at all. Truncating loses everything after the window; padding a two-second clip out to ten wastes most of the input; and averaging embeddings over successive windows blurs a clip containing two different sounds into a point that resembles neither.

The feature-fusion mechanism in the LAION paper is the response: rather than one crop, it combines a global downsampled view of the whole clip with several local crops, so a long recording produces one embedding that has seen all of it at coarse resolution and parts of it at full resolution. The checkpoints come in fusion and non-fusion variants for exactly this reason, and picking the wrong one is a common cause of disappointing retrieval on long files.

Zero-shot classification is retrieval in disguise

There is no classifier head. To classify a clip into a set of categories, you write one text prompt per category, embed all of them, embed the audio, and take the highest cosine similarity. Classification and search are the same operation with different inputs.

import numpy as np
import laion_clap

model = laion_clap.CLAP_Module(enable_fusion=False)
model.load_ckpt()  # downloads the default checkpoint

labels = [
    "the sound of a dog barking",
    "the sound of a car engine idling",
    "the sound of glass breaking",
]

# audio must be at 48 kHz, shape (batch, samples), float32
audio_emb = model.get_audio_embedding_from_filelist(
    x=["clip.wav"], use_tensor=False
)
text_emb = model.get_text_embedding(labels, use_tensor=False)

# both are L2-normalised, so a dot product is cosine similarity
scores = audio_emb @ text_emb.T
print(labels[int(np.argmax(scores[0]))], float(np.max(scores[0])))

Prompt wording is not cosmetic here. “Dog” and “the sound of a dog barking” are different points in the text space, and the second is closer to the caption distribution the model was trained on. Keeping prompt templates consistent across all categories matters more than making any single one perfect, because the comparison is between them.

The scores themselves are not probabilities. Applying a softmax over the similarity row produces something that looks like a distribution and is not calibrated, and it will look confident on a clip containing none of your categories, because the argmax over three options is still one of the three. Rejecting out-of-set audio needs a similarity threshold established against a held-out set of negatives, or an explicit “background noise” prompt competing for the argmax.

Where the shared space stops working

• Counting and order. The objective rewards matching a clip to a caption, and captions rarely specify counts or sequence. “Three knocks” and “knocking” are near neighbours; “a door closing then footsteps” and “footsteps then a door closing” are nearly the same point.
• Overlap. One embedding per window means a clip with a car, a dog and speech gets one vector that is a compromise. Detection of concurrent events needs shorter windows and per-window scoring, not a better prompt.
• Fine-grained music and speech. The general checkpoints are trained mostly on web video audio. Chord identity, key, instrument technique and speaker identity are largely below its resolution — the music-specific checkpoints exist for a reason, and for key or chords a purpose-built chromagram pipeline beats a general embedding by a wide margin.
• Domain shift in recording conditions. Training audio is web video: compressed, mixed, often with music underneath. A clean far-field recording from an industrial sensor is out of distribution, and the embedding degrades in ways the similarity score does not report.

Used within those bounds it is an unusually good default. One fixed-dimension vector per clip is directly usable for nearest-neighbour search over a sound library, for clustering, for deduplication, and as a frozen feature extractor under a small supervised head — which, with a few hundred labelled examples, routinely beats zero-shot prompting on a narrow task and costs almost nothing to train.