Classifying Music Mood and Tempo Automatically

Tempo extraction and mood classification get bundled together in product requirements because both end up as a field on a track. They are entirely different problems: one has a correct answer that is hard to compute, the other has no single correct answer at all.

The onset strength envelope

Beat tracking does not work on the waveform. It works on a one-dimensional signal that measures, for each frame, how much the spectrum has just brightened — the onset strength envelope. Compute a mel spectrogram, take the difference between consecutive frames, half-wave rectify it so only increases count, and sum across frequency. Rectification is the essential step: a note ending is a decrease in energy and is not an onset, so negative differences are discarded rather than taken in absolute value.

The frame rate of this envelope is set by the hop length. With librosa’s documented defaults of sr=22050 and hop_length=512, that is 22050 / 512 = 43.07 frames per second, or one sample every 23.2 ms. Everything downstream is quantised to that grid, which puts a floor on the achievable timing precision: a beat position can be wrong by up to about 12 ms from quantisation alone before any algorithmic error. For most applications that is irrelevant, and for anything that must align audio to a click track it is not.

From envelope to a tempo estimate

The envelope is periodic when the music is metrical, so its autocorrelation peaks at the beat period and its multiples. Weighting that autocorrelation by a prior over plausible tempi turns it into a tempogram, and the peak of the weighted function is the tempo estimate. librosa’s beat_track documentation gives the defaults — start_bpm=120.0, tightness=100, trim=True, units='frames' — and cites Ellis’s 2007 Journal of New Music Research paper “Beat tracking by dynamic programming” for the method, described there as three stages: measure onset strength, estimate tempo from onset correlation, then pick peaks in the onset strength approximately consistent with the estimated tempo.

The third stage is the part worth understanding. Given a tempo, beat positions are chosen by dynamic programming to jointly maximise onset strength at the chosen positions and regularity of the spacing between them, with tightness controlling the weight on regularity. High tightness produces a metronomic grid that ignores real tempo drift; low tightness follows the performance and can be dragged off by a syncopated passage. The default of 100 is a compromise tuned for popular music with a steady pulse, and it is the first parameter to change when tracking a live or classical performance.

Two properties of start_bpm catch people out. It is a prior, not an initialisation to be refined away — the tempo prior is a log-normal weighting centred on it, so leaving it at 120 while analysing a corpus of drum and bass systematically biases estimates downward. And the tempo it reports is a single global value, so any track with a tempo change gets one number that is wrong for both sections. Use a tempogram and look for a moving peak if that is plausible in your material.

Octave errors are the normal case

The autocorrelation of a beat-periodic signal peaks at the beat period and at every integer multiple and sub-multiple of it. Nothing in the signal distinguishes “85 BPM with a backbeat” from “170 BPM”; both descriptions are consistent with the same onset pattern, and which one a human calls the tempo is a matter of convention that varies by genre and, for the same track, by listener.

This is why tempo evaluation in the literature reports two numbers. Accuracy 1 counts an estimate correct if it is within a tolerance (usually 4%) of the ground truth; Accuracy 2 also accepts the estimate multiplied or divided by 2 or 3. The gap between them is the octave-error rate, and it is large enough that reporting only Accuracy 1 makes a good tracker look broken.

• If tempo feeds a downstream feature, fold the ambiguity in. Mapping to a canonical range by doubling or halving into, say, 70–140 BPM removes the error at the cost of genuinely fast and genuinely slow music being misrepresented.
• If tempo is shown to a user, show your confidence.The ratio of the second autocorrelation peak to the first is a usable proxy: when it is near 1, the estimate is a coin flip between two octaves.
• If tempo drives beat-synchronised playback, the octave matters less than the phase. A grid at double the intended tempo still lands on every real beat; a grid offset by half a beat is wrong everywhere.

Mood as two dimensions, not a word

Mood models almost universally sit on the circumplex model of affect from James Russell’s 1980 paper in the Journal of Personality and Social Psychology, which places affective states on two orthogonal axes: valence, running from negative to positive, and arousal, running from calm to energetic. The four quadrants recover the words people actually use — high arousal with high valence is excited or joyful, high arousal with low valence is angry or anxious, low arousal with high valence is content or serene, low arousal with low valence is sad or depressed.

Predicting two continuous values rather than one of eight labels changes the modelling in three useful ways. The loss becomes a regression loss, so a prediction of “slightly sad” against a ground truth of “very sad” is penalised less than one of “joyful”, which a cross-entropy over labels treats as equally wrong. Disagreeing annotators can be averaged into a point with a variance rather than resolved by majority vote. And you can query the space continuously, which is what a playlist that needs to ramp arousal over forty minutes actually requires.

The acoustic correlates split cleanly along the two axes, which is a large part of why the decomposition is used. Arousal is well predicted by things you can measure directly — tempo, RMS energy, spectral centroid, onset density, dynamic range. Valence is much harder and leans on harmony: mode (major against minor) is the classic correlate, along with harmonic consonance and, in vocal music, lyrics that the audio model does not have. The harmonic side of this depends on key and chord estimation, covered in key and chord detection.

The agreement ceiling

Mood is a perception, and perceptions differ. If two annotators listening to the same track place its valence 0.3 apart on a −1 to 1 scale, no model can be reliably closer to “the truth” than that, because the truth is a distribution. The honest way to report a mood model is against inter-annotator agreement: compute the correlation between two independent annotator groups on held-out tracks, then report your model’s correlation against the consensus on the same scale. A model at 0.6 where humans agree at 0.65 is close to done; the same model where humans agree at 0.9 has real room left.