Music Genre Classification: How a Model Hears Genre
9 min read · updated August 11, 2026
A genre classifier does not hear genre. It measures a few dozen numbers about the distribution of energy in time and frequency, and a learned function maps those numbers onto words that were invented by record shops. Both halves of that sentence are worth understanding, because the second half is what limits the first.
From waveform to time and frequency
Everything starts with a short-time Fourier transform. The signal is cut into overlapping windows, each window is multiplied by a taper (a Hann window, usually, to stop the edges of the frame creating false high-frequency energy), and each windowed chunk is transformed into a spectrum. Two parameters decide what survives. The window length sets the frequency resolution; the hop length sets how often you get a frame.
Take librosa’s documented defaults, since most published feature code uses them: sr=22050, n_fft=2048, hop_length=512, a Hann window, power=2.0. That gives a frequency bin every 22050 / 2048 ≈ 10.8 Hz and a new frame every 512 / 22050 ≈ 23.2 ms, so roughly 43 frames per second. Those two numbers are a trade, not a default you can improve on for free: lengthening the window to separate two close partials smears anything percussive across more time, and shortening it to catch a hi-hat transient blurs the low end where the bass notes are.
The linear spectrum is then usually re-binned onto the mel scale, which is approximately linear below 1 kHz and logarithmic above it. librosa’s mel filter bank defaults to 128 bands. This is a lossy projection chosen because it matches human pitch perception, and that choice is doing real work in a genre task — genre distinctions live in the region humans attend to, so throwing resolution at 15 kHz buys almost nothing.
Three feature families
Classical genre features fall into three groups, and each answers a different question about the same spectrogram.
- Timbre — what it sounds like. Mel-frequency cepstral coefficients are the standard summary: take the log of the mel spectrum, apply a discrete cosine transform, and keep the first 13 or 20 coefficients. The DCT decorrelates the bands and the truncation keeps the broad shape of the spectral envelope while discarding the fine harmonic structure. That is exactly the distinction between “a distorted guitar” and “a distorted guitar playing an A”. Spectral centroid (the energy-weighted mean frequency, heard as brightness), rolloff, and bandwidth sit in the same family.
- Rhythm — how it moves. Frame-to-frame increases in spectral energy give an onset strength envelope, sampled here at about 43 Hz. Its autocorrelation peaks at the beat period, and a global tempo estimate falls out of that. Genre is strongly tempo-conditioned in places (drum and bass sits near 170 BPM by convention, not by accident) and almost unconditioned in others.
- Harmony — what notes are present. A chroma vector folds every octave onto twelve pitch classes, so C2 and C5 land in the same bin. It is deliberately blind to register and timbre, which makes it a good descriptor of chord content and a useless one for telling a piano from a synthesiser.
A worked feature vector
A per-clip vector is built by computing these frame-wise and then summarising over time, because a classifier wants one fixed-length input per clip and a 30-second clip has about 1,290 frames. Mean and standard deviation over the clip is the usual summary — the standard deviation is not filler, it is what separates a track with constant timbre from one that changes texture every eight bars.
# Per-clip feature vector, 30 s at sr=22050, hop_length=512 # -> 22050/512 ~= 43 frames/s, ~1290 frames per clip 20 MFCC means 20 20 MFCC standard deviations 20 12 chroma means 12 spectral centroid mean + sd 2 spectral rolloff mean + sd 2 spectral bandwidth mean + sd 2 zero-crossing rate mean + sd 2 RMS energy mean + sd 2 global tempo (BPM) 1 ------------------------------------------------ -- total 63
Sixty-three numbers, from about 1.3 million samples. A gradient-boosted tree or a small MLP over that vector is a real baseline and trains in seconds. A convolutional network run directly on the mel spectrogram usually beats it, because it can learn features this list does not contain — but it is learning from the same transform, so the window and hop choices above still bound what it can see.
The label is the weak part
Everything above is signal processing and is as solid as the arithmetic. The target is not. Genre is a commercial and social category that changed over time, differs by country, and is applied inconsistently by the people who apply it. No amount of feature engineering fixes a target that two annotators disagree about.
The canonical demonstration is the dataset most genre papers were evaluated on. Bob Sturm’s 2013 paper “The GTZAN dataset: Its contents, its faults, their effects on evaluation, and its future use” catalogues repetitions, mislabellings and distortions in a corpus he notes appears in at least 100 published works, and shows that different systems are affected by those faults to different degrees. The practical consequence is that a reported accuracy on that corpus is not comparable across papers in the way it was assumed to be.
Where it breaks
- The album effect. Tracks from one album share production, mastering and instrumentation. Split a dataset randomly and the model can identify the album from its timbre fingerprint and read the genre off that. Accuracy looks excellent and collapses on a new artist. Split by artist, always.
- Mastering, not music. Loudness, compression and spectral tilt are era markers. A model can learn “this was mastered in 2015” and get a lot of genre labels right for free, which is a shortcut that does not survive a remaster.
- Single-label training on multi-genre music. A track that is honestly both is forced to be one, and the softmax makes the two labels compete. This is the same structural problem covered in multi-label audio tagging and in instrument recognition from polyphonic audio, and the fix is the same: sigmoid outputs and per-class thresholds.
- Tempo is not as informative as it looks. Octave errors mean a reported 85 BPM and 170 BPM are frequently the same track, which puts a genre-discriminative feature on the wrong side of a split. That failure is worth understanding on its own; see beat tracking and tempo extraction.