Music Key and Chord Detection From Audio

Key detection and chord detection are the same computation with different time scales on top of it. Both start by throwing away octave information, and almost everything they get wrong follows from what survives that step.

Folding a spectrum into twelve bins

Western tonal music is organised by pitch class: a C is a C whether it is played by a bass or a piccolo. A chromagram encodes exactly that. Take a time-frequency representation, map each frequency bin to a MIDI note number, take that number modulo 12, and sum the energy into whichever of the twelve pitch classes it landed in. The result is a 12-dimensional vector per frame, normalised, and that is a chroma feature.

The transform underneath matters. An STFT has linearly spaced bins, which is a bad match for pitch, because pitch is logarithmic: the semitone from A0 to A#0 is 1.6 Hz wide while the one from A6 to A#6 is 104 Hz wide. A single STFT resolution is therefore either too coarse in the bass or wastefully fine in the treble. The constant-Q transform fixes this by using logarithmically spaced bins with a constant quality factor, so every semitone gets the same number of bins across the whole range — three bins per semitone is a common choice, giving 36 bins per octave, which are then folded down to 12.

import librosa

y, sr = librosa.load("track.wav", sr=22050, mono=True)

# harmonic-percussive separation first: drums add broadband
# energy to every pitch class equally, which is pure noise here
y_harm, _ = librosa.effects.hpss(y)

chroma = librosa.feature.chroma_cqt(
    y=y_harm, sr=sr,
    bins_per_octave=36,   # 3 bins per semitone
    n_chroma=12,
    hop_length=2048,
)
# chroma.shape == (12, n_frames), rows ordered C, C#, D, ... B

The harmonic-percussive separation on the second line is not optional in practice. A kick drum or a snare deposits energy across the whole spectrum, which after folding is a roughly equal contribution to all twelve pitch classes — a DC offset on every frame that dilutes the actual harmony. Removing the percussive component first is usually the largest single accuracy improvement available for either task.

A worked chroma vector, and the contamination in it

Take a single note: C4 played on a piano, fundamental at 261.63 Hz. A naive expectation is that the chroma vector has all its energy in the C bin. It does not, and the reason is the harmonic series. Each partial of a real instrument sits at an integer multiple of the fundamental, and those multiples land on other pitch classes:

  partial   frequency    nearest note   pitch class   deviation
  1         261.63 Hz    C4             C             exact
  2         523.25 Hz    C5             C             exact
  3         784.88 Hz    G5             G             +2 cents
  4       1,046.50 Hz    C6             C             exact
  5       1,308.13 Hz    E6             E            -14 cents
  6       1,569.77 Hz    G6             G             +2 cents
  7       1,831.41 Hz    Bb6            A#           -31 cents

A single C note therefore deposits energy in C, G, E and, weakly and badly out of tune, A#. That is a C major triad plus a seventh, produced by one note. So the chroma vector for a real C major triad — three notes, each contributing its own series — looks something like this after normalising to a maximum of 1:

  C     C#    D     D#    E     F     F#    G     G#    A     A#    B
  0.95  0.03  0.06  0.04  0.61  0.05  0.04  0.72  0.05  0.07  0.04  0.06

  the three played notes  C, E, G  dominate, as expected
  but E and G are inflated by C's 5th and 3rd partials,
  and A# carries measurable energy from nobody's fundamental

Three consequences follow directly. Perfect fifths are systematically over-weighted, because the third partial of every note reinforces the pitch class a fifth above it — which is the mechanism behind the classic fifth-related errors in key estimation. Major thirds are over-weighted for the same reason via the fifth partial, biasing chord estimation toward major over minor. And the seventh partial lands about a third of a semitone flat, so it smears across two bins rather than reinforcing one, adding a floor of noise.

The standard mitigations are logarithmic compression of the magnitudes before folding, which reduces the dominance of loud low partials; spectral whitening; and explicit harmonic suppression, where a fraction of each bin’s energy is subtracted from the bins its partials would fall into. None of them removes the problem, because the partials are the signal — a note without them is not the note.

Key estimation by profile correlation

Global key estimation is short once the chromagram exists. Average chroma over the whole piece — or over a segment, more on that below — to get one 12-vector. Then correlate it against 24 reference profiles: a major profile and a minor profile, each rotated to all twelve tonics. The highest correlation is the estimated key.

The reference profiles are the interesting part. The classical ones come from Krumhansl and Kessler’s probe-tone experiments, in which listeners rated how well each of the twelve pitch classes fitted a preceding tonal context; the resulting rating vector is used directly as the expected pitch-class distribution for a key. Later profiles by Temperley and others adjust these to fit corpus statistics of actual music rather than listener judgements, and they behave differently on different repertoires. The choice of profile is a real parameter, not a detail.

Three failure modes are structural rather than fixable by better features:

• Relative major and minor share every pitch class. C major and A minor contain exactly the same seven notes. Only the weighting distinguishes them, and if a piece in A minor spends its time on the notes C major favours, no pitch-class histogram can tell them apart. Resolving it requires knowing where the piece rests, which is temporal information the average discarded.
• The dominant is one accidental away. C major and G major differ in a single pitch class, so a passage that avoids F and F# is ambiguous between them by construction.
• Modulation makes the global average meaningless. A piece that spends half its time in each of two keys averages to a chroma vector describing neither. Anything longer than a few minutes should be segmented and keys estimated per segment.

Because the first two confusions are musically close rather than arbitrary, evaluations of key detection conventionally award partial credit for a relative, parallel or fifth-related answer rather than scoring it as flatly wrong. When comparing published key-detection numbers, check whether the figure is a strict accuracy or a weighted score, because they are not comparable.

Chords need time as well as pitch

Chord recognition starts the same way and then adds structure. The template approach compares each frame’s chroma against 24 binary templates — major and minor triads on each of twelve roots — by cosine similarity, and takes the best match. On its own this produces output that flickers wildly between frames, because a passing note or a momentary imbalance in the partials flips the argmax.

Two pieces of temporal structure fix it. The first is beat-synchronous averaging: estimate beat positions, average chroma within each beat, and classify per beat rather than per frame. This aligns the analysis with the rate at which harmony actually changes and averages away passing notes. The second is a hidden Markov model over the chord sequence, with emission probabilities from the template similarities and a transition matrix that strongly favours staying on the current chord. Viterbi decoding then finds the most likely chord sequence rather than the most likely chord at each instant, and the self-transition bias is the single parameter controlling how much the output is smoothed.

Learned models replace the templates and often the decoder too, but the structure survives: a frame-level acoustic model producing chord posteriors, followed by a decoder imposing temporal continuity. The decoder is not a post-processing convenience, it is where roughly half the accuracy comes from.

What chroma throws away

Folding to twelve bins discards octave, and octave carries two things that matter.

The first is the bass note, and therefore inversion. C-E-G and E-G-C have identical chroma vectors, so a chromagram cannot distinguish a root-position C major from its first inversion. That distinction is musically significant and completely invisible. The standard remedy is a second, separate chromagram computed over a low frequency range only — a bass chromagram — used as an additional feature, which restores enough of the information to identify the bass pitch class.

The second is voicing and register, which carry instrument and texture information. That loss is usually deliberate: it is exactly what makes chroma robust to timbre, so the same chord played by a guitar or a string section produces similar vectors. If you want the instrument instead, chroma is the wrong feature and timbral features are the right one.

One property is worth exploiting rather than lamenting. Transposition is a cyclic rotation of the chroma vector — shifting a piece up a semitone rotates every frame by one bin. So transposition-based data augmentation is a rotation of the feature matrix and a corresponding rotation of the labels, costing nothing and multiplying a chord dataset by twelve. That is a much larger effective gain than most architectural changes, and it is available for anything built on chroma, including mood and tempo models that use harmony as an input feature.