Instrument Recognition From a Polyphonic Recording

A classifier that identifies a solo instrument from a clean recording is a first-week exercise. The same classifier on a four-piece band fails in a specific, diagnosable way, and the failure is half in the loss function and half in the physics of overlapping harmonics.

What the softmax assumes

A softmax over instrument classes exponentiates the logits and normalises them to sum to one. That normalisation is a statement: exactly one class is present. Training under it does not merely fail to represent two simultaneous instruments; it actively pushes their representations apart, because raising the score of guitar necessarily lowers the score of piano. Gradient descent on a mixture labelled “guitar” teaches the network that piano evidence in that mixture is evidence against the correct answer.

The visible symptoms are consistent. The model outputs one confident class for a mixture, usually the loudest or the most spectrally distinctive instrument. Its confidence is high in a way that calibration cannot fix, because the model is answering a well-posed question correctly — which single class best explains this? — and the question is not yours. And it becomes unstable across adjacent frames, flipping between two instruments as their relative levels change, because at any instant one must win.

Overlap in the spectrogram

The second half of the problem is that the mixture is genuinely ambiguous in the representation. Sound adds linearly in the waveform, and therefore in the complex spectrum; magnitudes add only approximately, and phase relationships mean two partials at the same frequency can partially cancel. What the model sees is one plane of energy with no separation.

Harmonic instruments make this worse than random overlap would. A harmonic series is a fundamental plus integer multiples, so instruments playing notes in simple ratios — which is what harmony is — have partials that coincide exactly:

Two notes a perfect fifth apart, A2 = 110 Hz and E3 = 165 Hz
(a 3:2 ratio). Partials, in Hz:

  A2 (110):  110  220  330  440  550  660  770  880  990
  E3 (165):  165  330  495  660  825  990

  coinciding: 330, 660, 990, ...  (every 3rd partial of A2
                                   meets every 2nd of E3)

An octave (110 and 220) is worse still: every partial of the
upper note lands exactly on a partial of the lower one.

At those coinciding frequencies there is no information whatsoever about how the energy divides between the two sources. A model identifying an instrument by its spectral envelope — the relative strength of its partials, which is what timbre largely is — has that envelope corrupted precisely where the harmony is most consonant. This is why instrument recognition is hardest on the material that sounds most musically coherent, and easiest on dissonant or percussive mixtures where the sources occupy different bins.

Two things do survive the collision and are what the network learns to use. Onset transients are broadband and brief, and the attack of a plucked string looks nothing like the attack of a bowed one even when their sustained partials overlap. And the non-coinciding partials — 220, 440, 550 in the example above — still carry a clean envelope for one source. A network with enough frequency-axis context can integrate over the partials that are not contaminated.

What multi-label training changes

Replacing the softmax with one sigmoid per instrument, trained with binary cross-entropy summed over classes, changes three things concretely.

• The gradients decouple. Evidence for piano no longer suppresses guitar. Each output head learns its own decision boundary against “everything else”, and a mixture containing both can drive both toward 1.
• The training data can be honest. A clip labelled with three instruments is now representable. Under a softmax you had to either pick one, which teaches a falsehood, or exclude the clip, which throws away the polyphonic material that is the whole problem.
• You inherit the threshold problem. There is no argmax any more. Each class needs an operating point chosen on validation data, and rare instruments need different ones from common ones. This is the same structure as general audio tagging; see multi-label audio tagging.

Partial labels and masked loss

Multi-label training exposes a data problem that single-label training hid. To train a sigmoid for “clarinet” you need clips labelled as definitely containing a clarinet and clips labelled as definitely not. Annotators do not produce the second kind at scale: nobody listens to 20,000 clips and confirms the absence of each of twenty instruments.

The reference dataset for this is OpenMIC-2018 from Humphrey, Durand and McFee, published at ISMIR 2018 and distributed on Zenodo: 20,000 ten-second Creative Commons clips from the Free Music Archive, partially labelled for the presence or absence of 20 instrument classes by crowd annotators. Partially is the operative word — for any given clip, most of the twenty classes have no annotation at all. By comparison IRMAS, the older benchmark, holds 6,705 examples over 11 classes at three-second scale.

The correct handling is a masked loss: compute binary cross-entropy only over the classes that were annotated for that clip, and multiply the rest by zero so they contribute no gradient in either direction.

Per clip: y in {0,1}^20 (labels), m in {0,1}^20 (1 = annotated)

  loss = - sum_c m_c * [ y_c * log(p_c) + (1 - y_c) * log(1 - p_c) ]
         / max(sum_c m_c, 1)

Worked, one clip with 3 of 20 classes annotated:
  annotated: guitar (y=1, p=0.82), drums (y=1, p=0.61),
             piano  (y=0, p=0.15)
  the other 17 classes: m = 0, no contribution

  -[ log(0.82) + log(0.61) + log(1 - 0.15) ] / 3
  = -[ -0.198 + -0.494 + -0.163 ] / 3
  = 0.855 / 3 = 0.285

The alternative — treating every unannotated class as a negative — is the mistake to avoid, and it is seductive because it makes the code simpler and the loss curve look fine. It trains the model that a clarinet is absent from thousands of clips nobody checked, which suppresses exactly the rare classes you have least data for. Positive–negative imbalance in OpenMIC is severe for several classes even before you introduce a false one.

A related trap is the sampling of the negatives you do have. Because annotators were asked about a class only when something prompted the question, the confirmed negatives are not a random sample of absences — they are skewed toward clips where the class was plausible enough to be asked about. A clarinet head therefore learns to discriminate clarinet from other woodwinds far better than from silence, which is usually what you want and is worth knowing when the model behaves oddly on material unlike anything in the corpus.

Evaluating without being fooled

• Per-class average precision, never overall accuracy. With twenty sigmoids and heavy imbalance, predicting all-negative gives a high accuracy and a useless model. Report AP per class and look at the worst ones.
• Evaluate only on annotated entries. The mask applies at evaluation time too. Scoring a prediction against an unannotated class is scoring against a guess.
• Stratify by polyphony. Report separately on clips with one, two and four or more instruments. A single aggregate number lets good performance on solo material hide the failure you actually care about, and the whole point of the multi-label change was the dense clips.
• Split by track and by artist. Ten-second clips drawn from the same recording share instrumentation, room and mastering. A random split leaks all three across train and test, and the resulting score is measuring memorisation. The same trap in genre work is described in music genre classification.