Sound Event Detection Beyond Speech

Machine noise, breaking glass, alarms, birdsong, a cough: audio classification on everything that is not speech. The modelling is the easy half. The hard half is that a monitor running continuously makes tens of thousands of decisions a day, and its false-positive rate has to be read in that light.

Three tasks that get conflated

The distinction matters commercially because a vendor claiming “detection” may be doing tagging with a sliding window, which produces onsets quantised to the window and offsets that are essentially fabricated. Ask what the output granularity is.

One structural point that catches teams coming from image classification: this is multi-label, not multi-class. A car alarm and rain and speech can all be present at once. The output layer is sigmoid, one independent probability per class, not a softmax — and using a softmax here means the model is trained to believe the classes compete, which they do not.

How the models are built

The pipeline is nearly identical to speech recognition up to the encoder: waveform, framing, mel spectrogram, a convolutional or transformer encoder. What differs is the head and the training data.

• The pretraining corpus is AudioSet — Google’s 2017 release of roughly two million ten-second clips drawn from YouTube and labelled against an ontology of 527 sound classes. Nearly every general-purpose audio classifier you can download was pretrained on it, which means it inherits its class list, its label noise and its distribution.
• The standard open baselines are convolutional networks trained on that corpus — PANNs, published by Kong and colleagues in 2020, being the widely used research family, and MobileNet-based variants being the usual choice where the model has to run on a device.
• You almost never train from scratch. Take a pretrained encoder, freeze most of it, and fit a small head on your classes. Even a few hundred labelled examples per class is often enough, because the representation is doing the work.
• Embeddings are often more useful than the classifier. For “does this sound like the recording from the day the pump failed”, similarity search over the encoder’s embeddings beats defining classes at all — and it does not require you to know in advance what you are looking for, which for anomaly detection is the entire point.

The false-positive arithmetic

Here is the number that decides whether the system survives contact with the people who have to act on it, and it is arithmetic rather than machine learning.

A continuous monitor makes one decision per window.

  window hop      h = 1 second
  channels        c = 1
  decisions/day   86,400 / h * c = 86,400

Suppose the detector has a per-window false-positive rate of 1%,
which sounds like a good model:

  false alerts/day = 86,400 * 0.01 = 864

Nobody looks at 864 alerts a day. The system is switched off in
week two.

Work backwards from what people will tolerate instead:

  target: 2 false alerts per day per channel

  required per-window FPR = 2 / 86,400 = 2.3e-5

That is a false-positive rate of about 0.0023% -- three orders of
magnitude better than the "good" model above.

Now the base rate, which makes it worse. Suppose the real event
occurs 5 times a day:

  true positives (at 90% recall)  = 4.5/day
  false positives (at FPR 2.3e-5) = 2/day
  precision = 4.5 / 6.5 = 69%

At FPR 1e-4 instead:

  false positives = 8.6/day
  precision = 4.5 / 13.1 = 34%   -- two thirds of alerts are wrong

A rare event over a large number of decisions is the classic base
rate problem, and no amount of model quality removes it. It is
managed by reducing the number of decisions.

Which is the practical lesson. Three mechanisms reduce the decision count, and all three are worth more than a better classifier:

1. Aggregate before alerting. Require k of the last n windows above threshold. A single 1-second spike is noise; five of the last eight is an event. This trades detection latency for a large reduction in false alarms, and the trade is almost always worth it.
2. Gate on context. Do not run the glass-break detector while the shop is open. Do not run the machine-fault detector while the machine is off. Most deployments can eliminate the majority of their decisions with a condition that has nothing to do with audio.
3. Two-stage detection. A cheap, high-recall, low-precision first stage that runs continuously, and an expensive accurate second stage that only sees the candidates. The second stage makes a few hundred decisions a day instead of 86,400, so its false-positive rate is affordable.

Metrics that respect time

Clip-level accuracy is the wrong instrument for a detector, because a system that gets the class right and the timing wrong scores well.

• Segment-based F1. Divide time into fixed segments (one second is conventional) and score each segment as a multi-label classification. Forgiving about boundaries, easy to compute, and the usual default.
• Event-based F1. Score whole events, with a tolerance — commonly 200 ms on the onset and a proportional tolerance on the duration. Far stricter, and the right measure when the product acts on individual events. It punishes fragmentation heavily: one real event reported as three short ones counts as one hit and two false alarms.
• Polyphonic sound detection score (PSDS). Introduced by Bilen and colleagues in 2020, and designed to remove the arbitrary operating-point and boundary choices the two above depend on by integrating performance over thresholds. Worth using when comparing systems rather than tuning one.
• Report per class, always. Sound event datasets are wildly imbalanced. A macro average over classes and a micro average over events say different things, and the class you care about is usually the rare one.

And report the operating point you chose along with the threshold. Detection performance is a curve; a single F1 is one point on it, chosen by somebody.

Getting the data

Data, not architecture, is where these projects succeed or fail.

• Weak labels are much cheaper than strong ones. “This clip contains a compressor fault” takes seconds to annotate; “the fault runs from 4.2 s to 7.8 s” takes minutes. Multiple-instance learning trains a frame-level detector from clip-level labels, and is the standard route when strong labels are unaffordable.
• Synthesise the mixtures. Isolated event recordings mixed into your real background noise at controlled signal-to-noise ratios gives you unlimited strongly-labelled data with exact timestamps, because you know when you mixed each event in. The standard caveat applies: models trained only on synthetic mixtures transfer imperfectly, so keep real recordings for evaluation.
• Collect the negatives that matter. The hard cases are the sounds that resemble your target — a dropped tray for breaking glass, a reversing alarm for a fire alarm. Mining these from real deployment audio and adding them as negatives does more for precision than more positives ever will.
• Record the environment before you promise anything. Reverberation, microphone placement and the noise floor of the actual site dominate performance, and none of them is visible in a dataset somebody else collected.

Deploying one

1. Run it on the edge if you can. These models are small enough for a microcontroller-class device in many cases, and processing audio locally means only events leave the site. That is both a bandwidth decision and a privacy decision, and the privacy half is usually the one that gets the project approved.
2. Decide about speech explicitly. A microphone in a workplace or a public space captures conversation whether you want it or not, and the fact that your model only looks for machine faults is not by itself an answer to anyone asking about it. Non-speech monitoring in occupied spaces raises the same questions as recording people does, and processing locally without retaining audio is the strongest available answer.
3. Log the audio around every alert, briefly. A few seconds either side, retained for a short fixed window. Without it you cannot tell a false positive from a missed cause, and the model never improves.
4. Give people a one-click “this was wrong”. It is the only source of hard negatives from the real environment, and the environment is what you cannot simulate.
5. Watch for drift. New machinery, a new HVAC unit, a different season of birdsong. Track the alert rate per channel as a time series; a step change in it is your earliest warning that the acoustic environment moved.