Labeling Challenges in Industrial IoT Sensor Data

A person can label a photograph in two seconds because the whole example is visible at once. A person cannot label a week of vibration data at all, because the thing being labelled is not visible, has no clear start, and is often only knowable after the machine has already failed.

Why this is not image labelling

Four properties make continuous industrial telemetry a different labelling problem, and each of them removes an assumption that image and text annotation quietly rely on.

• The example has no natural boundary. An image is one example. A sensor stream is continuous, and turning it into examples requires choosing a window length that is itself a modelling decision — so the labels are not independent of the model design, and re-windowing means re-labelling.
• The label is not present in the data a human can perceive. A domain expert can look at a photograph and see a defect. Almost nobody can look at a raw 5 kHz accelerometer trace and see an incipient bearing fault; it is visible in a spectrum after processing, if at all, and often only in hindsight.
• The positive class is vanishingly rare. A machine that fails twice a year, sampled continuously, produces two positive events against millions of windows. This is a far more extreme imbalance than most classification problems, and it means the labels you most need are the ones you have almost none of — see dataset balancing for what can and cannot be done about it.
• Ground truth may require destroying the asset. Confirming what was actually wrong with a bearing often means stripping the machine. Labels therefore arrive only when maintenance happens, and only in the form maintenance records take.

What labelling actually costs

The inputs below are assumptions and the arithmetic is the point. Suppose an engineer reviews telemetry to mark fault periods, and each hour of machine data takes four minutes of expert review with the aid of a plotting tool.

ASSUMPTIONS
  machines                 40
  telemetry                continuous, 24 h/day
  review rate              4 minutes of expert time per hour of data
  fully loaded expert cost $90 per hour  (assumed)
  target                   90 days of labelled history

DATA TO REVIEW
  machine-hours   40 * 24 * 90            = 86,400 hours

EXPERT TIME
  minutes         86,400 * 4              = 345,600 min
  hours           345,600 / 60            = 5,760 h
  cost            5,760 * $90             = $518,400
  elapsed, 1 FTE  5,760 / 1,800 h/yr      = 3.2 person-years

WHAT IT BUYS
  failures in 90 days, at 2 per machine-year:
    40 * 2 * (90/365)                     = about 20 events

Roughly $26,000 of expert time per labelled failure event, and 20
positive examples is not a training set for anything supervised.

That result is the argument, and it does not improve much with cheaper labour, because the binding constraint is expertise rather than hours. Exhaustive labelling of continuous industrial telemetry is not a budgeting problem to be solved with a larger budget; it is the wrong approach, and every practical system gets its labels somewhere else.

Where labels come from instead

Maintenance records as weak labels

The computerised maintenance management system already contains work orders with an asset, a date, a fault description and often a replaced part. That is a label source at essentially zero marginal cost, and it is what most deployed predictive maintenance runs on. It is also weak in three specific ways: the timestamp is when the work was done, not when the fault began; free-text descriptions are inconsistent and need mapping to a taxonomy; and a work order records what was replaced, which is not always what had failed.

Run-to-failure datasets

Where a component is cheap enough to run deliberately to destruction, the resulting record has unambiguous ground truth throughout its life. The public reference examples are NASA’s Prognostics Center of Excellence datasets, including the C-MAPSS turbofan degradation simulation described by Saxena and colleagues at the 2008 International Conference on Prognostics and Health Management, and the bearing run-to-failure sets. They are invaluable for method development and they are not your machine, so a model validated only on them has been validated on a different distribution.

Physics and simulation

Where the fault signature is analytically known — bearing defect frequencies follow from geometry and shaft speed — labels can be generated by construction, and synthetic faults can be injected into healthy data. The risk is a model that learns the injection procedure rather than the fault, which shows up as excellent synthetic performance and no real-world detection at all.

Unsupervised framing

The most common resolution is to avoid needing positive labels: characterise healthy operation, which you have in abundance, and flag departures from it. That converts the problem from classification to novelty detection and needs labels only for validation, which is a far smaller number. The methods are those in correlated anomaly detection.

The boundary problem

Even with a known fault, when it started is genuinely ambiguous, and this is not a shortcoming of the annotator. Degradation is continuous. A bearing that failed on the 14th was measurably different from healthy on the 2nd and imperceptibly different on the 1st. Any binary label imposes a cut on a gradient.

The consequences are practical. Two experts labelling the same failure will disagree on onset by days, so inter-annotator agreement measured on exact boundaries looks terrible even when both are right about the event. A model trained on windows just inside an arbitrary boundary is being taught that healthy-looking data is faulty, which caps its achievable precision. And evaluation metrics computed per window punish a detector that fires two days early — which is precisely the behaviour the system exists to produce.

The mitigations are all forms of admitting the ambiguity rather than resolving it. Label a transition band and exclude it from training. Frame the target as remaining useful life, a continuous quantity, which removes the cut entirely. Evaluate per event with a detection horizon — did the detector fire at all in the window before failure, and how early — instead of per window, since an operator cares about notice period and false alarms per month, not about window-level recall.

Designing around scarce labels

• Capture labels at the moment of maintenance. A two-field prompt in the technician’s existing workflow — what was wrong, when did it start being wrong — costs seconds and is worth more than any amount of retrospective review. This is the highest-return change available and it is organisational rather than technical.
• Record interventions, not just failures. A cleaned filter or a tightened belt changes the data as much as a fault does. Without those events, the model sees unexplained step changes and learns noise.
• Use active learning to spend expert time well. Have the unsupervised detector rank candidate periods and ask the expert only about the top few. The same expert hours produce disproportionately more information when spent on the ambiguous cases rather than on uniformly sampled healthy data.
• Keep raw data around the labels. When the window length or the feature set changes, the labels have to be reapplied to data at full resolution. A retention policy that downsamples everything older than a month destroys the ability to relabel, which is the expensive asset.
• Validate on events, held out by machine. Random window splits leak, because windows from the same failure appear in both train and test and the reported score is meaningless. Hold out whole machines or whole failure events.