Drift Detection That Actually Catches the Right Problems

Every MLOps platform offers drift detection. Most teams using it either ignore the alerts because they're noise, or react to false positives and retrain too often. The combination — high alert volume, low signal — trains the team to mute the monitoring entirely. Six months later, real drift goes uncaught.

Here's how to build drift monitoring that does its job.

Four drift signatures, not one

The catch-all term 'drift' actually covers four distinct phenomena. Data drift — input distribution changes (new camera angles, new lighting). Label drift — class frequencies shift in production (more defect type A this quarter, less type B). Concept drift — the mapping from input to label changes (the definition of 'defective' updated). Prediction drift — model output distribution shifts even when inputs look stationary.

Each requires a different response. Most monitoring conflates them.

Data drift signals that matter

Don't monitor every input feature with KS statistics — you'll get false positives from natural variation. Instead, monitor embeddings of your input data. Train a small autoencoder or use foundation model embeddings, then track distributional shift in the embedding space. This catches genuine domain shift while ignoring noise.

Practical threshold: if embedding centroid distance between rolling 7-day windows exceeds 2 standard deviations of historical, investigate. Otherwise mute.

Label drift is usually under-monitored

If your production traffic is being labeled (continuous learning loop) or sampled-and-labeled (audit), track class frequency changes. A 30% shift in 'defect type A' incidence often precedes a real-world process change at your customer — your model isn't broken, but it will be if you don't catch it.

Concept drift requires labels

You cannot detect concept drift from inputs and predictions alone. You need ground-truth labels on a sampled subset of production data, comparing prediction accuracy against the labeled baseline over time.

This is the most expensive monitoring to operate but the most valuable. Allocate budget for sampling and labeling a holdout from production weekly.

Prediction drift is the canary, not the disease

Output distribution shifts often signal underlying data or concept drift. Easy to monitor (no labels needed). Useful as an early warning. Not actionable on its own — when it triggers, you investigate to find the root cause in one of the other three categories.

The retraining decision

Don't retrain on every drift alert. Retraining is expensive in compute, in dataset preparation, and in the risk of model regression. The right pattern: drift signal triggers an investigation, investigation produces evidence of root cause, root cause determines whether the fix is retraining, dataset expansion, or a deployment patch.

Intellabel's MLOps tier wires production monitoring into the same workflow that owns labeling and training — so when drift triggers, you can sample production frames into a labeling queue, expand the dataset, and retrain through the same pipeline that built the original model. The shorter that loop, the lower the operational cost of drift, and the more aggressive you can be about catching real drift early.

Tune for precision, not recall

An alert your team mutes is worse than no alert at all. Better to miss subtle drift than to flood your engineers with daily false positives. Start with conservative thresholds, calibrate against historical incidents, and tighten only after the team trusts the signal.

From Labeling to Structured AI Data Pipelines