Synthetic Data: When It Saves You and When It Sinks Your Model

Synthetic data is having its third hype cycle in five years. The marketing claim hasn't changed: replace expensive real-data labeling with cheap generated data, train models that work in the real world. The reality hasn't changed either: synthetic data helps in narrow cases, hurts in most, and never replaces real labeling at scale.

Here's what to actually do with it.

Where synthetic data wins

Rare-event classes. A defect type that occurs in 1 of 10,000 parts cannot be sampled cost-effectively from real production. Generate it synthetically, add it to the training set, accept that the model will be slightly worse at detecting it than if you had real samples — but better than not training on it at all.

Privacy-regulated domains. Medical imaging where real samples cannot leave the hospital can be augmented with synthetic samples generated from a diffusion model fine-tuned on the real data. This is genuinely useful where compliance prevents data sharing.

Domain randomization for sim-to-real transfer in robotics. Robotics policies trained in simulation with sufficient domain randomization transfer to real-world hardware with measurable success. This is a real and growing application.

Where synthetic data loses

The 80% case: replacing real labeled data with cheaper synthetic data and expecting equivalent performance. Models trained primarily on synthetic data consistently underperform models trained on real data when evaluated against real production samples. The gap is usually 10-25 accuracy points, not the 'roughly equivalent' that marketing materials suggest.

The reason: domain gap. Synthetic images look almost like real images. The 'almost' is what your model learns to depend on. In production, the dependencies don't hold.

Overconfidence is the silent killer

Models trained on synthetic data often have artificially high training accuracy — synthetic samples are easier to classify because they're synthetic. The model reports 98% accuracy. Production accuracy is 73%. The team trusts the 98% and ships. The customer experience tells the real story two weeks later.

If you use synthetic data, your evaluation set must be 100% real-world samples. No exceptions.

The honest economics

Generating 10,000 high-quality synthetic images costs $2,000-8,000 in compute and engineering time. Labeling 10,000 real images costs $1,000-3,000 with AI-assist. Real labeling is often cheaper than synthetic generation for non-rare-event cases. The synthetic data marketing has had it backward.

The hybrid pattern that works

Real data as the foundation. Synthetic data for rare classes only. Synthetic data for data augmentation (rotations, lighting variations, occlusion synthesis on real backgrounds) — this is what 'synthetic augmentation' should mean. Evaluation always on real samples.

This pattern delivers the real benefits (better coverage of rare cases) without the cost (model degradation from synthetic overfit).

Vendor claims to ignore

'95% accuracy with 90% synthetic data' — almost always measured on a synthetic test set, not real-world. 'No labeling required' — true for the generated data, but you still need real labeled data for evaluation and fine-tuning. 'Closes the domain gap' — domain gap shrinks but doesn't close; the residual is real and observable.

What to do this quarter

If your project has rare classes you cannot sample, pilot synthetic data for those specific classes. Measure the resulting accuracy against a real evaluation set. If it lifts model performance on the rare class without degrading performance elsewhere, keep it. Otherwise, abandon and focus on real-data labeling. Intellabel's labeling workflows and active learning loops are built around real data; synthetic data is a supplement, not a replacement. The teams that get the most from AI in 2026 still label real data well.

From Labeling to Structured AI Data Pipelines