Combining synthetic and real data works best when each source does the job it is good at. Synthetic data supplies volume, exact labels and the rare cases you can't capture. Real data anchors the model to what your cameras actually record. A hybrid training pipeline is the set of decisions that connects the two: how you mix them, in what order, in what proportion, and how you check that the mix helps.
This guide is written for computer vision teams. It covers the three main ways to combine the data, how to pick a ratio, a step-by-step pipeline, and the case where adding real data makes the model worse.
A hybrid training pipeline is a training setup that uses both synthetic and real images to build one model, with a defined strategy for how the two are combined and a real-world test set that decides whether the combination works. The synthetic part is usually rendered from 3D scenes or produced by generative models. The real part is usually smaller, more expensive and closer to production conditions.
Most guides on this topic come from tabular data, where synthetic records are used for privacy. Vision is a different problem. The gap between a render and a photo is visual (lighting, textures, sensor noise), so the pipeline has to manage that gap, not only the data volume. For background on why the two sources behave differently, see our comparison of synthetic data vs real data.
Both datasets go into one training set and the model sees them together from the first epoch. It's the easiest setup to run, and it works well when the domain gap is small: good assets, realistic rendering, a camera model close to the real one. The main control you have is the sampling ratio, meaning how often a batch draws from each source.
If the gap is large, mixing has a weakness. The model can learn two separate "worlds" and switch between them, which shows up as strong scores on synthetic validation and weaker ones on real images.
Here training happens in two stages. The model first learns from a large synthetic set, then a second, shorter stage adapts it to real images. A review of synthetic-to-real ratio studies recommends this sequential approach when the domain gap is large, since the fine-tuning stage pulls the model toward the real distribution after it has learned general shapes and classes from synthetic data.
This is the default choice when real images are few or expensive to label. It also lets you reuse one synthetic pretrained model across several sites, each with its own short fine-tune.
The third approach starts from real data. You train on what you have, find where the model fails, and generate synthetic images only for those failures: a rare class, night scenes, a camera angle missing from the collection, heavy occlusion. Synthetic data here is a patch, not a foundation.
It's the most data-efficient option, and it's how many teams start. It does require a good error analysis, since you need to know which scenarios to render. Our guide on generating edge cases for computer vision models covers how to plan them.
There is no universal ratio. Published results disagree because tasks, models and domain gaps differ. One study fine-tuned DenseNet201 on traffic image classification across seven real-to-synthetic configurations and found the best accuracy at 1 real image for every 3 synthetic ones (Istanbul Sabahattin Zaim University). That is one result on one task, useful as a starting point, not a rule.
A practical way to find your ratio is to sweep it. Fix the real data, then train with synthetic shares of, for example, 0%, 50%, 75% and 90%, and compare every run on the same real test set. Three factors usually move the optimum:
Expect diminishing returns. Past a certain point, more synthetic images add training time without moving the real-world metric, and at strict thresholds they can pull it down.
The first step is the one teams skip most often, and it's the one that makes the rest measurable.
Adding real data is usually assumed to help. Sometimes it doesn't. If the real set is small, it carries its own biases: one site, one season, one camera, one annotator's habits. Fine-tuning on it can pull a model that generalized well toward those quirks. A write-up from the edge AI community reported a 13.8% drop in mAP50-95 after fine-tuning a synthetic-trained detector on a limited real dataset, even though local metrics on that dataset improved.
Three signs point to this problem. The score on the fine-tuning set rises while the score on an independent real test set falls. Performance drops on scenarios the real set doesn't contain. Errors start to follow the labeling style of the real set rather than the objects themselves.
The fixes are simple. Keep fine-tuning short and use a lower learning rate. Keep some synthetic data in the fine-tuning batches instead of switching to real only. And make sure the test set comes from different conditions than the fine-tuning set.
The only number that matters is performance on real data the model has never seen. Track the gap between synthetic validation and real test scores; if it's large, the domain gap is the bottleneck, and better assets or randomization will help more than another ratio change. Break results down per class and per condition, because a hybrid set often improves rare classes while leaving common ones flat, and an averaged metric hides that.
It also pays to test the synthetic data on its own before mixing it. Our checklist on how to validate synthetic data quality before model training covers those checks, and the sim-to-real gap article explains how to tell an appearance problem from a content problem.
There is evidence the payoff can be large. A study on a hybrid generation pipeline found that fine-tuning on its synthetic images outperformed fine-tuning on real photographs for the same ObjectNet classes (Journal of Electronic Imaging).
The synthetic half of a hybrid pipeline is only as useful as the scenes behind it. Vivid 3D builds labeled synthetic datasets from simulation-ready 3D assets, with control over lighting, materials, camera placement and scene composition, so you can generate a broad pretraining set or a narrow batch for one failure case. See how it works on the synthetic data for computer vision page, or book a demo to plan how to combine synthetic and real data for your model.
There isn't a single best ratio. Studies report different optimums depending on task and domain gap, and one traffic classification study found 1 real to 3 synthetic worked best. Run a ratio sweep and choose by score on a fixed real test set.
For some tasks with realistic rendering, synthetic-only models get close to real-trained ones. In most production systems, a small amount of real data for fine-tuning and, above all, for testing still gives more reliable results.
Mix when the domain gap is small and you have a reasonable amount of real data. Pretrain on synthetic and fine-tune on real when the gap is large or real images are scarce. Keep some synthetic data in the fine-tuning stage to avoid overfitting to a small real set.
It depends on the task, but the real test set matters more than the real training set. A few hundred carefully labeled real images for evaluation is a common starting point; the real share of training data is then tuned through experiments.
