One published case sums up the imbalance problem on its own. On a real, imbalanced dataset of ultrasonic weld images, a standard neural network reached an accuracy of 0.671, reflecting only majority-class performance, with F1 score, precision and recall all at 0.000 and an area under the curve of 0.500.
Two thirds correct, and not a single defect detected anywhere in the entire test set. The model had learned the only strategy the imbalance made rational: always answer conforming. This is not a laboratory curiosity, it is the default behaviour of any supervised learning faced with an extreme minority class, and it is why a defect detection project cannot be steered with the usual metrics.
This article approaches rarity from every practical angle: what it breaks, how to measure it, how to act on collection, on annotation, on learning, and what synthetic data actually contributes. It extends the complete guide to defect detection by computer vision.
Why imbalance is the central problem of defect detection
On a controlled line, conforming parts massively dominate the defect detection corpus, and that dominance is not a corpus flaw: it exactly reflects the industrial reality the system will face.
Two superimposed imbalances
They are usually discussed as one and they are not. The first imbalance sets conforming against defective, with ratios routinely reaching several hundred to one. The second, less discussed and often more troublesome, concerns the distribution across defect types. It follows a long tail: a few frequent failure modes concentrate most examples, while a multitude of rare types appear only a handful of times. A large industrial dataset illustrates that structure, with roughly 500,000 samples across 81 categories derived from real manufacturing processes.
The two imbalances call for genuinely different answers. The first is well handled by anomaly approaches, which require no defective examples. The second requires examples, since the task is distinguishing types from one another, and that is where collection and synthesis strategies become decisive.
What imbalance breaks
Three distinct mechanisms stop working simultaneously, and their conjunction is what makes the problem formidable. Aggregate metrics first, as the opening case shows: flattering accuracy can coexist with zero recall. Learning next, since the loss function is dominated by the majority class and the model has no incentive to learn rare cases. And quality control last, because a uniformly drawn verification sample contains almost no rare defects and says nothing about whether the corpus covers them.
Measuring before correcting
The first action in a defect detection project is not rebalancing but measuring properly, otherwise corrections are steered blind.
Three principles are enough to restore an honest measurement. Report indicators class by class and never in aggregate, which would immediately have exposed the zero recall in the opening example. Separate omissions from false detections explicitly, since they have neither the same causes nor the same economic consequences. And favour indicators insensitive to imbalance, such as per-class recall with its associated precision, rather than a global accuracy that says nothing once the majority class dominates.
A statistical sizing requirement follows directly from this. Estimating a detection rate on a class represented by seven examples has no statistical meaning. Either accept that nothing can be concluded about that class and document it, or build a deliberately enriched evaluation set, accepting that its composition no longer reflects real prevalence and stating so.
Acting on collection of rare defects
Before any modelling trick, the honest answer to rarity is to obtain more real examples, and several levers exist before resorting to synthesis.
Pre-filtering
This is the single highest-leverage intervention available. Manually traversing tens of thousands of conforming images to extract a few defects is the dominant cost line of a defect detection project. Pre-filtering, even crudely, changes the economics: an anomaly model trained on conforming parts alone, or a simple image processing rule, is enough to discard most uninteresting images and concentrate human effort on candidates.
One precaution makes the difference between a useful shortcut and a silent trap. The precaution is simple and always worthwhile: keep a random, unfiltered sample to measure what the pre-filter misses. Without that measurement, the corpus silently inherits the pre-filter’s bias, and the final model will learn to detect only what the pre-filter already knew how to spot.
Pooling across lines and sites
The cheapest additional examples are often ones the organisation already owns. Where the same product is manufactured on several lines or sites, pooling corpora mechanically multiplies the examples available for each rare type. Two precautions apply. Acquisition conditions must be close enough, or the source must be retained as metadata to allow later per-site analysis. And the ontology must be realigned, since two sites rarely use the same vocabulary for the same defects, even within one group.
That realignment work is tedious and regularly profitable, and it is rarely anyone’s job: it turns several insufficient corpora into one usable corpus, with no additional collection.
Active learning
Random sampling is a poor use of annotation budget on an imbalanced corpus. Rather than annotating at random, the model under training is asked which images would help it most, generally those on which it hesitates. The gain is particularly marked on heavily imbalanced corpora, since uncertainty naturally concentrates on rare and borderline cases. The trade-off is organisational: the method requires working in iterations rather than one batch, which has contractual implications to anticipate.
Archives and scrap
Two sources of real examples are regularly overlooked although they cost almost nothing. Both predate the project and require no new production. Line image archives, where they exist, often hold years of production, though only documented scrap records make the interesting cases findable. And physically retained scrap parts held by the quality department constitute a source of real examples that need only be re-imaged under the chosen setup. This second route is by far the most cost-effective when available, because it supplies real defects under the exact conditions of the system.
Controlled provocation
When nothing else works, defects can be manufactured on purpose. On some processes, defective parts can be deliberately produced by degrading a parameter. The approach costs both material and machine time, and it produces defects sometimes more pronounced than those occurring naturally. It nonetheless remains valuable for those categories that no other route can supply, provided such examples are documented as provoked and not mixed indiscriminately with natural defects in an evaluation set.
Acting on annotation
The defect detection annotation strategy can itself be adapted to imbalance, which is less obvious and often forgotten.
Three adaptations are worth building into the protocol from the start. The first useful adaptation is differentiating annotation effort. There is no reason to annotate an obviously conforming image and a borderline defect image with the same care. Rapid classification suffices for the first, while the second justifies precise segmentation, double reading and arbitration. That differentiation, written into the protocol, markedly reduces cost without degrading what matters.
The second adaptation concerns the uncatalogued anomaly class. In a long-tail domain, the initial ontology is necessarily incomplete, and the channel through which new types surface determines whether the project can evolve. That class must be easy to use, systematically reviewed by the quality expert, and give rise to documented, versioned ontology extensions.
The third concerns the preservation of uncertainty rather than its removal. On borderline cases, forcing a binary label destroys valuable information. Retaining an uncertainty annotation, or the divergent multiple annotations, allows a consensus threshold suited to the use to be chosen later rather than freezing one interpretation.
Acting on sampling and learning
Classical rebalancing techniques apply to defect detection and provide real value, provided their limits are understood.
They are cheap to apply and worth applying, within limits. Oversampling the minority class, weighting the loss function or using losses designed for rare classes do improve recall. They create no information, however: repeating the seven available examples of a defect type a hundred times teaches the model those seven examples, not the variability of the type. On very sparsely represented classes, these techniques shift the problem from underfitting to overfitting without solving it.
It is precisely that limit which has pushed the field towards two more structural answers: synthetic data generation and the anomaly approach.
Synthetic data in defect detection
Generating the examples that reality does not supply is the most active research area in the field, and the one where published results demand the closest reading.
Three families of generation
They require neither the same resources nor the same prerequisites. Physical simulation renders images from geometric models into which a defect has been inserted, with full control over its nature and automatic labelling. It demands detailed modelling and suffers from the gap between simulation and reality. Generative models learn defect appearance directly from existing images and support controlled editing. Composition, finally, inserts a real defect taken from one image into a conforming image, which is simple and effective where the defect is localised and the background homogeneous.
What the results show
The published gains are large enough to be worth stating precisely. The weld case cited in the introduction continues spectacularly. After adding synthetic images, the model reached an F1 score of 0.884 with perfect precision, representing up to a 38.9 per cent increase in accuracy on the test set, from only a few dozen to a few hundred generated images.
More recent work confirms the required volume is modest: a lightweight adaptation method for a diffusion model is tuned on as few as twenty to fifty real images of a rare class and consistently improves that class’s recall and F1 compared with training on real data alone, with a predictable relationship between the synthetic-to-real ratio and performance.
The golden rule
One sentence in that same study carries more practical weight than any of its performance figures: all evaluations are performed on held-out sets composed exclusively of real images. That rule must never be broken. Evaluating a model on synthetic data produces a flattering and meaningless figure, since generator and detector share the same biases.
What synthesis does not do
The limits are as instructive as the gains. An end-to-end assessment of a diffusion-based generation pipeline for surface defects concludes with useful nuance: its strongest value lies in strengthening scarce real datasets rather than substituting for them, the same work emphasising the importance of domain-specific adaptation and annotation quality control.
The practical conclusion is clear, and worth retaining exactly as stated. Synthetic data does not remove the need to collect real data, it makes a limited real collection pay off. A project with no real example of a defect type cannot generate one, since the generator must learn the appearance somewhere. The effective sequence is therefore: obtain a few dozen real examples, generate from them, evaluate on held-out real data, and continue collecting in operation.
The evaluation set as a special case
One point deserves isolating, because it is a source of erroneous conclusions even in published work: an evaluation corpus in defect detection cannot follow the same rules as a training corpus.
Three requirements distinguish an evaluation corpus from everything else in the project. It must consist exclusively of real data, with no unflagged synthetic or provoked example. It must be sufficiently populated on every class about which anything is to be asserted, which often means deliberately enriching its composition and therefore no longer reflecting production prevalence. And it must stay frozen, with no after-the-fact correction, otherwise the measurement loses its demonstrative value.
That last point requires keeping alongside it an estimate of real prevalence, measured on a random unenriched sample, without which a detection rate cannot be converted into a number of parts escaping per shift. That conversion is what interests the industrial client, and it needs both measurements.
The anomaly approach as a structural answer
The most radical route available is to stop requiring defective examples altogether. A model trained only on conforming parts flags any deviation, which resolves the imbalance problem and the unknown defect problem at once.
The price is twofold and has to be accepted knowingly rather than discovered later. The system flags without naming, which limits its use where the decision depends on defect type. And it also flags normal but unusual variation, producing false alerts that only good control of acquisition conditions can contain.
Neither approach is sufficient alone, which is why the field has converged on combining them. The architecture that works best in practice combines both logics: anomaly detection for broad coverage including new types, and a supervised model to classify and qualify frequent types. That combination directly defines the annotation strategy: a large corpus of conforming parts, cheap to assemble, and a finely annotated corpus on frequent defects and on cases flagged by the anomaly system.
Knowing when to stop collecting
A question rarely asked explicitly determines when a rare-defect project can be considered finished: how many examples of a type are enough?
There is no universal figure, but a workable operational answer exists and it is empirical rather than theoretical. Collect until adding examples stops changing the measured performance on that class, tracked across successive iterations. That saturation point varies enormously from one defect type to another, arriving quickly on visually stereotyped defects and receding for a long time on those with variable appearance.
Two practical consequences follow from adopting that criterion. The target is set per class rather than globally, which means some classes are declared complete while others continue collecting. And the criterion presupposes a held-out evaluation set large enough for that class to make the measurement meaningful, which returns to the sizing question and is often the real binding constraint. In practice, a defect detection project is limited less by its ability to collect training examples than by its ability to assemble enough real examples to prove anything about them.
The cost of rarity
The cost structure of a rare-defect project differs markedly from a conventional vision project, and ignoring that leads to systematically overrun budgets.
Three lines dominate, and only one of them is annotation in the usual sense. The largest line is not annotating defects but finding them. Locating two hundred examples of a type occurring once in ten thousand means traversing, pre-filtering or archiving two million images, and that work takes weeks even when largely automated. The second line is arbitration: rare cases are by construction the ones nobody has a habit for, which multiplies escalations to the quality expert. The third is rework, since the ontology extends as discoveries accumulate and earlier batches sometimes have to be re-examined against a new class.
The consequence for budgeting is that a defect detection quote built on a unit price per image describes the work poorly. A credible estimate rests on three distinct volumes: images traversed, images annotated and decisions arbitrated, whose ratios vary by a considerable factor with prevalence.
Annotation within a rare defect detection strategy
Everything above converges towards an annotation organisation quite different from a conventional vision project.
Four characteristics define an annotation programme suited to rarity. It is iterative rather than sequential: several short cycles of collection, annotation, training and error analysis, each steering the next towards zones of uncertainty. It is differentiated: minimal effort on the conforming volume, maximal effort on rare and borderline cases. It is evolving: the ontology extends as types are discovered, with rigorous versioning so batches stay comparable. And it is continuous: rarity means collecting interesting cases spans the system’s life, not an initial campaign.
This organisation has a rarely anticipated contractual consequence. A defect detection project sits poorly with a single fixed fee defined on an image volume, because the useful volume is not known in advance and value concentrates on an unpredictable fraction of the corpus. An iteration-based framework, with a decision point after each cycle, matches the nature of the problem better.
The most common mistakes
These failures recur often enough across rare-defect projects that naming them is usually enough to avoid them.
- Steering the project on global accuracy and missing zero recall on the minority class.
- Drawing conclusions on a class represented by a handful of examples without flagging the uncertainty.
- Pre-filtering without keeping a random sample large enough to measure what is discarded.
- Oversampling a few examples heavily and mistaking overfitting for progress.
- Evaluating a model on synthetic data rather than on real data held out for that purpose.
- Hoping to generate examples of a defect type for which no real example exists.
- Mixing provoked and natural defects in an evaluation set without documenting it.
- Annotating conforming images and borderline cases with the same care.
- Providing no uncatalogued anomaly class and closing the door to new types.
- Treating collection as a single campaign when rarity requires sustaining it across the system’s life.
What to take away
In defect detection, imbalance is not a dataset flaw to correct, it is a property of the industrial problem to be handled by method. Rebalancing techniques help, synthesis helps more, but none replaces correct measurement and a collection strategy conceived for the long run.
Three decisions structure an approach that works. Measure class by class from the first batch, separating omissions from false detections, to avoid steering on an indicator that hides exactly what matters. Invest first in collecting a few dozen real examples per type, since that is the threshold above which synthesis becomes usable. And organise the project in short iterations with differentiated annotation, rather than as a single campaign sized on an image volume.
For approaches, ontology and annotation tasks, the complete guide to defect detection sets the general frame. For a concrete application on a material family where fine defects are particularly hard to isolate, the article on defect detection on metal surfaces covers steel, welds and machining.
To explore delivery arrangements, supported formats and applicable control mechanisms, see our dedicated page on annotation for industry. And if you are building a corpus in a rare-defect domain and want the collection and annotation strategy scoped, let us discuss your project.
