There is a disconcerting gap between published performance in defect detection and what the same methods deliver on a production line. On the field’s historical reference dataset, current approaches reach segmentation scores routinely between 92 and 97 per cent, a level indicating saturation of the dataset. The problem therefore looks solved.It is not. On a dataset designed to reproduce more realistic conditions, notably lighting variations absent from the training data, results from a recent challenge fall to segmentation scores between 54 and 63 per cent depending on the test split. The same class of method, the same task, and performance halved as soon as acquisition conditions move.That gap is the central subject of industrial defect detection, and it plays out largely before the algorithm: in defining what counts as a defect, in assembling the dataset and in how it is annotated. This guide covers the field end to end, from available approaches to annotation tasks, by way of sensors, ontology and quality control.
What defect detection by computer vision covers
The term defect detection covers the use of computer vision to identify, locate or qualify an anomaly on a product, a component or a surface, with a view to a conformity decision. That broad definition spans quite different situations.Defect, anomaly, non-conformity
Everyday vocabulary conflates three notions with different practical consequences, and separating them is worth doing before anything else. An anomaly is a deviation from expected appearance, regardless of severity. A defect is an anomaly identified as such by a quality reference. A non-conformity is a defect exceeding a tolerance threshold and triggering a decision: scrap, rework, downgrade.The distinction is not academic. It determines what the model must produce. An anomaly system flags what is out of the ordinary without naming it. A defect detection system names and locates. A control system decides. All three require different annotations, and conflating the levels at scoping is the field’s most frequent error.Structural and logical defects
A second, more recent distinction separates structural from logical defects. A structural defect is a local alteration of the material: scratch, crack, stain, missing matter, inclusion. A logical defect concerns arrangement: a missing part, an extra component, an incorrect order, an unexpected colour in an otherwise correct assembly. Methods effective on the first often fail on the second, because a logical defect has no local signature and is only visible at whole-object scale.Why this differs from general-purpose vision
Three characteristics separate defect detection from conventional object recognition, and each has direct consequences for dataset construction.Extreme imbalance
On a controlled line, the defect rate runs to fractions of a per cent. Building a balanced training set would mean traversing enormous volumes of conforming parts to gather a few hundred defective examples, and some defect categories appear only a handful of times a year. This is the field’s central problem, and it governs the choice of approach as much as the annotation strategy.The unknown defect
This asymmetry has no equivalent in most vision applications. An object recognition system knows the list of what it must find. A defect detection system does not: a new failure mode can appear at any time, tied to a change of raw material, tooling or settings. A model trained to recognise known defects will not flag one of a new type, and that silence is more dangerous than a false alarm.Tolerance rather than class
In general-purpose vision, an object either belongs to a class or does not. In industrial inspection, the same defect type is acceptable below a certain size and disqualifying above it, with the threshold depending on the zone of the part, its destination and sometimes the end customer. Annotation must therefore capture not only defect nature but the attributes that allow the tolerance rule to be applied.The main families of approach
Four families of approach coexist in defect detection, and the choice depends directly on the availability of defective examples.Classical methods
Thresholding, morphology, template correlation, dimensional measurement: these remain relevant where the defect is geometrically defined and the environment fully controlled. They are fast, explainable and need no dataset. They break down as soon as normal appearance varies, which is the case for most real surfaces.Supervised learning
Classification, detection or segmentation trained on annotated examples of each defect type. It is the highest-performing approach where examples exist in sufficient number, and the most annotation-intensive. It presupposes knowing the defect list in advance, which makes it vulnerable to new failure modes.Unsupervised anomaly detection
This is the approach that has advanced most recently, and it addresses the imbalance problem directly. The model learns only on conforming examples and flags any deviation. The field’s reference datasets are built on that principle: the historical reference set comprises 5,354 high-resolution images across 15 categories, including 10 object classes and 5 texture classes, with anomalies spanning more than 70 defect types such as scratches, dents and contaminations, each category providing a training set of defect-free images.That design choice reflects the industrial reality it was built from. The advantage is decisive: assembling a set of conforming parts is trivial on a running line. The trade-off is that the system flags without naming, and raises alerts on normal but unusual variations, which creates an acceptance problem in production.Few-shot and pre-trained models
A fourth route has emerged with general-purpose vision models pre-trained on very large image collections, which can be adapted to a new inspection task from a handful of examples. The appeal is obvious where defective samples are scarce, and the results on well contrasted defects are often usable.Two cautions apply. Performance degrades on the fine, low-contrast defects that constitute the hard part of most real inspection tasks, so a promising demonstration on obvious defects predicts little about the ones that matter. And adaptation from few examples inherits whatever those examples happen to contain, which makes their selection a design decision rather than a convenience.Hybrid approaches
In practice, mature deployments combine both logics: anomaly detection for broad coverage including unknown defects, and a supervised model to classify and qualify the frequent defects for which examples exist. That architecture requires a two-level annotation strategy, and it is markedly more robust than a single approach.Sensors and acquisition conditions
In defect detection, acquisition quality weighs more on the final result than the choice of architecture, and it is where a project is won or lost.Lighting
This is the dominant parameter. Grazing light reveals relief defects, diffuse light reveals colour defects, dark-field lighting reveals fine scratches. The same defect can be perfectly visible under one setup and invisible under another. Designing the lighting around the defects sought, rather than hoping an algorithm will compensate, is the highest-return decision in a defect detection project.The stability of that lighting matters as much as its design. The results cited in the introduction show it: it is precisely the introduction of lighting variation between training and test that halves performance. In production, ambient light, lens contamination and source ageing produce exactly that kind of drift.Modalities
The choice of modality follows from the physical nature of the defect sought. Colour or monochrome imaging covers most cases. Three-dimensional imaging, through profilometry or stereo vision, becomes necessary where the defect is a deviation of shape rather than appearance. Thermography reveals internal defects or process anomalies. Radiography detects inclusions and porosity invisible at the surface. Each modality calls for its own annotation conventions, and a multimodal project must maintain articulated but distinct ontologies.Throughput
The last acquisition constraint is temporal rather than optical. Line speed constrains exposure time, hence lighting, and imposes a limit on inference time available per part. These constraints are fixed upstream and determine irreversible technical choices, which is why they belong in the scoping documentation rather than being discovered at integration.Annotation tasks in defect detection
Five output formats structure defect detection projects, with very different costs and uses.They are listed here from cheapest to most expensive, which is not the order in which they should be considered. Image-level classification indicates the presence or absence of a defect, and possibly its type. It is the fastest to produce and the least informative, since it does not say where to look. The bounding box localises coarsely, which suffices for sorting and alerting. The polygon or pixel mask delineates the defect precisely, the only option where defect area or length enters the tolerance rule. Counting applies to multiple defects, porosity or inclusions, with the requirements specific to enumeration tasks. And attribute annotation qualifies each defect beyond its type: severity, estimated depth, zone of the part affected.The choice is not a technical preference: it follows from the quality decision rule. If the rule states that a defect larger than two millimetres in a critical zone is disqualifying, the annotation must allow size to be measured and the zone identified, which rules out plain classification. Starting from the quality rule to define the primitive, rather than the reverse, avoids producing a corpus that cannot support the decision.Building the defect ontology
This is the most decisive work in a defect detection project, and the most often rushed.Naming by appearance or by cause
The first question to settle is what the class names actually refer to, and it is where quality departments and vision teams routinely talk past each other. Quality teams name defects by their cause: welding fault, cooling problem, tool drift. A vision model sees only appearance, and two different causes can produce indistinguishable images. An ontology built on causes therefore produces classes the model cannot separate, and mediocre agreement between annotators.The answer is a two-level ontology: an appearance level, reliably annotatable from the image alone, and a cause level, filled in where the information is available elsewhere. The two connect, and the corpus stays usable for maintenance as well as for inspection.Severity
Severity is what links observation to decision, and defining it deserves as much care as the class list. A severity scale must be defined with objective criteria, size, position, contrast, rather than with qualifiers like light, medium or major. A subjective scale produces considerable variability between annotators, and that variability shows up directly in model behaviour near the decision threshold, which is exactly where it matters.Service classes
Beyond the defect classes themselves, three technical classes are indispensable and routinely forgotten. One for images of insufficient quality, with explicit criteria. One for acknowledged uncertainty, which beats a forced decision. And one for visible but uncatalogued anomalies, which is the channel through which new failure modes reach the quality team.Measuring what matters industrially
Academic metrics and industrial metrics do not answer the same question, and translating between them is part of scoping a defect detection project.A research paper optimises a detection score averaged over categories. A production line cares about two numbers with direct financial meaning: the escape rate, meaning non-conforming parts that passed, and the false reject rate, meaning conforming parts wrongly scrapped. These sit on a trade-off curve, and the operating point is a business decision rather than a technical one, since the cost of an escape and the cost of a false reject differ by orders of magnitude depending on the industry.The practical implication for annotation is that the corpus must support measuring both, which means it must contain enough conforming parts to estimate a false reject rate meaningfully. A dataset built only from defective examples, which happens more often than one would expect, cannot answer half the question the plant actually has.The public benchmark trap
Reference datasets in defect detection are valuable for comparing methods and misleading if a production performance expectation is inferred from them.Two properties explain the optimism they induce. Conditions there are controlled, categories few and normal appearance little varied. More recent sets partly correct that bias: one of them adds eight scenarios comprising more than 8,000 high-resolution images, with test data captured under varying lighting conditions not necessarily present in the training data, and non-public ground truth for part of the test set evaluated only through a server.The practical conclusion is that a published performance figure means something only relative to its conditions. Before committing to a target, you have to measure on your own data, under your own lighting, with your own production variability.Quality control of defect detection annotation
A defect detection corpus presents a specific control difficulty: the truth is sometimes undecidable from the image alone.An experienced operator sometimes hesitates, and that hesitation is information to preserve rather than a problem to suppress. Three mechanisms combine. Independent double reading on low-agreement classes, generally fine defects and cases near the severity threshold. Arbitration by a quality expert, whose decisions must be documented and redistributed, since each arbitration creates a precedent applying to hundreds of cases. And insertion of reference cases into the production flow, which measures individual drift on a repetitive and tiring task.Metrics must be reported per class rather than aggregated. On an imbalanced corpus, a global indicator is dominated by conforming images and stays excellent while detection of rare defects collapses. Omissions and false detections must also be separated explicitly, since they have neither the same causes nor the same industrial consequences: an omission lets a non-conforming part through, a false detection degrades yield.Cost and annotation profiles
A defect detection project has an unusual cost profile, dominated not by volume but by rarity.The consequence is that volume-based quoting misrepresents the work. Time spent traversing conforming images to reach defective examples is often the largest line, which makes pre-filtering by a model or a simple rule particularly worthwhile. Quality control also weighs heavily, because contested cases are frequent and arbitrating them mobilises a quality expert whose time is scarce.Role allocation, by contrast, is favourable. Spotting and delineating a visible defect is a perceptual decision and delegates well to trained annotators, provided the protocol is precise and illustrated. Qualifying severity, deciding conformity and arbitrating belong to the quality expert. A well organised chain reserves their time for those three uses and spares them the traversal of conforming images.From prototype to production line
A project rarely fails at feasibility stage and often fails at industrialisation. Four gaps explain most of those failures.None of them is visible during a proof of concept, which is precisely why they are so frequently missed. Drift in acquisition conditions, already discussed, is the first. Product drift comes next: a change of material supplier, shade or process alters normal appearance and triggers unjustified alerts. The third gap concerns operator acceptance, since people stop attending to a system generating too many false alarms. The fourth is the absence of a feedback loop: with no mechanism allowing an operator to flag a system error and feed it into a correction corpus, the model degrades without anyone measuring it.These four gaps share the property of not being algorithmic problems, which explains why they escape technical validation phases. The defence is a design that plans from the outset for continuous collection, periodic annotation of contested cases and retraining. An industrial vision system is not a product delivered once, it is a maintained capability.Sectors and their specifics
The defect detection principles set out here hold everywhere, but each sector shifts the difficulty to a particular point.Knowing where the difficulty concentrates in a given sector is what allows effort to be directed rather than spread. In metalworking and surface treatment, the challenge is the variability of normal appearance: gloss, texture and machining marks produce a very noisy background against which fine defects are hard to isolate. In electronics and assembly, difficulty moves towards logical defects, absence, inversion or substitution of a component, which demand an understanding of arrangement rather than local analysis. In textiles and flexible materials, deformation of the part between two acquisitions rules out template comparison and requires methods tolerant to geometry. In food processing, natural product variability is such that the boundary between acceptable variation and defect becomes a commercial decision as much as a technical one.This diversity explains why no generic solution installs itself without scoping work. What transfers from one project to the next is the method, not the model.The most common mistakes
These failures recur often enough across inspection projects that naming them is usually enough to avoid them.- Building the ontology on causes rather than on visible appearance.
- Defining severity through subjective qualifiers with no measurable criteria.
- Choosing the annotation primitive without starting from the quality decision rule.
- Extrapolating a published benchmark performance to your own conditions.
- Neglecting lighting design and relying on the algorithm to compensate.
- Reporting global accuracy on a heavily imbalanced corpus.
- Merging omissions and false detections into a single indicator.
- Providing no class for visible but uncatalogued anomalies.
- Delivering a system with no feedback loop or retraining plan.
