The ratio between available images and annotated ones is the most revealing figure in an industrial project. A reference dataset for industrial inspection illustrates it in its very construction: out of 18,000 images in total, 4,000 carry segmentation annotations covering 44 defect types, while the remaining 14,000 are released without labels to simulate the data-rich and annotation-rare scenario of real-world production.
That imbalance describes a factory exactly. Images are not scarce: an instrumented line produces thousands a day. What is scarce is a corpus, meaning a chosen, documented and annotated subset that allows a model to be trained and above all evaluated. Getting from one to the other is design work, not collection work.
This article covers that design: what to collect and under which strategy, how to handle confidentiality, how to choose which images to annotate and how to document the resulting corpus. It extends the complete guide to defect detection.
Why collection precedes defect detection
A defect detection corpus is designed backwards, starting from what it will need to demonstrate.
Two corpora, two logics
The most useful and most neglected distinction separates the training corpus from the evaluation corpus. The first exists to teach the model: it tolerates some noise, benefits from volume, and can be artificially enriched in rare cases. The second exists to measure: it must be representative of real production, annotated with greater care and frozen before any measurement.
That distinction has a counterintuitive consequence for collection. A training corpus benefits from being enriched in defects, an evaluation corpus must reflect real prevalence. Building both with the same sampling strategy means either having training data poor in defects, or obtaining a performance figure that says nothing about behaviour on the line.
What the corpus must cover
Four dimensions of variability must be represented in a defect detection corpus, and overlooking them is the main cause of gaps between pilot and production. Product variability, meaning references, shades, materials and batches. Process variability, meaning machine settings, shifts and speeds. Acquisition variability, meaning stations, lighting and its ageing. And temporal variability, since a collection over one week captures neither seasons nor exceptional campaigns.
That last dimension deserves emphasis. Industrial defects arrive in waves correlated with process incidents, and a short collection produces a corpus reflecting that period’s incidents rather than the real distribution of failure modes.
Collection strategies in defect detection
Three approaches coexist, and a mature project combines them rather than choosing.
Exhaustive collection
The first records everything that passes and sorts afterwards. It is the simplest to implement and the most expensive in storage. It has one decisive merit: it preserves the ability to build a representative evaluation corpus later, which no selective collection can reconstruct. On a high-throughput line it presupposes a retention policy and a storage architecture decided upfront.
Triggered collection
The second records only when an event occurs: rejection by existing inspection, operator alert, batch change, machine incident. This produces a corpus dense in interesting cases at modest storage cost. Its bias is obvious and must be documented: it captures only what the existing arrangement already detects, and structurally misses the defects nobody sees yet.
Enriched collection
The third deliberately produces defective parts by intentionally detuning the process. It is the only way to obtain examples of rare defects within a reasonable timeframe, and it is common practice. Its limit is that induced defects do not always resemble natural ones: they tend to be starker, more isolated and unaccompanied by the variations that surround them in production. These images suit training; they suit evaluation much less.
Choosing what to annotate: active sampling
Once images are collected, the question becomes which ones to annotate. This is the main saving available in a defect detection project, and it is well documented.
The principle is iterative. A small initial batch is annotated, a first model trained, and that model then selects subsequent images by their estimated learning value. A study on porosity detection on machined aluminium parts reports that the best-performing strategy, combining entropy-based uncertainty, spatial diversity and random selection, achieved an F1-score of 86.70 per cent and a recall of 82.99 per cent after ten iterations using only 2,400 annotated images.
The important point in that result is the composition of the winning strategy. Uncertainty alone concentrates effort on ambiguous cases but neglects areas where the model is confidently wrong. Diversity ensures coverage. The random component guarantees that a selection bias does not take hold. All three together outperform any one alone, which argues against single-criterion approaches.
One limit must nonetheless be understood. An analysis of the field notes that active learning reduces the required volume of labelled images, but can produce an incomplete ground truth by missing the annotations of defective parts classified as false negatives and not queried by the selection strategy.
In other words, the model selects what it already knows it does not know, and stays blind to what it does not know it is missing. Two countermeasures follow: retain a share of random sampling at every iteration, and build the evaluation corpus outside the active selection loop, by independent draw.
Confidentiality in industrial defect detection
This is the obstacle that most often blocks a defect detection project before its technical phase, and it is structural rather than accidental.
A survey of industrial datasets puts it plainly: in most cases manufacturers are highly confidential with their data, so there are few publicly available manufacturing datasets, with some hesitating to share due to privacy and security concerns or fear of giving competitors an advantage.
That reluctance is well founded, and the concern is not abstract. A production image reveals far more than a defect: part geometry and therefore design, tooling condition, throughput, sometimes a customer name through a marking. An annotation project must therefore treat confidentiality as a design constraint rather than a contractual clause.
What actually protects
Four technical measures cover most of the risk. Working in a controlled remote environment with no local copy, which removes the principal leak vector. Cropping to the region of interest, which avoids transmitting the whole part when only one zone is inspected. Removing or blurring markings, customer references and identifiers carried by the part. And anonymising production metadata, since dates and batch numbers can be replaced by working identifiers.
One frequently forgotten precaution applies: people. An image taken on a line sometimes captures hands, a badge, a reflection. Personal data regulation applies fully here, and the protocol must provide for excluding or processing those images.
Where the work happens
A practical question follows from those measures: where do annotators work. Three configurations exist, from most to least constraining. On-site annotation, in the client’s premises, offers maximum protection and immediate availability of domain expertise, at the cost of limited capacity and high expense. Remote access to a client-hosted environment combines security and capacity, and is the most common compromise. Transfer to the provider’s environment is the most flexible and presupposes established contractual trust and verifiable guarantees, an information security certification easing that demonstration considerably.
The reuse question
The most sensitive contractual point concerns what ultimately becomes of the corpus. Three regimes are possible and must be made explicit: use strictly limited to the project with deletion at completion, retention by the client alone, or authorisation of derived use by the provider. The third, which would let a provider improve its generic models, is legitimate but must be negotiated explicitly rather than assumed.
Qualifying images before annotating them
One intermediate step is regularly skipped in defect detection projects: quality triage of collected images. On a production flow, a non-negligible share of acquisitions is unusable for reasons unrelated to the defect sought.
Five causes recur: part mispositioned or out of frame, motion blur, saturation by a reflection, obstruction by conveyor hardware, and acquisition triggered on an empty station. Their proportion commonly ranges from a few per cent to several tens depending on installation maturity.
Two distinct reasons make this triage indispensable. The first is economic: annotating an unusable image costs the same as a useful one and yields nothing. The second is methodological: these images should form a documented class rather than being silently discarded, since their rate is a direct indicator of installation health and its drift signals a mechanical problem before it affects production.
Documenting a defect detection corpus
An undocumented corpus loses most of its value within months, because nobody knows any longer what it contains or what it does not.
Six elements constitute the minimum documentation, and none is expensive to produce as work proceeds. Origin, meaning lines, stations and collection periods. The sampling strategy used, with its acknowledged biases. Composition, broken down along the relevant variability dimensions. The inclusion and exclusion criteria applied. The annotation protocol in its applicable version, with change history. And the training, validation and test split, with the rule that produced it.
This documentation is not merely good practice. Without it, diagnosing underperformance becomes impossible: you cannot tell whether a model fails on a configuration because it is difficult or because it was absent from the corpus. Those two causes call for opposite responses.
The pre-existing inspection record
One asset is routinely overlooked when a project starts: the plant already inspects. Whether manually or through an older automated system, it has been producing conformity decisions, sometimes for years, and often recording them.
That record has three uses. It gives a real prevalence estimate per defect class, which is exactly the figure needed to size collection and to set expectations. It identifies which classes actually occur, letting the ontology be trimmed to what the line produces rather than to what a standard lists. And where images were kept alongside decisions, it provides a starting corpus at essentially no collection cost.
Two cautions apply. Historical decisions carry the judgement of whoever made them, including its inconsistencies, so they are a guide rather than a ground truth. And a record produced by an existing automated system inherits that system’s blind spots, which is precisely what a new project usually aims to fix. Used with those reservations, the existing record shortens the scoping phase considerably.
Splitting the datasets
One frequent technical error deserves separate treatment, because it silently invalidates any performance measurement.
The split between training and test must be made at the physical unit level rather than the image level. Several images of the same part, several views of the same defect, several acquisitions of the same coil distributed across both sets produce overstated performance that no later analysis corrects.
On continuous products the rule is stricter still: two neighbouring segments of one coil closely resemble each other, and the split must be by coil rather than by segment. Ideally the test corpus comes from batches or periods distinct from the training data, which brings the measurement closer to real deployment conditions.
Storage and the image lifecycle
One material constraint often determines what will be possible later, and it is settled at the outset.
A line fitted with several high-resolution cameras produces volumes that make full retention expensive beyond a few weeks. Retention policy is therefore a trade-off: keeping long allows a representative corpus to be reconstructed after the fact, keeping little reduces cost but closes that possibility permanently.
The usual compromise combines three retention durations rather than one. Short exhaustive retention, allowing time to select. Long sampled retention, drawn randomly at a fixed rate, preserving the ability to build a representative evaluation corpus. And permanent retention for annotated images and notable cases. This arrangement costs little and protects against the commonest regret in industrial projects, discovering six months later that an interesting period was not kept.
How many images
The question comes up in every defect detection project and has no general answer, but it has a method.
The volume needed depends mainly on three factors: the number of classes, the visual difficulty of each, and the breadth of normal variability to cover. A two-class task with sharp defects on a stable product is handled with a few hundred examples per class. A twenty-class task, several of them diffuse, on a highly variable product, needs an order of magnitude more.
The useful method is to measure rather than to estimate. A pilot batch annotated, then a model trained on increasing fractions of the corpus, produces a performance curve against volume whose flattening indicates the point beyond which further annotation yields little. That curve is built in a few days and usefully replaces rules of thumb.
One point deserves adding: the evaluation corpus follows a different logic. Its size depends on the statistical precision sought on the measurement, not on learning needs. Establishing a low residual defect rate with acceptable uncertainty requires a number of conforming parts well above what intuition suggests.
The role of synthetic data
Generating artificial images is frequently presented as the answer to a shortage of defects in industrial defect detection. It has a genuine place, provided its scope is understood.
The boundary is fairly clear in practice. It works well for increasing the appearance diversity of already observed defects: variations in position, orientation, scale, contrast. It works less well for creating defect types never seen, since that requires modelling a physical phenomenon that is not known. And it cannot under any circumstances serve for evaluation, a model measured on images it partly learned to generate measuring nothing.
The prudent rule is to use synthetic data as a complement within the training corpus, documenting the proportion and method precisely, and to keep the evaluation corpus entirely real.
Organising collection over time
Collection is not a phase of a defect detection project but a permanent function, and organising it as such changes a project’s trajectory.
Three distinct collection streams deserve establishing from the start. Initial collection, sized by the pilot batch and designed to cover known variability. Exception collection, triggered at every new failure mode, material change or setting change. And disagreement collection, fed by cases where the system and the operator diverge, which is the richest source of information once the system is live.
This arrangement presupposes minimal infrastructure: storage with a retention policy, a naming convention tying each image to its part, line, date and batch, and an annotation process able to absorb small frequent batches rather than only large one-off ones.
Sharing roles with the industrial client
A defect detection corpus project involves two parties whose competences do not overlap, and clarifying the division avoids most friction.
The industrial client holds what nobody else can supply: access to images, knowledge of the process, the quality reference and the ability to induce defects to enrich the corpus. They also hold the confidentiality constraint and the decision on what may leave the site.
The provider brings the sampling method, protocol drafting, annotation capacity and the control arrangement. Above all they bring an outside view of the categories, which has value in itself: quality teams have used a vocabulary for years in which some distinctions are not visually grounded, and making that explicit is often useful independently of the project.
The most frequent friction concerns rhythm. A plant thinks in production campaigns, an annotation project in weekly batches, and waiting for a particular material batch can idle a team. Anticipating this through a shared collection calendar, rather than discovering the constraint midway, is a simple and effective measure.
When to start annotating
A final question of sequencing decides more than it appears in a defect detection project: how much collection should precede the first annotation.
The instinct is to collect broadly first and annotate once the corpus looks complete. That instinct is wrong in both directions. Annotating too early means encoding conventions before the range of cases is understood, and reworking them later. Annotating too late means discovering only after months that a defect class the protocol assumed to exist barely occurs, or that a critical one was never captured.
The workable sequence starts annotating a small, deliberately diverse batch as soon as a few weeks of collection exist. That batch is not primarily training data, it is reconnaissance: it establishes what the images actually contain, exposes the ambiguities in the draft ontology and produces the timing measurements a budget needs. Collection then continues in parallel, informed by what the first batch revealed rather than by assumptions made before any image was examined.
The most common mistakes
These failures recur often enough across industrial corpus projects that naming them is usually enough to avoid them.
- Building training and evaluation corpora with the same sampling strategy.
- Collecting over a short period and missing temporal variability.
- Splitting datasets at image level rather than part or batch level.
- Relying solely on triggered collection and inheriting the existing system’s bias.
- Using induced defects in the evaluation corpus.
- Using single-criterion active selection and creating a blind spot.
- Building the evaluation corpus inside the active selection loop.
- Treating confidentiality as a contractual clause rather than a design constraint.
- Not documenting composition and sampling, making any diagnosis impossible.
- Using synthetic images to measure performance.
What to take away
Building a defect detection corpus is not a collection exercise but a design one. Images abound; what is missing is a set whose composition, biases and limits are known, and which supports measurement as much as learning.
Three decisions structure a usable corpus. Separate from the outset the logic of the training corpus, enriched in defects and tolerant, from that of the evaluation corpus, representative and frozen. Use active selection combining uncertainty, diversity and randomness to reduce annotated volume, while keeping evaluation outside that loop. And document origin, sampling and composition from the first batch, since that is the only genuinely irrecoverable information.
For approaches, ontology and annotation tasks, the complete guide to defect detection sets the frame. For the profile of the people who will annotate that corpus, and the product knowledge the task genuinely requires, the article on annotators trained in the trade covers the required skills and how to organise iterations with technical teams.
To explore delivery arrangements, supported formats and applicable control mechanisms, see our dedicated page on annotation for industry. And if you hold production images and want the corpus construction strategy scoped before annotation starts, let us discuss your project.
