A segmentation corpus is not sized in number of images. What determines what a system will be able to do is the range of conditions the corpus covers, and that range is decided before the first annotation.
This article sets out that design. It extends the article on segmenting small objects.
What defines a corpus
Five decisions characterise an image segmentation corpus. The formulation adopted, which determines what the corpus permits to be learned. The nomenclature, whose granularity fixes the fineness of the result. The coverage, which bounds the system’s domain of use. The tracing conventions, which determine internal consistency. And the separation between the sets, which alone conditions the validity of any measurement. One important practical consequence follows. Those five decisions precede the first image, and none corrects without reworking a substantial share of the corpus already produced.The axes of coverage
Five axes structure the diversity of an image segmentation corpus. Acquisition conditions, lighting, weather and settings. Scene configurations, density, clutter and occlusion. Scales, from the close view to the distant one. Sites or sources, one type of scene varying appreciably by place. And rare configurations, precisely those where the system will fail. One important observation follows for an image segmentation project. The last axis is the most decisive and the hardest to cover, failures bearing precisely on what the corpus has never presented.Why volume does not suffice
Three observations explain how a large image segmentation corpus can be weak. Images from one session resemble each other to the point of adding little. A natural distribution under-represents precisely the difficult cases. And a large volume on a narrow domain prepares for nothing outside it. One practical consequence follows. Those three observations mean a corpus is sized by its coverage rather than by its count, a limited but diverse batch being worth more than a large homogeneous one.The separation of the sets
One rule governs the validity of any measurement in image segmentation. Three principles compose that rule in image segmentation. Separation is by source, site or session, never by image. Two images from one acquisition must not end up on either side. And the evaluation set’s composition is decided by what it must measure, never by random draw. One important observation follows. The first principle is the most violated, a random split by image producing an overstated performance no deployment will confirm.What the evaluation set requires
Four differences distinguish it from the image segmentation training set. Exhaustiveness becomes imperative, an omitted region counting as a false detection. Double annotation is required on the classes with high variability. The composition deliberately covers the difficult cases rather than the natural distribution. And stability over time conditions any comparison between two successive versions. One important practical consequence follows for an image segmentation project. The third difference is the most misunderstood, a set representative of how often situations occur measuring mostly the easy and leaving invisible what will cause trouble.Designing the nomenclature
Five principles govern this design in image segmentation. Taking an existing reference where the domain has one. Defining each class by a criterion observable in the image. Verifying that two neighbouring classes are distinguishable at the available resolution. Providing a class for the indeterminate, giving it an explicit meaning. And testing the complete list on real project images before freezing it. One observation follows. The third principle avoids the most expensive defect, a distinction impossible to make producing a variability neither training nor control will reduce.The sampling plan
A method replaces the random draw with a decided composition. Four stages compose that plan in image segmentation. Listing the coverage axes relevant to the intended domain of use. Fixing a target proportion per combination of axes, relying on the use rather than on what happened to be collected. Selecting the images to respect those proportions rather than drawing from the archive. And verifying the composition actually obtained before launching any annotation. One important practical consequence follows. The second stage is what distinguishes, a proportion based on natural frequency reproducing the imbalance of the terrain while a proportion based on the use deliberately over-represents the difficult conditions.Building progressively
Four stages organise the progressive construction of an image segmentation corpus. A pilot batch that tests the conventions adopted and measures the real throughput. A first production batch covering all the principal conditions. An intermediate evaluation that identifies precisely the system’s weaknesses. And enrichment batches targeted precisely on those weaknesses. One important practical consequence follows. The fourth stage is far more effective than a uniform increase, a batch targeted on the failures observed improving more than an equivalent volume drawn at random.Recovering rare configurations
Three methods permit reaching what ordinary collection does not encounter. Prolonged collection, which eventually crosses them at the price of considerable time. Staging, practicable for some situations and impossible for others. And querying an existing model, whose uncertain detections locate the atypical scenes within an archive already collected. One important observation follows for an image segmentation project. The third method is the most effective and the least used, an unannotated archive already containing the cases a further collection would go looking for, which shifts the problem towards selection.The enrichment cycle
Four stages organise the improvement of a corpus after first deployment. Recording the cases where the system fails, which presupposes a collection arrangement planned upfront. Classing those failures by cause, uncovered condition, ambiguous class or misapplied convention. The targeted selection of images showing precisely the dominant cause. And annotating that new batch with the existing conventions rather than with new rules. One important practical consequence follows for an image segmentation project. The second stage avoids this cycle’s commonest error, a failure attributed to a lack of data when it comes from a badly defined class, which leads to annotating more without correcting the cause.What merging corpora permits
One practice widens coverage without annotating more. An existing corpus, public or from another project, completes an insufficient coverage. Four conditions make that merge possible in image segmentation. Identical formulations, a semantic corpus and an instance corpus not merging directly. Class tables reconciled, identifier by identifier. Compatible placement conventions, which is verified on the area distributions. And homogeneous size thresholds between the two, failing which part of the corpus ignores what the other annotates. One observation follows. The third condition is the hardest to verify and the most decisive, a systematic shift between two merged corpora introducing an inconsistency the model will learn as noise.Measuring coverage
Four indicators qualify the real diversity of an image segmentation corpus. The distribution by acquisition condition, whose imbalance announces a targeted failure. The number of distinct sources or sites, which bounds generalisation. The counts per class, which reveal the categories too rare to be learned. And the proportion of difficult configurations, whether deliberately over-represented or not. One practical consequence follows. The second indicator predicts generalisation better than the image count, a large corpus from a single site preparing less well than a modest one from several.What the corpus must declare
Six pieces of information accompany a delivered image segmentation corpus. The formulation and the nomenclature with their definitions. The tracing conventions, placement rule and size threshold. The distribution by condition and by site. The method of separation between the sets. The disagreement measured between annotators. And the explicit list of configurations not covered. One practical consequence follows. The last is the rarest and the most useful, it explicitly bounds the domain of use and prevents a system being deployed where nothing prepared it.The corpus as an asset
Five properties determine whether an image segmentation corpus gets reused. Retention of the source images at original resolution. Retention of the vector representation where it exists. Written and dated conventions, which make each batch interpretable. Condition and source metadata, without which no coverage analysis is conceivable. And the class table frozen from the outset, which makes any later merge possible. One important observation follows. The fourth property is the most often omitted and the one conditioning enrichment, a corpus with no condition metadata not permitting anyone to identify what it lacks.What makes a corpus age
Four developments degrade an image segmentation corpus’s value over time. A change of sensor, which alters the appearance without changing the objects. An evolution of the terrain, new equipment or new practices. A change of use, which demands a granularity the corpus does not have. And an evolution of the reference nomenclature adopted in the domain. One observation follows. Those four developments are handled by enrichment where the sources and conventions have been kept, and by reconstruction otherwise.What the first corpus should aim for
Four principles govern a first image segmentation corpus. Covering broadly rather than deeply, diversity mattering more than the count per condition. Adopting the simplest nomenclature that serves the use, enrichment staying possible. Building the evaluation set from the start, since it conditions every later measurement. And keeping the sources and associated metadata, which make enrichment practicable. One observation follows. The first principle contradicts instinct, a client naturally preferring a large volume on the conditions they have mastered while the value sits in the conditions they have not yet.What distinguishes a useful corpus
Five properties separate a corpus that serves from one that exists. It covers the domain of use genuinely intended rather than the one that was simply accessible. It deliberately over-represents the conditions in which the system will fail. Its separation between the sets permits a measurement deployment will confirm. Its conventions are written, which makes each batch interpretable and mergeable. And its enrichment had been planned before the first delivery. One observation follows for an image segmentation project. Those five properties all belong to design and none to execution, which explains how a carefully annotated corpus can prepare for nothing useful.The public corpora and their use
Four observations situate these resources in image segmentation. They supply a large volume on common and generally well documented classes. Their conventions differ from yours, which forbids any merge without prior verification. Their coverage reflects a domain of use that is not yours. And their licence directly conditions the commercial use of the trained system. One practical consequence follows. The third observation is the most decisive, a public corpus usefully serving as a general base and never dispensing with a corpus of your own covering the real operating conditions.What the pilot batch brings to the design
Four pieces of information come out of a pilot batch and serve design rather than production. The feasibility of the nomenclature, the cases of hesitation revealing the badly separated classes. The disagreement between annotators, which bounds the contractual requirement. The throughput per contour type, which turns a sampling plan into a budget. And the list of configurations not anticipated, which enriches the reference before production begins. One observation follows for an image segmentation project. The fourth is the most valuable and the least expected, a few real images bringing out situations no scoping meeting would have envisaged.What the provider brings to the design
Five contributions go beyond production in image segmentation. Listing the coverage axes relevant to the intended domain. Verifying the nomenclature is workable at the available resolution. Measuring disagreement between annotators, which bounds the contractual requirement. Designing the sampling plan from the domain of use. And organising the enrichment, planned well before the first delivery. One commercial consequence follows. The second contribution is the most distinguishing, a client frequently discovering that two of their classes are not distinguishable at the resolution of their images, information no volume-based quote would have brought them.Three checks on an existing corpus
Three checks qualify an image segmentation corpus before exploiting it. The distribution by source and by condition, which reveals a narrow coverage no count compensates. The method of separation between the sets, which decides whether the published measurements mean anything. And the counts per class, which identify the categories far too rare to be learned. Those three checks run on the metadata of a delivered corpus, they require no inspection of any annotation, and they say in minutes what the corpus actually prepares for.What design costs and returns
Four observations situate the design effort in image segmentation. It takes a few days against months of production. It determines what the corpus will permit, which execution can no longer correct. It avoids the most expensive corrections, the ones requiring all the existing work to be reworked. And it produces upfront the documents later exploitation will require anyway. One important practical consequence follows. The second observation sums up the chapter, faultless execution on a badly designed corpus producing a careful result that does not serve, while correct execution on a well designed corpus produces a usable one.Approaching the design of a corpus
Five questions scope the design of an image segmentation corpus. What domain of use must the system cover. That answer determines the diversity required. What formulation does the use require. It bounds what the corpus will permit. Is the separation by source. A separation by image invalidates the measurement. Does the evaluation set cover the difficult cases. Otherwise it measures the easy. And is enrichment planned from the design stage. Its absence fixes the corpus at its first scope. Those five answers determine what the corpus will permit. Asking them before starting avoids a large corpus that prepares only for the situations already mastered.The question that sizes the corpus
One question determines the size and composition better than a target figure. In what conditions will the system have to operate unsupervised. A restrictive answer, a single site and controlled conditions, permits a modest corpus covering that narrow domain solidly. An open answer, several sites and every condition, requires a campaign where diversity takes precedence over volume and which deliberately covers the rare cases. That question belongs to the product definition and not to annotation, it is asked of whoever owns the system rather than of whoever buys the corpus, and it predicts the budget better than the number of images requested.Three decisions before the first image
Three decisions commit what an image segmentation corpus will permit. Settling the intended domain of use, which determines the diversity required more than the volume does. Fixing the method of separation by source, since a separation by image invalidates every later measurement. And building the evaluation set covering the difficult cases deliberately, before production absorbs it. Those three decisions cost one meeting, they precede the first image, and their absence produces a large corpus nobody can establish what it actually prepares for.What this chapter leaves to the next
The decisions set out here presuppose a stable context. Three developments call it into question. Assistance advances and alters the split between production and verification. Generalist models reduce the volume needed on common classes. And uses shift towards tasks segmentation alone no longer suffices to describe. Those three developments change what an image segmentation corpus will have to contain, and they constitute the subject of the following chapter, which closes this series.What this chapter teaches
One cross-cutting observation deserves closing this examination. A corpus is designed by its coverage rather than by its volume. Three findings compose it. Failures bear on what the corpus has never presented, which makes rare configurations the decisive axis. Separation by source is the condition of any valid measurement, a separation by image overstating the performance. And enrichment targeted on the failures actually observed improves more than a uniform increase. That finding extends the preceding chapters, design weighing more than execution on what the corpus will permit.Why this chapter changes the sales conversation
One shift in framing follows from everything above. A client who arrives asking for a number of images is asking the wrong question. Three things follow from saying so in an image segmentation engagement. The conversation moves from unit price to what the corpus must enable, where the provider has something to contribute. The design contributions become visible as work rather than as free advice attached to a volume quote. And the client leaves with a better project even if they buy elsewhere, which is the position from which repeat business tends to come. One practical consequence follows for an image segmentation provider. Most competitors will answer the volume question as asked, so answering the design question instead is a cheap way to be the only party in the room who discussed what the client actually needs.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Sizing a corpus in number of images.
- Separating the sets by image rather than by source.
- Building an evaluation set representative of natural frequency.
- Omitting rare configurations from the collection plan.
- Freezing a nomenclature without testing it on real images.
- Creating two classes indistinguishable at the available resolution.
- Increasing the corpus uniformly rather than targeting the failures.
- Neglecting to declare the configurations not covered.
- Assuming a large volume compensates for narrow coverage.
- Not planning the enrichment from the design stage.
