Assigning a class to every pixel looks like a question of precision. It is in fact a question of definition: what costs is not following a contour, but knowing where that contour runs when the material itself does not say.
This article sets out what this family of tasks covers, what separates it from simpler geometries, and the decisions that determine its cost.
What the term covers
Three distinct tasks carry the name image segmentation. Their confusion produces expensive misunderstandings. Semantic segmentation assigns a class to every pixel without distinguishing individuals. Instance segmentation separates objects of the same class from one another. And panoptic segmentation combines the two, handling countable objects and continuous materials separately. One important practical consequence follows. Those three formulations have neither the same cost nor the same usefulness, and an image segmentation project asking simply for segmentation leaves open the most decisive question of its scoping.What separates this task from other geometries
Four properties separate image segmentation from boxes and points. The deliverable is a surface rather than a frame, which describes the object’s real shape. Every pixel receives a decision, which multiplies the number of judgements by several orders of magnitude. The boundary between two regions concentrates most of the difficulty. And exhaustiveness becomes the rule, no zone of the image being allowed to stay without an assigned class. One important observation follows. The last property is the most structuring, a box permitting what does not interest you to be ignored while image segmentation requires the whole surface to be handled.Objects and materials
This distinction governs the whole family. It precedes the choice of formulation in image segmentation. Some categories can be counted and others merely occupy a surface. Three differences separate them. A countable object has its own boundaries, a continuous material does not necessarily. Two neighbouring objects must be separated, two zones of the same material need not be. And the question of number has meaning for the former and none for the latter. One practical consequence follows for an image segmentation project. That distinction determines which formulation suits, instance segmentation having no meaning on a continuous material and semantic segmentation losing the counting information on objects.What the boundary costs
One observation contradicts intuition about the cost of image segmentation. The work concentrates on contours and not on surfaces. Four consequences follow. A large homogeneous zone is handled almost as fast as a small one. A fragmented object costs far more than a compact one of the same area. Contour complexity predicts the load better than the pixel count. And disagreement between annotators concentrates almost entirely on the boundaries. One important observation follows. Those four consequences make the cumulative length of the boundaries the real unit of work, which makes costing by pixel count or object count systematically wrong in image segmentation.The boundaries that do not exist
This difficulty dominates image segmentation. It is irreducible. Some transitions have no objective limit in the image. Four situations illustrate it in image segmentation. Focus or motion blur, which spreads the transition over several pixels. Translucent or partly transparent objects. Materials that merge into one another, vegetation and ground notably. And fine structures whose thickness is below the image’s resolution. One important practical consequence follows. Those four situations call for a convention rather than additional effort, the precision requested exceeding the information the image contains.What the deliverable can be
Three representations coexist in image segmentation. They do not carry the same information. The mask, an image where every pixel carries a class identifier. The polygon, a sequence of vertices describing a contour. And the compact encoding, which describes the regions without storing the value of every pixel. One observation follows for an image segmentation project. The polygon is easily modified and loses fine detail, the mask keeps everything and is hard to correct, which makes this choice a life-cycle decision rather than a question of format.The uses that justify it
Five needs genuinely call for image segmentation. Area measurement, which no box supplies. Shape, where the object’s geometry carries the useful information. Contact between regions, which a box does not describe. Separating nested objects, whose frames overlap entirely. And complete scene understanding, where no zone should stay undefined for the use. One practical consequence follows. Those five needs do not include detection or counting, tasks for which a box suffices and costs a fraction of the price in image segmentation.The life cycle of a project
Six stages structure a full image segmentation project. Scoping, which determines the formulation, the precision and the class scope. Writing the conventions, in particular those handling uncertain boundaries. The pilot batch, which measures the real throughput and reveals the uncovered cases. Production, organised in batches with the conventions corrected between each. Control, distinct from production and concentrated on the boundaries. And documentation, which declares the conventions used and the corpus’s limits. One important practical consequence follows. The third stage is the one most often compressed and the only one producing a defensible costing, throughput in image segmentation varying by a large factor with the complexity of the contours encountered.The conventions to write
Six decisions cover most of the disagreements in image segmentation. They are written before production. The treatment of blurred transitions, with an explicit rule for placing the boundary. What to do with occluded zones, a region interrupted or reconstituted. The size threshold below which an object is not segmented. The treatment of reflections and shadows, attached to the object or to the background. The class receiving the zones the nomenclature does not cover. And the precision expected, expressed as a contour tolerance rather than as an intention. One important observation follows. The first decision is the most decisive and the most often omitted, a contour placed at the start, the middle or the end of a transition producing appreciably different areas.What the nomenclature determines
One scoping decision weighs as much as the choice of formulation. Four properties of a nomenclature affect cost in image segmentation. The number of classes, which multiplies the boundary decisions between neighbouring categories. Granularity, a fine category multiplying the cases of hesitation. The presence of a catch-all class, which absorbs what the others do not cover. And the existence of classes that frequently touch, each contact producing a boundary to trace. One important observation follows. The last property is the least anticipated, two rarely adjacent classes costing far less than two classes that abut everywhere, which makes the adjacency matrix more predictive of cost than the class count itself.What the required precision changes
Four levels of requirement occur in image segmentation. Their cost varies considerably. The coarse contour, sufficient where only position counts. The fitted contour, the standard for most projects. The pixel-level contour, which follows the slightest irregularity. And the sub-pixel contour, which describes a progressive transition. One practical consequence follows for an image segmentation project. Moving from the second to the third level produces the largest cost variation, and it is justified only where the downstream use genuinely exploits that fineness.What resolution changes
One property of the image directly affects feasibility in image segmentation. Four effects follow in image segmentation. A low resolution makes some boundaries undeterminable, the object and its neighbour sharing the same transition pixels. A high resolution increases the length of contour to follow, hence the load. Fine structures appear only above a certain threshold, below which they are absent rather than imprecise. And consistency between images of different resolutions becomes problematic, one object not calling for the same level of detail. One important observation follows. The last effect requires treating resolution as corpus metadata, an undeclared mix producing a precision requirement that varies with nothing to explain it.Quality control
Five verifications apply to an image segmentation corpus. Exhaustiveness, no pixel being allowed to stay without a class. Boundary consistency, two neighbouring regions having to touch with no overlap and no gap. Area plausibility per class, an unusual proportion signalling an error. Detection of isolated regions, islands of a few pixels signalling imprecision. And comparison between annotators, restricted to the boundary zones alone. One important observation follows. The last verification is the only representative one, global agreement on an image being dominated by the large easy surfaces and able to coexist with massive disagreement where the point of the corpus lies.What the model learns from a mask
Three observations link the quality of an image segmentation corpus to the behaviour of the system trained. A contour systematically placed inside the object produces a model that underestimates areas. An unstable convention between annotators produces a model imprecise at boundaries rather than an averaged model. And a badly defined catch-all class teaches the model to pour into it everything it does not recognise. One practical consequence follows for an image segmentation project. The first observation is the most measurable, a placement bias being detectable by comparing the mean areas per class between two annotators, which makes it a simple and rarely practised check.What automates
Four levels of assistance occur in image segmentation. Their return differs strongly. Selection by similarity, which extends a zone on a colour or texture criterion. The active contour, which adjusts a roughly traced boundary. Mask proposal by designation, a click producing a whole region. And full pre-annotation by a model, which proposes both classes and boundaries. One practical consequence follows for an image segmentation project. The third level has advanced considerably and it shifts the work towards verification, which changes the nature of control more than the volume of production.The pilot batch
Four images compose an image segmentation pilot batch that genuinely tests the conventions. A simple image with sharp contours, which supplies the reference throughput. An image with fragmented contours, vegetation or fine structures, which measures the dominant cost factor. An image containing blurred transitions, which tests the contour placement rule. And an image with objects in contact, which tests the formulation adopted on instance separation. One observation follows. The throughput gap between the first two images is this batch’s most useful figure, it directly measures the effect of contour complexity and it supplies the basis for a costing the image count does not permit in image segmentation.What this work costs
Five factors determine the load of an image segmentation project. Contour complexity, the dominant factor and poorly correlated with object count. The precision required, from the coarse contour to the pixel-level one. The number of adjacent classes, which multiplies the boundaries to decide. The exhaustiveness requirement, which forbids ignoring the zones of no interest. And the formulation adopted, instance separation adding a cost only in the contact zones. One observation follows. The first factor produces a considerable variation, an urban scene and an image of isolated objects on a plain background not comparing even at equal object counts.The verticals and their requirements
Four contexts use image segmentation with appreciably different expectations. Medical, where the boundary carries a clinical consequence and where inter-expert variability is the limit. Geospatial, where objects are seen from above and where continuous materials dominate. Autonomous driving, where scene exhaustiveness counts as much as object precision. And industry, where a defect’s area is the measured quantity and where the tolerance is stated numerically. One practical consequence follows. Those four contexts call for different conventions on the same technical gesture, which makes a generic image segmentation reference inapplicable without adaptation to the domain.What the corpus must carry
Six pieces of information accompany an image segmentation delivery and condition its reuse. The class table, with the identifier used for each in the masks. The formulation adopted, semantic, instance or panoptic. The contour placement rule in uncertain transitions. The size threshold below which an object was not segmented. The treatment applied to indeterminate zones and the identifier carrying them. And the disagreement measurement between annotators, restricted to the boundaries. One practical consequence follows. The third is the one whose absence is paid latest, two corpora using different placement rules not being combinable while nothing in the masks signals it.What makes a corpus reusable
Four properties determine whether an image segmentation corpus serves beyond its initial project. Retention of the source images at their original resolution. Declaration of the formulation and of the contour placement rule. A clear separation between the catch-all class and the genuinely unannotated zones. And maintenance of the correspondence between mask and image, which a resampling breaks silently. One observation follows. The last property produces the most discreet failure, a mask resized independently of its image introducing an offset of a few pixels nobody notices before training.Approaching a segmentation project
Five questions scope an image segmentation project. Must the objects be counted. That answer settles between semantic and instance. Must the whole image be covered. That answer determines exhaustiveness. What contour precision does the use require. That answer is the principal cost lever. What blurred transitions does the data contain. That answer calls for a convention. And would a box suffice. That question deserves asking before any other. Those five answers determine feasibility and budget. Asking them before starting avoids committing to a spend a simpler geometry would have spared.The question that precedes all the others
One question determines whether this family of tasks suits the need. Does the downstream use exploit shape or only position. A position-oriented answer, detection, counting or tracking, makes the box sufficient and divides the budget by a large factor. A shape-oriented answer, area measurement, contact between regions or object geometry, justifies image segmentation and its cost. That question is asked in one sentence, it belongs to the use and not to the technique, and it avoids this field’s commonest expense, a mask produced where a frame would have served.Three decisions before producing
Three decisions commit the rest of an image segmentation project. Choosing the formulation, semantic, instance or panoptic, starting from the counting question rather than from a technical preference. Fixing the contour precision expected and checking it against a downstream use that genuinely exploits it. And writing the contour placement rule for uncertain transitions, before each annotator adopts a different one. Those three decisions cost one meeting, they precede the first image, and their absence produces a careful corpus whose areas vary with whoever traced them.What this cluster will cover
Fourteen chapters extend this introduction and they fall into four sets. The formulations, semantic against instance then panoptic, which refine the choice set out above. The method, tracing masks, model assistance and handling fine structures. The verticals, medical, satellite and autonomous driving, where the same gestures serve different requirements. And the economics, cost, quality, formats, mistakes and corpus construction. One observation follows. Those four sets all return to the same property, the boundary being at once the unit of cost, the site of disagreement and the object of control in image segmentation.What this chapter teaches
One cross-cutting observation deserves closing this introduction. The cost of this task lies in the boundaries and not in the surfaces. Three findings compose it. The cumulative length of the contours constitutes the real unit of work, which makes costing by object or pixel count systematically wrong. Part of the boundaries does not exist in the image, which shifts the question of precision towards convention. And exhaustiveness distinguishes this family from all the others, a box permitting what does not interest you to be ignored while a mask requires everything to be handled. That finding orients the chapters that follow, each of which returns to one of those three properties.Why the scoping conversation matters more here
One observation about how these projects go wrong belongs at the close. Image segmentation is the geometry clients most often over-specify. Three reasons produce that in image segmentation. A mask looks more thorough than a box, which makes it the safer-seeming request even where nothing downstream consumes the extra information. The cost difference is not visible from the request, since asking for a mask takes the same breath as asking for a box. And the precision level is rarely stated, so a provider quoting conservatively prices the strictest reading of an unstated requirement. One practical consequence follows for an image segmentation provider. Asking what the downstream system consumes, before quoting, frequently reduces the scope and always improves the quote’s credibility, since a provider who has understood the use is quoting a project rather than a volume.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Requesting segmentation where a box would have sufficed.
- Not specifying whether instances must be separated.
- Costing by pixel or object count rather than by boundary length.
- Omitting the contour placement rule in a blurred transition.
- Requiring pixel-level precision with no downstream use that exploits it.
- Neglecting the treatment of reflections and shadows.
- Choosing a delivery format without considering future corrections.
- Measuring annotator agreement globally rather than at the boundaries.
- Omitting the class for indeterminate zones.
- Confusing semantic and instance segmentation at scoping.
