Generalist segmentation models have changed the gesture without changing the problem. They produce a region from a click, which shifts the work towards verification, and that shift alters the nature of control far more than the volume of production.
This article sets out what they change. It extends the article on segmentation masks.
What these models do
One principle separates these models from specialised image segmentation systems. They delimit a region from an indication without knowing what it contains. Three properties follow in image segmentation. They work on categories they have never encountered. They produce no class, the designation staying with the annotator. And they respond to several kinds of indication, whether a point, a rough box or an approximate trace. One important practical consequence follows. The second property determines their use in image segmentation, they accelerate the tracing and leave the category decision entire, which remains where the substantive errors concentrate.Where they work well
Four image segmentation situations suit them. A compact object clearly detached from its background. Sharp contrast between the object and its immediate surroundings. A medium size, neither tiny nor filling the whole image. And a closed shape, with no openwork structure or fine cut. One important observation follows. Those four situations describe precisely the objects manual tracing already handled fastest, which means the gain concentrates where the need was weakest in image segmentation.Where they fail
Five configurations defeat them in image segmentation. Progressive transitions, where the model places a sharp boundary on a zone that has none. Openwork structures, whose voids it fills in. Objects of the same appearance in contact, which it fuses or separates quite arbitrarily. Continuous materials, which have no contour at all to designate. And categories whose domain definition does not coincide with a visual boundary. One practical consequence follows for an image segmentation project. The last configuration is the most problematic, a model delimiting what is visible while the reference asks for what is defined, a gap nothing in the result signals.The gap between contrast and convention
This difficulty is specific to assistance in image segmentation. It deserves separate treatment. The boundary proposed follows the contrast, the boundary expected follows a rule. Four situations produce that gap in image segmentation. A placement rule at the middle of a transition, while the model picks the maximum contrast. A cast shadow attached to the object by convention and excluded by the model. A reflection included by the model and excluded by the rule. And an object whose definition includes a visually distinct part, such as a tool’s handle or a sign’s post. One important observation follows. Those four gaps produce contours that are sharp and wrong, a category of defect that escapes any rapid visual check in image segmentation.The problem of scale
An ambiguity specific to these models deserves setting out. One indication can designate several nested levels of division. Three cases illustrate that ambiguity in image segmentation. A click on a window can designate the pane, the frame or the building. A click on a leaf can designate the leaf, the branch or the tree. And a click on a wheel can designate the tyre, the wheel or the vehicle. One important practical consequence follows. Those three cases show the model ignores the reference’s granularity, which makes constraining by a box not more comfortable but necessary as soon as the nomenclature holds nested categories.The modes of indication
Four ways of indicating a region exist and they are not equivalent. The single point, fast and ambiguous, a click on a wheel being able to designate the wheel or the whole vehicle. Positive and negative points, which lift that ambiguity at the price of a few extra clicks. The rough bounding box, which constrains the region and gives the best ratio of effort to reliability. And the approximate trace, which guides the boundary without fixing it precisely. One important practical consequence follows for an image segmentation project. The first mode produces most of the scale errors, an unconstrained model choosing the level of division it finds most salient rather than the one the reference asks for.What validation bias becomes
This known mechanism takes a particular form here. Accepting a proposal takes less effort than tracing a region. Three manifestations follow in image segmentation. A sharp but convention-shifted contour is validated with no examination. A fine structure omitted by the model is not added, its absence not leaping out. And a boundary placed on the contrast becomes the corpus’s de facto norm. One important practical consequence follows. The third manifestation is cumulative, a fully assisted corpus ending up encoding the model’s convention rather than the project’s, which makes a share of unassisted batches indispensable.What control must change
Five adaptations are required where assistance produces most of the regions. Verifying conformity with the placement rule, distinct from verifying sharpness. Searching for omitted fine structures, invisible on an otherwise correct contour. Checking the openwork zones, which the model systematically fills. Comparing mean areas with an unassisted batch, revealing a systematic shift. And tracking the correction rate, the only measure distinguishing help from hindrance. One observation follows for an image segmentation project. The fourth adaptation is the most effective, an area gap between assisted and unassisted batches revealing in one calculation what visual examination does not show.The shift is measured then corrected
One operation turns a systematic defect into a uniform correction. Four stages compose that operation in image segmentation. Manually annotating a sample of regions according to the project’s convention. Producing the same regions by assisted designation. Measuring the mean gap in area and in boundary position, class by class. And applying a uniform correction where that gap proves constant. One important observation follows. The fourth stage is possible only where the gap is constant, which the third verifies, a variable shift belonging to a convention problem rather than to a model bias.How to use it correctly
Five principles govern effective use of this assistance. Reserving designation for the classes and shapes where it works, measured on a pilot batch. Keeping manual tracing for openwork structures and soft transitions. Verifying conformity with the convention rather than the sharpness of the contour. Correcting the shift systematically where it is known and constant. And maintaining a share of fully manual batches throughout the whole project. One practical consequence follows. The fourth principle is the most worthwhile and the least practised, a systematic shift being corrected by a uniform treatment rather than region by region in image segmentation.What the annotator does differently
Five gestures distinguish an annotator working with this assistance in image segmentation. Constraining the designation with a box rather than an isolated point. Checking the boundary against the project’s rule and not against the appearance. Spotting filled voids in openwork structures, the characteristic defect of this assistance. Discarding a whole proposal rather than correcting a largely wrong contour. And switching to manual tracing as soon as the class being handled is among those where assistance fails. One practical consequence follows. The fourth gesture is the least natural, the temptation to rescue a mediocre proposal often consuming more time than a fresh trace.What the gain really becomes
Four observations qualify the announcement of a general gain in image segmentation. The gain is large on compact contrasted objects. It is nil on continuous materials, which often represent most of the surface. It is negative on openwork structures, correction costing more than tracing. And it shrinks in proportion as the verification becomes more demanding. One important observation follows. Those four effects mean the real gain is measured per class rather than globally, a single figure announced on a mixed corpus being systematically higher than the gain observed.Measuring the real gain
Five indicators quantify the real contribution of this assistance in image segmentation. Time per region with and without assistance, recorded class by class. The rate of proposals accepted without modification. The rate of proposals discarded entirely, revealing a badly suited class. The mean area gap between assisted and manual batches. And verification time, recorded distinctly from production time. One observation follows. The third indicator is the most useful for steering, a high rejection rate on a class signalling it should be switched to manual tracing rather than subjected to assistance that slows it.What assistance changes in the costing
Four effects modify an image segmentation project estimate. Time per region falls on the classes that lend themselves to it. Verification time rises, conformity with the convention becoming a check in its own right. The background’s share of the budget grows, since only the object part accelerates. And the setup cost stays low, these models requiring no prior corpus at all. One important practical consequence follows for an image segmentation project. The third effect surprises at costing, an assisted project seeing its load redistribute towards continuous materials without the total falling by the announced proportion.The pilot batch that qualifies the assistance
Four images suffice to decide where to use this assistance in image segmentation. An image with compact contrasted objects, which measures the maximum gain achievable. An image containing an openwork structure, which checks the behaviour on voids. An image with soft transitions, which measures the gap between contrast and convention. And an image dominated by continuous materials, which shows the share of the work assistance does not touch. One observation follows. The fourth image is the one most often omitted and the one that corrects the estimate, a pilot batch composed of clean objects announcing a gain the real corpus will never produce.What it does not replace
Four decisions stay entire despite this assistance. Assigning the class, which the model does not produce. The placement rule, which the model ignores. Separating identical objects in contact, where it decides arbitrarily. And handling fine structures, which it either omits or thickens. That first decision deserves emphasis. It means a corpus’s most expensive substantive error, category confusion, stays exactly where it was before this assistance.What these models will not replace
Three limits belong to the nature of these models rather than to their maturity. They delimit what stands out visually, while a reference defines what matters for a use. They ignore the granularity expected, several nested divisions being equally valid from their point of view. And they have access to no placement rule at all, the convention being external to the image. One observation follows for an image segmentation project. Those three limits are lifted by no gain in performance, since they bear on information the image does not contain, which distinguishes them from the defects a following version will correct.What this assistance changes in the delivered corpus
Three consequences extend beyond production and deserve declaring. The boundaries carry the signature of the model used, which makes two batches produced with different tools subtly inconsistent. The distribution of areas tightens, assistance producing more regular contours than the human hand. And a model trained on this corpus learns the assisting model’s convention as much as the project’s own. One important practical consequence follows for an image segmentation project. The first consequence justifies declaring the assistance tool used in the corpus documentation, information nobody records and which nonetheless explains the gaps between batches.The complete assisted chain
Five components organise an assisted production in image segmentation. Designation, limited to the classes qualified on the pilot batch. Manual tracing, kept for openwork structures and continuous materials. Uniform correction of the measured shift, applied in processing rather than region by region. Automatic checks, slivers, holes and aberrant areas, run on the export. And human conformity verification, targeted on the classes where the shift proves variable. One practical consequence follows. The third component is the most worthwhile and the least present, it handles in one operation a defect human verification would catch case by case.Approaching an assisted project
Five questions scope an assisted image segmentation project. What share of the classes lends itself to designation. That answer bounds the gain. Does the placement rule coincide with the contrast. A negative answer requires a systematic correction. Do openwork structures feature in the reference. Their presence calls for manual tracing. Is an unassisted share planned. Its absence makes drift undetectable. And does control verify the convention or the sharpness. That distinction determines what will be caught. Those five answers determine the real gain and the control arrangement. Asking them before starting avoids a fast corpus whose boundaries encode a rule nobody chose.The question that decides on its use
One question determines whether this assistance will contribute anything to the project. What share of the annotated surface belongs to compact contrasted objects. A high answer makes the assistance decisive, most of the work benefiting directly from designation. A low answer makes it marginal, the gain bearing on a fraction of the corpus while verification applies to the whole. That question is answered by examining a few representative images, it precedes any discussion of tooling, and it avoids announcing a saving the corpus’s composition forbids.What to tell the client
Four framings make this conversation honest in image segmentation. Announcing a gain per class rather than a global gain, the corpus’s composition determining the result. Setting out that control changes nature, conformity with the convention replacing the examination of sharpness. Flagging that the background does not benefit from assistance, which bounds the saving on exhaustive corpora. And recalling that the class stays human, assistance not touching the most expensive error. One observation follows for an image segmentation provider. The second framing is the most distinctive, a client generally hearing that assistance reduces control and discovering that it shifts it towards a different and equally necessary verification.Three decisions before switching it on
Three decisions frame the use of this assistance in image segmentation. Determining class by class where designation applies, on the basis of a pilot batch rather than an intuition. Measuring the gap between contrast and convention before production, which permits correcting it uniformly. And fixing the share of unassisted batches at the outset, a value that is not renegotiated under schedule pressure. Those three decisions cost a day, they precede the first image, and their absence produces a fast corpus whose boundaries encode a model’s convention rather than the project’s.What this chapter leaves to the next
One domain puts these limits particularly on display. Medical imaging combines three properties that work against this assistance. Boundaries there are frequently progressive, a lesion merging into the neighbouring tissue. The definition of the structures belongs to a clinical knowledge the appearance does not carry. And variability between experts constitutes the upper limit of what a corpus can reach. Those three properties make this vertical the case where the gap between contrast and convention is most marked, and they constitute the subject of the following chapter.What this chapter teaches
One cross-cutting observation deserves closing this examination. This assistance accelerates the gesture and displaces the risk. Three findings compose it. The gain concentrates on the objects manual tracing already handled fastest, which makes it lower than a demonstration suggests. The boundary proposed follows the contrast and not the convention, which produces contours that are sharp and wrong. And the class stays entirely human, which leaves the most expensive error exactly where it was. That finding matches the one the preceding clusters established about assistance, it moves the starting point without moving the limiting factor.What to say when a client has already seen a demo
One conversation recurs and it is worth preparing for. A client who has watched a click produce a clean mask arrives expecting a transformed budget. Three points reset that image segmentation expectation without dismissing the tool. The demo used the objects the tool handles best, which are also the ones a person traced quickly anyway. The demo showed a boundary that looked right, which is not the same as one that follows the project’s rule, and the difference is invisible on screen. And the demo had no background to annotate, which is where a large share of an exhaustive corpus’s work actually sits. One practical consequence follows for an image segmentation provider. Proposing to measure the gain on the client’s own images rather than arguing about it converts this conversation into a pilot batch, which settles the question with a figure and positions the provider as the party that measured rather than the party that resisted.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Announcing a global gain rather than a gain per class.
- Using designation on openwork structures.
- Verifying contour sharpness rather than conformity with the rule.
- Assuming the model knows the domain definition of the categories.
- Omitting the share of unassisted batches.
- Correcting a systematic shift region by region.
- Expecting the model to separate two identical objects in contact.
- Neglecting omitted fine structures in the control.
- Forgetting that the background does not benefit from this assistance.
- Believing the class is produced along with the region.
