The choice between these two formulations looks technical. It is in fact settled on a single question, and that question concerns not the precision expected but what the system will have to do with the objects once it has recognised them.
This article sets out the criteria. It extends the guide to image segmentation.
What each formulation produces
Two image segmentation deliverables are clearly distinguishable. Semantic segmentation produces a class map, each pixel carrying a category and nothing else. Instance segmentation produces a set of regions, each corresponding to a separately identified object. One important practical consequence follows. The first does not permit counting, two cars in contact forming a single inseparable zone, which makes this formulation unfit for any use that enumerates.The question that settles it
One question alone separates these two image segmentation formulations. Will the system have to distinguish one object from another of the same class. Three uses require that distinction in image segmentation. Counting, which presupposes knowing where one individual ends and the next begins. Tracking, which associates an object in one image with the next and therefore requires identifying it. And individual measurement, the area or the dimension of a particular object rather than of a whole zone. One important observation follows. Those three uses cover most industrial needs, which explains why instance segmentation is more often necessary than the initial scoping assumes.What makes semantic sufficient
Four situations make the simpler image segmentation formulation appropriate. The categories concerned are continuous materials, road, sky or vegetation, that nobody counts. The quantity expected is a share of area rather than a number of objects. Objects of the same class never touch in the real data. And the downstream use consumes a class map rather than a list of separately identified objects. One practical consequence follows for an image segmentation project. The third situation deserves verifying on the real data rather than assuming, a class that appears isolated frequently proving to be in contact in a fraction of the images.Where the surcharge sits
One observation contradicts the idea that instance segmentation costs uniformly more. The surcharge concentrates on the contact zones between objects of the same class alone. Three consequences follow in image segmentation. A scene of isolated objects costs practically the same in both formulations. A scene of objects in contact produces a considerable gap. And the proportion of objects in contact predicts the surcharge better than any other single variable. One important observation follows. That proportion is measured on a few images in an hour, which turns a question of principle into a figure and permits deciding the formulation on a concrete basis.The boundary between two identical objects
This difficulty is specific to instance segmentation. It deserves setting out separately. Separating two objects of the same class presupposes a boundary appearance does not always supply. Four situations illustrate it in image segmentation. Two objects of the same colour and texture in direct contact. An object partly masked by another of the same class. A stack whose individual limits are not visible. And a dense crowd where the individuals overlap very largely. One practical consequence follows. Those four situations call for an explicit convention, the separation resting on an interpretation rather than on an observation in image segmentation.The conventions specific to instances
Five decisions are added to the general scoping of an image segmentation project. The treatment of an object cut by the image border, either a full instance or an ignored object. The treatment of an object fragmented by an occlusion, one instance or several. The separation rule where two objects touch with no visible boundary. The visibility threshold below which a masked object is not annotated at all. And the treatment of groups that are too dense for their members to be separated individually. One important observation follows for an image segmentation project. The second decision produces the most frequent disagreement, a car cut in two by a post legitimately counting as one instance or as two according to the rule adopted.The separation rule to write
One convention deserves separate treatment since it produces most of the disagreement. Three formulations coexist where two objects of the same class touch with no visible boundary. Separation at the apparent point of contact, a fast method and arbitrary in detail. Separation by continuity of shape, which extends each object’s silhouette and requires an interpretation. And assumed fusion into a single instance, flagged by an attribute, which openly declines to decide. One important observation follows for an image segmentation project. The second formulation produces the most useful and the most expensive corpora, it asks the annotator to reconstruct an invisible limit and it therefore calls for a specific disagreement measurement.The case of dense groups
This situation recurs in almost every image segmentation corpus. It deserves a dedicated rule. A zone contains too many objects for their separation to be practicable. Three treatments coexist in image segmentation. Exhaustive separation, expensive and often arbitrary beyond a certain level of density. The group annotated as a single region, with an attribute flagging that it contains several. And outright exclusion of the zone, explicitly marked as unannotated. One practical consequence follows. The second treatment is the most informative, it preserves the presence of the objects without claiming a count the data does not permit, and it prevents a dense zone counting as an absence.What the delivery format changes
A technical consequence follows from the choice of formulation. The two are not stored the same way in image segmentation. Three differences separate them in image segmentation. A semantic map fits in a single image indexed by class, compact and simple to read. A set of instances requires either one mask per object or a unique identifier per region, which increases the storage. And converting instances to semantic is immediate, the reverse operation being impossible. One important practical consequence follows. The third difference guides a decision of prudence, a corpus annotated in instances serving both uses while a semantic corpus closes the door to counting for good.What control verifies differently
Four verifications distinguish the control of an instance corpus. The number of instances per class and per image, whose fall signals fusions. Consistency between neighbouring instances, two overlapping regions signalling an error. The distribution of individual areas, whose spike of small values signals fragmentations. And agreement between annotators on the object count, a measurement quite distinct from agreement on boundaries. One observation follows for an image segmentation project. The last verification is specific to this formulation, two annotators being able to trace the same contours and count a different number of objects, a disagreement no overlap measurement reveals.What each formulation costs in control
Three differences separate the control of the two corpora. A semantic corpus is verified by superposition, a class map comparing visually to the image at a glance. An instance corpus requires verifying the division as well, two correct regions being capable of being badly separated. And agreement between annotators is measured on two dimensions instead of one, boundaries and counting being able to diverge independently. One practical consequence follows for an image segmentation project. The second difference explains why an instance corpus requires proportionally longer control, the surcharge therefore bearing on production and on verification alike.What the model learns from each formulation
Three observations link the formulation to the behaviour of the system trained. A semantic model learns to recognise a material and naturally fuses what touches. An instance model additionally learns to separate, a distinct task the data must teach it explicitly. And the quality of separation depends directly on how many examples of objects in contact the corpus contains. One practical consequence follows for an image segmentation project. The third observation requires deliberately over-representing contact configurations, a corpus reflecting the natural distribution containing too few such cases for the model to learn to handle them.What assistance brings differently
Three observations situate automatic help according to the formulation adopted. Mask proposal by designation directly serves instance segmentation, a click producing a region. Semantic segmentation benefits more from propagation by similarity across the large homogeneous zones. And separating objects in contact stays the weak point of both, models frequently fusing what the appearance does not distinguish. One practical consequence follows. The third finding means assistance fails precisely on what distinguishes the two formulations, which leaves the instance surcharge largely with the human.The cases where semantic appears to suffice and does not
Four situations mislead at scoping and surface in operation. A use that does not count today but will as soon as a second version of the system is requested. A class rarely in contact in the demonstration images and frequently in contact in production. A global area measurement that becomes a per-object measurement once the first results are examined. And temporal tracking added afterwards, which requires a per-object identity the corpus simply does not carry. One important observation follows for an image segmentation project. Those four situations share a common structure, the counting need appearing after the corpus is built, which then requires a full rework rather than an enrichment.Measuring the surcharge before deciding
One operation turns this decision into a quantified trade-off. Four measurements are taken on a sample of segmentation images in an hour. The proportion of objects in contact with another of the same class. The cumulative length of boundaries between objects of the same class, against the total contour length. The mean number of objects per contact group, which predicts the difficulty of separating them. And the proportion of images containing at least one group too dense to be separated at all. One important practical consequence follows for an image segmentation project. The second measurement is the most direct, it expresses the surcharge as a share of the total work and it compares immediately with the price gap between the two formulations.Combining the two formulations
This arrangement is frequent in image segmentation. It deserves setting out since it reduces cost. One corpus can use both according to the class. Three principles govern that combination. Continuous materials receive semantic segmentation, counting having no meaning. Countable objects receive instance segmentation where the use requires it. And countable but never counted objects receive semantic segmentation, which saves the entire separation cost. One important observation follows. That combination has a name, it constitutes panoptic segmentation, and it is frequently adopted without being named by projects that simply applied common sense class by class.The verticals and their dominant formulation
Four image segmentation contexts lean naturally one way or the other. Geospatial mostly uses semantic, continuous materials dominating scenes viewed from above. Medical depends on the structure targeted, a single organ calling for semantic and multiple lesions calling for instances. Autonomous driving requires both, road and sky belonging to semantic and vehicles to instances. And industry leans towards instances, a quality control counting and measuring each defect on its own. One observation follows for an image segmentation project. The second context best illustrates the general principle, the formulation depending on the structure annotated rather than on the domain, which forbids deciding by analogy with a neighbouring project.The pilot batch that settles it
Four images suffice to settle this question on real data. An image with clearly separated objects, which supplies the reference throughput common to both formulations. An image with objects of the same class in contact, which directly measures the instance surcharge. An image containing a dense group, which tests the rule adopted for those zones. And an image with an object fragmented by an occlusion, which tests the counting convention. One observation follows for an image segmentation project. The first two images together supply the deciding figure, the throughput gap between them being the real instance surcharge on the client’s data rather than a general estimate.What the corpus must declare
Five pieces of information accompany a delivery and distinguish the two formulations. The formulation adopted, class by class where the corpus combines them. The counting rule for objects fragmented by an occlusion. The treatment of objects cut by the image border. The convention applied to groups that are too dense, with the attribute flagging them. And the visibility threshold below which a masked object was not annotated at all. One practical consequence follows for an image segmentation project. The fourth is the most decisive for exploitation, a corpus whose dense zones are marked permitting them to be excluded from a count while a silent corpus makes them count as a single object.How to present this trade-off to a client
Four framings make this conversation productive. Asking the counting question rather than setting out the two techniques, the answer settling it alone. Quantifying the surcharge on the client’s data rather than announcing a general ratio. Proposing the combination by class, which reduces the spend without reducing usefulness. And flagging that semantic closes the door to counting, a piece of information that frequently changes the decision. One observation follows for an image segmentation provider. The last framing is the most useful, a client hesitating between the two options for budget reasons discovering that they are in fact trading an immediate saving against a later full rework.Approaching this choice in a project
Five questions scope this choice in an image segmentation project. Is a count expected. A positive answer imposes instances. Is tracking between images planned. That answer likewise imposes instances. What proportion of objects touch in the real data. That answer quantifies the surcharge. Do all classes call for the same treatment. A negative answer opens the combination. And what rule applies to groups that are too dense. Its absence produces inconsistency between annotators. Those five answers determine the formulation and its cost. Asking them before starting avoids a corpus that does not permit the intended use or that paid for a use that does not exist.Three decisions before starting
Three decisions commit an image segmentation project on this question. Answering the counting question by asking the downstream user rather than the technical buyer. Measuring the share of boundaries between objects of the same class on a real sample, which quantifies the surcharge. And deciding the formulation class by class, which reserves the spend for the categories that genuinely require it. Those three decisions cost one meeting and an hour of measurement, they precede the first image, and they avoid this field’s most expensive error, a semantic corpus reworked in full because a count became necessary.What this chapter leaves open
One question stays entire after this trade-off. Combining the two formulations in one corpus raises difficulties this chapter has not treated. Three of them deserve flagging. The boundary between a continuous material and a countable object, which belongs to a convention and not to an observation. Format consistency, a corpus mixing class map and instance list having to articulate them. And quality measurement, since no single metric accounts for both formulations at once. Those three difficulties define exactly what the combined formulation must resolve, and they constitute the subject of the chapter that follows this one.What this chapter teaches
One cross-cutting observation deserves closing this examination. This choice is decided on a use and quantified on a measurement. Three findings compose it. The counting question settles it alone, with no consideration of precision intervening. The instance surcharge concentrates on the contact zones, which makes it measurable on a few images rather than estimable. And separating objects in contact is the weak point of automatic assistance, which leaves that surcharge human. That finding extends the preceding chapter, the boundary remaining the site where cost, disagreement and decision concentrate.The upgrade path nobody plans
One practical question follows from everything above and it is rarely asked at scoping. What happens if the counting need appears after delivery. Three options exist in image segmentation and none is cheap. Re-annotating the affected classes from scratch, which costs the original price of those classes over again. Separating the existing regions by hand, which is faster than re-annotating and produces boundaries that were never verified against the image. And accepting a model trained on fused objects, which counts a row of contacting objects as one and degrades wherever density rises. One practical consequence follows for an image segmentation provider. The second option is the one clients choose and the one worth warning against, since it produces a corpus that passes every automatic check while carrying separations nobody looked at, which is a worse position than the honest re-annotation it was meant to avoid.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Choosing the formulation on a technical preference rather than on counting.
- Assuming instance segmentation costs uniformly more.
- Not measuring the proportion of objects in contact before deciding.
- Applying the same formulation to every class.
- Omitting the rule for objects fragmented by an occlusion.
- Omitting the rule for objects cut by the image border.
- Leaving dense groups with no defined treatment.
- Expecting assistance to separate objects in contact.
- Measuring annotator agreement without counting instances.
- Delivering semantic segmentation where a count was expected.
