The combined formulation solves one real problem and creates three others. It requires deciding, category by category, a distinction neither geometry nor appearance supplies: what can be counted and what merely occupies a surface.
This article sets out what it covers. It extends the article on choosing between semantic and instance segmentation.
What this formulation produces
A single image segmentation deliverable combines the two preceding approaches. Every pixel receives a class, and pixels belonging to countable objects additionally receive an individual identifier. Three properties follow in image segmentation. Coverage is complete, no zone staying without a class. Countable objects are separated from one another. And continuous materials are not, no identifier dividing them. One important practical consequence follows. This formulation avoids the principal defect of the two preceding ones, semantic losing the count and instances leaving the background undescribed in image segmentation.The distinction that grounds it
This decision precedes any image segmentation production. It is more delicate than it appears. Each class must be placed on the countable side or the continuous side. Three criteria guide that placement in image segmentation. The existence of proper boundaries, a countable object having limits a continuous material does not. The meaning of counting, the question of number having an answer for one and none for the other. And the downstream use, which determines whether the individual information will be exploited. One important observation follows. Those three criteria do not always agree, a tree being countable in principle and rarely counted in practice, which makes this placement a project decision rather than a property of the world.The classes that resist placement
Four categories sit badly on either side in image segmentation. Objects that form sets, trees in a forest or grains in a heap. Structured surfaces, a facade holding countable windows on a continuous material. Divisible objects, a puddle or a zone of degradation having no natural number. And categories whose meaning changes with scale, an isolated building and a district not calling for the same treatment. One practical consequence follows for an image segmentation project. Those four categories must be settled explicitly in the reference, failing which each annotator decides on intuition and the corpus loses its consistency.How to place an ambiguous category
A four-question method settles the cases set out above. Does the number of items appear in an expected deliverable. A positive answer imposes countable status. Do two items of this category touch in the real data. A negative answer makes the status of no practical consequence. Does the expected measurement bear on one item or on a total area. That answer separates them directly. And is the separation cost acceptable for this category. That answer can overturn a decision of principle. One important practical consequence follows for an image segmentation project. The second question resolves most cases with no discussion, a countable category whose items never touch receiving continuous status with no loss of information at all.What this formulation imposes
Four constraints distinguish this image segmentation deliverable from the two preceding ones. Exhaustiveness, inherited from the semantic formulation. Non-overlap, a pixel being able to belong to one region only. Consistency between the two layers, an instance’s class having to match that of its pixels. And declaration of each class’s status, countable or continuous. One important observation follows. The second constraint is the most structuring, it forbids the overlaps instance segmentation tolerates and it therefore forces the zones where two objects superimpose to be decided.The question of overlap
A practical difference separates this formulation from instance segmentation alone. Two partly overlapping objects cannot claim the same pixels. Three consequences follow for an image segmentation project. A partly masked object is described only by its visible part. Reconstructing the complete shape becomes impossible, the information not being stored. And computing an individual area bears on the visible part rather than on the object. One important practical consequence follows for an image segmentation project. The third consequence directly affects measurement uses, visible area and real area diverging sharply in cluttered scenes.What control verifies
Five image segmentation verifications apply specifically to this formulation. Exhaustiveness, no pixel staying without an attribution. Uniqueness, no pixel carrying two memberships. Class consistency between the instance identifier and the pixel label. The absence of an identifier on classes declared continuous. And the presence of an identifier on every region of the countable classes. One important observation follows. Those five verifications are entirely automatable, which makes formal control of this formulation simpler than that of an instance corpus, where the permitted overlap forbids some of these rules.What the annotator does differently
Five gestures distinguish producing a panoptic corpus in image segmentation. Handling the background before the objects, which guarantees full coverage and avoids gaps. Placing the objects on top, each new region covering what precedes it. Checking the order of superposition, a distant object not being allowed to mask a near one. Assigning an identifier only to the classes declared countable. And checking for gaps at the end of the image rather than during the tracing. One practical consequence follows for an image segmentation project. The third gesture is specific to this formulation, the order of superposition carrying an implicit depth information an inattentive tracing inverts with nothing to signal it.What the format must carry
Three representations coexist for this type of image segmentation deliverable. Two superimposed images, one carrying the classes and the other the instance identifiers. A single image whose every value jointly encodes the class and the individual. And a list of regions accompanied by a class map for the background. One important observation follows. The second representation is the most compact and the most fragile, an encoding error producing a silent class shift where the first two separate the two pieces of information and permit checking them independently.What quality measurement complicates
A difficulty specific to this formulation deserves setting out. No simple metric accounts for both layers at once. Three inadequacies illustrate that in image segmentation. An overlap measurement per class ignores counting errors. A detection measurement per instance ignores the quality of the continuous materials. And an average of the two masks which of the layers failed. One practical consequence follows for an image segmentation project. Those three inadequacies require reporting the two layers separately, a single indicator depriving the diagnosis of the information that would permit correcting.Reporting quality correctly
Four indicators account for a panoptic corpus in image segmentation. Mean overlap per continuous class, which measures the quality of the background. The detection and fusion rate on countable classes, which measures the separation. The class error rate on correctly delimited instances, which isolates category confusion. And the proportion of pixels correctly attributed in both class and individual. One observation follows. The third indicator is the most rarely reported and the most useful for diagnosis, a well-traced but misclassified region belonging to a nomenclature problem where a badly traced region belongs to a method problem.What this formulation costs
Four factors determine its surcharge over the simple formulations. Separating countable objects in contact, as in instance segmentation. Background exhaustiveness, absent from an instance corpus alone. Resolving overlaps, which exists in neither of the other two formulations. And settling ambiguous classes, a scoping cost rather than a production one. One observation follows. The third factor is specific to this formulation, it bears on the zones where several objects superimpose and it grows sharply with scene density.What this formulation costs in the background
One observation deserves isolating since it surprises at costing. The background represents most of the surface and an underestimated share of the work. Three reasons explain it. Boundaries between continuous materials are often blurrier than those of objects, vegetation against ground notably. Those boundaries are long, a horizon line or a kerb edge crossing the whole image. And automatic assistance is least effective there, no salient contour guiding the designation. One important practical consequence follows for an image segmentation project. Those three reasons make the background the line quotes most underestimate, a client reasoning in object counts while the load spreads across the whole surface.What assistance brings here
Three observations situate automatic help on this image segmentation formulation. Mask proposal by designation serves countable objects well, a click producing a separate region. The background benefits poorly from that assistance, continuous materials having no salient contour to designate. And resolving overlaps stays human, the order of superposition belonging to an understanding of the scene. One observation follows. The second finding produces a characteristic imbalance, assistance accelerating the countable part and leaving the continuous part intact, which shifts the proportion of the work towards the background across successive batches.When it is justified
Four situations make this image segmentation formulation appropriate. Complete scene understanding, where the background counts as much as the objects. The coexistence of continuous materials and countable objects in the same images. The requirement of coverage with no gaps, imposed by the downstream use. And the need for a single deliverable rather than two corpora to articulate. One practical consequence follows for an image segmentation project. The second situation is the most discriminating, a scene composed only of objects on a neutral background not justifying this additional complexity.When it is not justified
Four situations make this formulation needlessly expensive. The background has no value for the downstream use. No countable class is present, semantic then sufficing. Objects overlap heavily and the complete shape is needed. And the two layers serve distinct uses that need no articulation. One observation follows. The third situation deserves attention, the non-overlap constraint making this formulation unfit for a use measuring the real area of cluttered objects.The reference to write
Six decisions compose the reference of a panoptic project in image segmentation. The list of classes with their status, countable or continuous, with no exception left implicit. The arbitration rule for ambiguous categories, with the already settled cases as examples. The superposition convention, which object goes in front where two regions dispute pixels. The treatment of partly masked objects, with or without an attribute flagging the occlusion. The class receiving the zones the nomenclature does not cover. And the size threshold below which a region is absorbed by its neighbour. One important practical consequence follows. The last decision avoids a frequent defect, panoptic corpora accumulating regions of a few pixels at the boundaries between classes, artefacts that pollute the count without carrying information.The pilot batch that tests the formulation
Four images compose a pilot batch suited to this image segmentation formulation. An image where background and objects share the surface roughly equally, which supplies the reference throughput. An image with heavily overlapping objects, which tests the superposition convention. An image containing an ambiguous category, which verifies that the arbitration rule holds on a real case. And an image with fragmented boundaries between continuous classes, which measures the cost of the background. One observation follows. The last image is the one most often forgotten, a pilot batch composed of scenes with clean objects systematically underestimating the cost of the background, which nonetheless represents most of the surface.The verticals that use it
Four image segmentation contexts naturally turn to this formulation. Autonomous driving, where road and sky are continuous and where vehicles and pedestrians must be counted. Mobile robotics, where navigation requires a complete map and manipulation requires objects to be identified. Urban scene analysis, where buildings and roadway coexist with countable street furniture. And agriculture, where soil and crop form a continuous background punctuated by plants or weeds to count. One observation follows. Those four contexts share a common property, they require complete scene understanding in order to act, which distinguishes this formulation from a use where only certain objects matter.Converting an existing corpus
One frequent situation deserves treating, a client often holding a corpus in a simple formulation. Three conversions arise in an image segmentation project. From a semantic corpus to panoptic, which requires separating the countable objects, work equivalent to fresh annotation on those classes. From an instance corpus to panoptic, which requires annotating the background and resolving the existing overlaps. And from panoptic to either simple formulation, an immediate operation with no loss for the intended use. One important practical consequence follows. The second conversion holds an unexpected difficulty, the overlaps permitted in the original corpus having to be settled one by one, which makes it more expensive than annotating the background alone would suggest.Approaching a panoptic project
Five questions scope a panoptic image segmentation project. Which classes are countable. That placement precedes everything else. Must the background be described exhaustively. A negative answer rules out this formulation. Do objects overlap frequently. That answer quantifies a cost specific to this approach. Must the measured area be the visible one or the real one. That answer can disqualify the formulation. And how are ambiguous classes settled. Leaving them untreated produces inconsistency. Those five answers determine relevance and cost. Asking them before starting avoids adopting a formulation for its apparent completeness rather than for its usefulness.The question that decides on this formulation
One question determines whether this approach suits the need. Must the system know what is everywhere, or only where certain objects are. An everywhere-oriented answer, navigation, scene understanding or a decision grounded in context, justifies the full coverage and its cost. A certain-objects-oriented answer, detection, counting or targeted inspection, makes the background superfluous and divides the spend. That question is asked in one sentence, it belongs to the use and not to the technique, and it avoids this formulation’s most characteristic expense, a background annotated with care no downstream process will ever consult.Three decisions before producing
Three decisions commit a panoptic image segmentation project. Placing each category between countable and continuous, relying on the contact question rather than on an ontological intuition. Fixing the superposition convention, which determines which object keeps the disputed pixels. And verifying that the downstream use genuinely exploits the background, the only justification for the surcharge over an instance corpus. Those three decisions cost one meeting, they precede the first image, and their absence produces a complete corpus a substantial part of whose work serves nothing.What this chapter leaves to the next
One question stays open after this examination of the formulations. The three preceding chapters treated what must be produced without treating how to trace it. Three difficulties belong to that execution. The gesture itself, whose fluidity determines throughput more than the choice of formulation does. The precision achievable, which depends on the tracing method as much as on the attention paid. And the handling of fine structures, which neither the formulation nor the convention resolves. Those three difficulties constitute the subject of the following chapter, which concerns producing the masks themselves.What this chapter teaches
One cross-cutting observation deserves closing this examination. This formulation resolves an articulation and imposes a constraint. Three findings compose it. Placing categories between countable and continuous is a project decision rather than a property of the world. The non-overlap constraint makes formal control simpler and real area measurement impossible. And no single metric accounts for both layers, which requires reporting them separately. That finding extends the two preceding chapters, the boundary remaining the site of the decision and the convention remaining what fixes it.What this formulation is often bought for
One observation about how these projects begin belongs at the close. Panoptic is frequently requested because it sounds like the complete option. Three signs indicate that reasoning rather than a real image segmentation need. The request names the formulation before naming the use, which reverses the order the preceding sections established. Nobody can say what the downstream process does with the background, which is the only thing the surcharge buys. And the overlap question has never been raised, though it can rule the formulation out entirely for a measurement use. One practical consequence follows for an image segmentation provider. Raising these three points costs nothing and frequently reduces the scope to an instance corpus, which is a smaller sale and a better project, and clients tend to remember which of the two a provider proposed.The single check worth running first
One verification separates a usable panoptic corpus from one that merely looks complete. Count the distinct regions per image and examine the smallest ones. Three findings emerge almost every time in image segmentation. Slivers of a few pixels sit along the boundaries between classes, produced by imprecise tracing rather than by anything real. Some countable classes carry regions far below any plausible object size, which signals fragmentation rather than detection. And a handful of pixels somewhere carry an identifier belonging to a class declared continuous. One practical consequence follows. This check runs in seconds on a delivered corpus, it needs no reference and no domain knowledge, and it catches the three defects this formulation produces that no visual inspection reveals.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Adopting this formulation for its completeness with no use that requires it.
- Not declaring explicitly which classes are countable.
- Leaving ambiguous categories to the annotators’ judgement.
- Measuring a real area on a deliverable that stores only the visible.
- Reporting a single metric for both layers.
- Neglecting the cost of resolving overlaps.
- Assigning an instance identifier to a continuous class.
- Leaving pixels unattributed in an exhaustive deliverable.
- Assuming the class placement holds from one project to another.
- Choosing this formulation where the background has no value.
