Autonomous driving imposes what no other vertical requires simultaneously: describing the whole scene without exception, distinguishing every object individually, and doing so consistently from one image to the next. Those three constraints compound and determine the cost.
This article sets out what this context demands. It extends the article on satellite image segmentation.
What distinguishes this context
Five properties separate this image segmentation vertical from the others. Exhaustiveness is imposed, a navigation system being unable to ignore a zone. Countable objects coexist with continuous materials in every image. Occlusion is permanent, the ground-level view superimposing everything it observes. The scene scrolls past, which adds a requirement of temporal consistency. And conditions vary strongly, night, rain and backlight belonging to the domain of use. One important practical consequence follows. Those five properties make this the vertical where the panoptic formulation is imposed rather than chosen, no simpler formulation covering the need.What exhaustiveness imposes
This requirement governs the vertical and it has a direct justification. A system that ignores a zone treats it as free. Three consequences follow in image segmentation. No catch-all class can absorb what has not been identified without consequence. Distant and poorly resolved zones must receive a class despite their uncertainty. And the sky itself constitutes a class in its own right, its absence leaving a hole in the map. One important observation follows for an image segmentation project. The first consequence distinguishes this vertical, an indeterminate class there having to mean potential obstacle rather than zone of no interest.What motion adds
Four effects follow from the scene scrolling past in image segmentation. One zone is observed several times from neighbouring angles, which creates exploitable redundancy. An inconsistency between successive images shows on reading the sequence. Motion blur degrades the boundaries on near and fast objects. And an object appears then disappears, which raises the question of when its first and last annotation occur. One practical consequence follows. The second effect supplies a check specific to this vertical, accelerated reading of an annotated sequence revealing class jumps an image-by-image examination would not show.The classes that cause trouble
Five categories concentrate the image segmentation disagreements on this vertical. The limit between carriageway and verge, often progressive and yet decisive. Road markings, which belong to the carriageway or constitute a class of their own. Objects partly resting on the carriageway, the wheel of a parked vehicle or a low branch. Overhanging vegetation, which occupies space without occupying ground. And objects whose nature stays undecidable at range, a distant silhouette or a dark mass. One important observation follows. The first category deserves particular treatment, the limit of drivable surface being the information the system uses most directly and the least sharp in the image.What degraded conditions impose
Four situations belong to the domain of use and not to the exceptions. Night, where unlit objects are poorly distinguished from the background. Rain, which produces reflections and changes the carriageway’s appearance. Backlight, which saturates part of the image. And fog or snow, which sharply reduce the useful range. One important practical consequence follows for an image segmentation project. Those four situations must appear in the corpus in sufficient proportion, a system trained on favourable conditions failing precisely when driving becomes difficult.Temporal consistency
This requirement is specific to this image segmentation vertical. It deserves separate treatment. One zone annotated differently from image to image produces an instability the system will reproduce. Four situations defeat it in image segmentation. An object whose class changes mid-sequence without the scene changing. A carriageway boundary that oscillates from one image to the next for no reason. An object appearing late although it was already visible earlier. And an object kept well after it has left the useful field. One observation follows. Those four situations are detected by reading the sequence rather than by isolated examination, which makes this check specific to this vertical and absent from the others.The annotation distance
A scope decision specific to this vertical deserves setting out. A distant object occupies few pixels and delimiting it becomes arbitrary. Three options coexist for an image segmentation project. Annotating everything visible, which produces regions of a few pixels whose shape means nothing. Fixing a size threshold below which the object joins a generic class. And fixing a distance threshold beyond which a whole zone is marked as distant and not detailed. One important practical consequence follows. The third option is the most coherent with the use, a driving system not acting on what lies beyond a certain distance, which makes useless a level of detail the corpus would nonetheless pay for.What quality control exploits
Five image segmentation verifications suit this context. Complete coverage, no pixel staying without a class. The stability of a zone’s class along a sequence. The plausibility of positions, a vehicle being unable to occupy certain zones of the image. The continuity of the carriageway surface, whose interruption signals an omission. And the proportion of each class per lighting condition, revealing an imbalance in the corpus. One practical consequence follows for an image segmentation project. The second verification is specific to this vertical and very effective, a class jump on a static zone signalling an error nothing else reveals.The hierarchy of error severity
A particularity of this vertical deserves stating. Not every image segmentation error carries the same consequence here. Four levels are distinguishable in image segmentation. An obstacle classified as free space, the gravest error since it permits a dangerous movement. Free space classified as an obstacle, inconvenient but conservative. A confusion between two obstacle classes, whose effect depends on the behaviour expected. And a boundary imprecision on a zone that is already correctly classified. One important practical consequence follows. That hierarchy must be reflected in the control, a uniform sampling giving the same attention to all four levels while the first justifies reinforced examination.What this work costs
Four factors determine the image segmentation load on this vertical. Exhaustiveness, which forbids ignoring the zones of no immediate interest. Object density, an urban scene and a clear road not comparing. Instance separation on objects in contact, vehicles in a queue notably. And the length of the sequences, temporal consistency adding to the per-image work. One observation follows. The first factor produces a cost clients anticipate badly, describing the background often representing more work than describing the objects the system cares about.What propagation between images brings
A lever specific to this vertical deserves separate treatment. Redundancy between successive images permits reusing work already done. Four benefits follow for an image segmentation project. Stable zones propagate with a simple adjustment rather than a fresh trace. Temporal consistency is obtained by construction on the propagated zones. The work concentrates on what actually changes from one image to the next. And throughput rises with the sampling frequency, two close images differing only slightly. Three caveats accompany it. A propagated error repeats across the whole sequence. Propagation smooths real changes where it is accepted without examination. And the carrier’s motion displaces the whole scene, which limits direct propagation to high frequencies.What assistance brings here
Three observations situate automatic help on this image segmentation vertical. Propagation between successive images works well on stable zones. Designation serves countable objects and not continuous materials. And model pre-annotation advances fast on this domain’s common classes, which are widely documented. One important practical consequence follows. The first finding is specific to this vertical, the redundancy between successive images offering a lever the other image segmentation contexts do not have.What the corpus must cover
Five axes define the coverage of an image segmentation corpus on this vertical. Lighting and weather conditions, from full daylight to rainy night. Road types, motorway, dense urban, rural and roadworks. Traffic densities, from clear to congested. Rare configurations, an unusual obstacle or an atypical vehicle. And the geographies covered, signage and practice varying from one country to another. One important observation follows for an image segmentation project. The fourth axis is the hardest to cover and the most decisive, a driving system’s failures bearing precisely on what it has never met.What rare configurations require
One difficulty dominates building a corpus on this vertical. Failures bear on what the system has never encountered. Three methods permit recovering those configurations in image segmentation. Prolonged collection, which eventually meets them and costs considerable time. Staging on a closed circuit, practicable for some situations and impossible for others. And querying an existing model, whose uncertain detections locate the atypical scenes within an already collected archive. One important practical consequence follows. The third method is the most effective and the least used, an archive of unannotated sequences already containing the rare cases a further collection would go looking for, which shifts the problem from acquisition to selection.The reference to write
Six decisions compose the reference of an image segmentation project intended for this vertical. The list of classes with their status, countable or continuous, inherited from the panoptic formulation. The placement rule for the limit between carriageway and verge. The treatment of road markings and surfacing joints. The distance beyond which a zone is no longer detailed. The meaning given to the indeterminate class, potential obstacle rather than zone of no interest. And the treatment of partly overhanging objects, low vegetation or signage. One observation follows. The last decision is the least obvious, an object occupying space without occupying ground raising a question the two-dimensional representation cannot settle alone.The pilot batch on this vertical
Four sequences compose an image segmentation pilot batch suited to this vertical. A sequence in favourable conditions and moderate traffic, which supplies the reference throughput. A dense urban sequence, which measures the dominant cost factor and tests instance separation. A sequence in degraded conditions, night or rain, which reveals the real fall in throughput. And a sequence containing roadworks, which tests the atypical classes and the flagging instruction. One observation follows. The throughput gap between the first and the third sequence is this batch’s most useful figure, it quantifies a factor clients are unaware of and that determines the cost as soon as the corpus must cover the real domain of use.What the corpus must declare
Six pieces of information accompany an image segmentation delivery intended for this vertical. The list of classes with their status and the exact meaning of the indeterminate class. The distance beyond which zones are no longer detailed. The distribution of the corpus by lighting and weather condition. The distribution by road type and by traffic density. The temporal consistency control arrangement applied. And the list of configurations not covered, which explicitly bounds the domain of use. One practical consequence follows. The third is the one most often missing and the one best predicting the system’s behaviour, an unbalanced distribution announcing a failure in the underrepresented conditions.What segmentation brings to perception
One observation situates this task among the other sources of onboard perception. Four contributions are specific to image segmentation here. The drivable surface, continuous information no box describes. Delimiting objects of irregular shape, vegetation or an atypical obstacle. Contact between regions, a wheel touching the carriageway or a pedestrian approaching it. And covering the zones the object nomenclature simply does not provide for. One practical consequence follows for an image segmentation project. The first contribution alone justifies this task in a perception chain, the drivable surface being the information that decides movement and the one no other representation supplies.Three checks to put in place
Three verifications address most of the defects specific to this vertical. The coverage check, which detects any pixel left without a class. Accelerated sequence reading, which reveals class jumps on static zones. And counting the distribution by condition, which shows whether the corpus covers the announced domain of use. Those three checks run on a delivered export, they need no external reference, and they cover respectively exhaustiveness, temporal consistency and representativeness, the three requirements this vertical compounds.Approaching an autonomous driving project
Five questions scope an image segmentation project on this vertical. Which classes must be counted. That answer fixes the instance share of the work. Out to what distance is annotation required. That answer bounds the volume. Which degraded conditions feature in the corpus. Their absence limits the domain of use. Is temporal consistency checked. Its absence lets instabilities through. And how are objects undecidable at range handled. Their class must clearly mean caution. Those five answers determine the load and the corpus’s validity. Asking them before starting avoids a corpus that describes the easy situations well.The question that frames the corpus
One question determines what the corpus must cover. In what conditions will the system be permitted to operate. A restrictive answer, motorway in fair weather, considerably reduces the corpus and permits solid coverage of that narrow domain. An open answer, every road type and every condition, requires a campaign deliberately covering night, rain and the rare configurations. That question belongs to the product definition and not to annotation, it is asked of whoever commissions the system rather than of whoever buys the corpus, and it determines the budget more than the image count does.Three decisions before the first sequence
Three decisions commit an image segmentation project on this vertical. Fixing the distance beyond which a zone is no longer detailed, which bounds the volume without reducing usefulness. Giving the indeterminate class the meaning of potential obstacle, a nomenclature decision with direct consequences. And settling the corpus distribution by condition, relying on the intended domain of use rather than on what happened to be filmed. Those three decisions cost one meeting, they precede the first sequence, and their absence produces a large corpus that mostly describes the situations where driving is easy.What this vertical brings to the others
Three practices born here hold beyond autonomous driving. The hierarchy of error severity, applicable anywhere a class confusion carries unequal consequences depending on the direction of the error. The control by sequence reading, transferable as soon as a corpus’s images follow one another in time. And the indeterminate class carrying an active meaning rather than a rejection, necessary whenever an unidentified zone must trigger caution. One observation follows. Those three practices answer one same situation, that where annotation feeds an automatic decision and not a human analysis, a configuration that shifts the question from correctness towards consequence.What this chapter closes
The three verticals examined have shown the same thing in three forms. Medical showed a limit coming from expert judgement, documented before the project. Satellite showed a limit coming from the data, the image not carrying the boundaries requested. And autonomous driving shows a limit coming from the use, an error’s consequence varying with its direction. Those three limits call for the same response, a written and declared convention, which explains why the following chapters concern cost, quality and formats rather than further domains.What this chapter teaches
One cross-cutting observation deserves closing this examination. This vertical compounds the constraints the others meet separately. Three findings compose it. Exhaustiveness and counting are imposed together, which makes the panoptic formulation obligatory rather than chosen. The sequence supplies a lever and a requirement, redundancy accelerating the work and temporal consistency adding to the control. And an indeterminate class there means potential obstacle, which forbids the catch-all class of the other verticals. That finding closes the series of verticals, each having shown a constraint the general conventions must absorb.What a provider should ask for up front
One request costs nothing and changes what a provider can deliver on this vertical. Ask for the operational design domain document before quoting. Three things follow from having it in image segmentation. The corpus distribution can be planned against the declared domain rather than against whatever footage exists. The distance threshold follows from the system’s stated acting range instead of being guessed. And the list of conditions the corpus will not cover becomes a shared document rather than a later discovery. One practical consequence follows for an image segmentation provider. Clients who have this document rarely think to share it, and clients who do not have it learn something useful from being asked, which makes the request productive either way.The one thing to measure before committing
One figure decides whether a quote on this vertical will hold. Annotate one image in favourable conditions and one at night or in rain, then compare the times. Three things that ratio settles in image segmentation. It converts the corpus distribution into a budget, since the share of degraded conditions is known from the domain document. It tells the client what covering their stated domain actually costs, rather than what covering daylight costs. And it surfaces the trade-off early, a client sometimes narrowing the domain once the figure is visible. One practical consequence follows. Two images and an afternoon produce this ratio, and a provider who has it prices the real project while one who does not prices the easy half of it.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Using a catch-all class without giving it the meaning of potential obstacle.
- Neglecting the limit between carriageway and verge.
- Building a corpus dominated by favourable conditions.
- Not checking class consistency along a sequence.
- Omitting distant zones on the grounds of uncertainty.
- Handling road markings with no explicit rule.
- Underestimating the cost of describing the background.
- Neglecting propagation between images as a throughput lever.
- Omitting rare configurations from the sampling plan.
- Assuming a corpus covers a geography it has never visited.
