The eleven preceding articles in this series on satellite image segmentation flagged errors as their subjects arose. This one gathers them and classifies them by when they occur, a method permitting a difficulty to be detected where it appears rather than at delivery.
One observation runs through this inventory: nearly all these errors are corrected at a negligible cost when they are committed and at a considerable cost afterwards. It extends the article on
satellite segmentation models.
Scoping errors
This family occurs before any data is handled.
It is the most expensive, since it invalidates everything after it.
Targeting objects too small for the available resolution is the commonest error. The rule set out in the opening article, that an object must cover several pixels to be reliably delineated, settles that question in a minute and it is rarely applied.
Requesting a non-observable class is the second. A use, an administrative status or a function are not read from an image, a distinction the nomenclature article drew between cover and use.
Confusing announced revisit with usable acquisitions is the third, a frequently cloudy area offering only a fraction of theoretical acquisitions.
And conceiving a vegetation project as a single image project is the fourth, the corresponding article having shown the distinguishing information resides in the series.
Those four errors are detected in one meeting and they eliminate infeasible projects before any commitment.
Nomenclature errors
This family conditions all the satellite image segmentation work that follows.
Its effects are durable.
Fixing the nomenclature after annotation has begun requires reworking what was produced, a change of classes invalidating existing work.
Defining classes by labels rather than by observable criteria produces variability between annotators nothing can subsequently separate from real error.
Omitting priority rules leaves each annotator settling multiple membership alone.
Not fixing the minimum mapping unit produces disagreement about workload and heterogeneity in the corpus.
And multiplying classes with no decision depending on them raises annotation cost, raises validation cost and degrades per-class performance simultaneously.
Convention errors in satellite image segmentation
This family concentrates most of a corpus’s real variability.
Not fixing the boundary convention is the dominant error, the corresponding article having shown boundaries concentrate disagreement and determine the quality ceiling.
Neglecting the edge case library leaves ambiguous configurations settled differently by each annotator and on each occasion.
Handling mixed pixels with no explicit rule produces systematic inconsistency exactly where it is hardest to detect.
Omitting the treatment of shadowed areas leaves part of the image to individual judgement.
And confusing footprint with roof outline on buildings, or footprint with centreline on networks, produces a deliverable that does not match the intended use.
One observation runs across those five errors. Each is corrected in minutes with an expert before production, and none is corrected afterwards without reworking the corpus.
Corpus building errors
This family affects representativeness.
Its effects appear only at deployment.
Sampling by availability rather than under a plan produces a corpus whose scope cannot be established.
Building a corpus over a single season does not permit evaluating a year-round product, a difficulty the water and vegetation articles detailed.
Neglecting difficult configurations in favour of favourable cases produces a measured performance deployment will not reproduce.
Not retaining acquisition metadata, date, sensor and processing level, makes any later diagnosis impossible.
And taking an existing database as ground truth without measuring its discrepancy caps achievable accuracy at that database’s own and imports its nomenclature.
Method errors
This family produces results that look good without being so.
That makes it particularly dangerous.
Splitting training and evaluation randomly rather than geographically produces performance reflecting memorisation, two neighbouring chips sharing too much information.
Evaluating per chip rather than over whole areas produces a figure that does not describe behaviour on a complete scene.
Relying on a global metric masks failures on minority classes and on contours, as the corresponding articles showed.
Using an augmentation chain designed for natural imagery produces physically impossible examples.
And not measuring annotator disagreement deprives the project of the upper bound on what it can achieve.
Technical errors
This family belongs to data handling.
It produces systematic effects.
Losing georeferencing when passing through a general-purpose tool makes the result unusable or imposes an approximate registration.
Mixing different projections introduces offsets that will be attributed to the model.
Neglecting the offset between annotated imagery and production imagery produces a systematic discrepancy on every contour.
Using a different processing level between corpus and production presents the model with a texture or radiometry it does not know, a difficulty the radar and hyperspectral articles detailed.
And loading a whole scene into memory rather than reading it by windows simply blocks processing beyond a certain size.
Post-processing errors
This family intervenes late.
It can degrade a good satellite image segmentation result.
Regularising too aggressively removes real detail in favour of improved appearance.
Simplifying contours without documenting the tolerance introduces a geometric discrepancy whose magnitude nobody will know.
Skeletonising without handling intersection artefacts produces a network with branches that do not exist.
Reconnecting segments abusively creates links absent on the ground.
And converting to vector without verifying topology produces overlapping polygons or gaps, which makes the result unusable in a geographic information system.
Delivery errors
This family concerns technically correct work.
It compromises its use.
Delivering in a format foreign to the client’s practice produces an unused result whatever its quality.
Omitting composition documentation makes the corpus unusable for any later claim and unextendable for a second project.
Not recording the conventions used prevents a third party from understanding what the deliverable describes.
Omitting the exclusion log turns a known gap into an invisible one.
And delivering one format where two were useful, raster and vector, imposes on the client a conversion whose parameters will not be documented.
Durability errors
This family occurs after delivery.
It is the least anticipated of all.
Not planning a maintained reference corpus makes performance drift observable without being measurable.
Neglecting the evolution of sources leaves an arrangement running on data whose characteristics have changed.
Omitting to record the corpus version and the split used makes a result irreproducible.
And leaving knowledge of the conventions in one person’s head makes the corpus unextendable the moment they leave.
Those four durability errors share a common property. They cost nothing as long as nothing changes, and they cost complete reconstitution as soon as something does, which makes them invisible until the moment they become maximal.
Organisational errors
This family belongs to no technical moment.
It is nonetheless frequent in satellite image segmentation.
Assigning annotation to people without ground knowledge produces plausible and wrong delineations, visual recognition not being sufficient in this field.
Having the same people annotate and check removes the independence of the check, an annotator naturally validating their own logic.
Not making the case library accessible during the work reduces it to a document consulted at the start then forgotten.
Neglecting staff rotation lets the conventions a person had internalised leave with them unwritten.
And optimising throughput at the expense of consistency produces a large corpus whose internal variability cancels the benefit of volume.
That last error deserves emphasis. A corpus twice as large and half as consistent is worth less than a smaller regular one, the disagreement bound applying to the whole.
The common structure of these errors
This observation explains their persistence.
Most of them share three properties that make them hard to avoid.
They are committed by omission rather than by action, a decision not taken being less visible than a wrong one.
They produce no immediate effect, their consequences appearing weeks or months later.
And correcting them produces no visible deliverable, which makes it hard to justify to a sponsor awaiting a result.
That deep asymmetry between prevention cost and correction cost is the dominant characteristic of this inventory. A written boundary convention costs an hour; reconstituting it after production costs the whole corpus.
The comparative cost of errors
Not all of them cost the same.
This economic reading clarifies priorities.
A scoping error costs the entire project, nothing produced serving.
A nomenclature or convention error costs the corpus, whose rework is the dominant line of a budget.
A building error costs the corpus extension, the missing examples having to be produced without the existing ones being lost.
A method error costs the evaluation redone and, more seriously, the credibility of a result already announced.
A technical error costs a reprocessing, generally automatable and therefore modest.
And a durability error costs reconstitution, more than the initial build since the context has been lost.
That gradation leads to a practical conclusion in satellite image segmentation. Prevention effort is most efficiently placed on the first three families, where it costs days and avoids months.
Warning signals
Six signs are visible early.
They indicate a satellite image segmentation project is starting badly.
Nobody can name the decision the result will inform.
Annotators ask each other how to handle a case, which means the answer exists nowhere.
The same disagreements recur on the same configuration, which means the convention covers it inadequately.
Evaluation is planned on part of the same set as training.
The schedule contains no corpus building phase, which indicates that work has not been identified.
And no end user has taken part in the discussions, which leaves the question of real use open.
The prevention arrangement
Six measures address the majority of the errors listed.
Their total cost is modest.
The four-question scoping, resolution, observability, availability and ground truth, which takes a meeting and eliminates infeasible projects.
The documented nomenclature with observable definitions, priority rules and minimum unit, which takes half a day with an expert.
The illustrated edge case library, built on real examples before production.
The annotation pilot, which establishes a throughput, reveals ambiguous cases and measures annotator disagreement.
The geographic split of the evaluation set, decided before the corpus is built rather than after.
And documentation produced during, composition, conventions and metadata, which cannot be reconstituted afterwards.
The whole represents days on a project spanning months, and it addresses the causes of the majority of the failures listed.
Reworking a defective corpus
A satellite image segmentation provider is regularly approached after unsatisfactory first work.
Four questions determine what can be salvaged and they take days.
Which family does the defect belong to. A technical error is corrected by reprocessing, a convention error requires re-examining the annotations, a scoping error requires starting again.
Are the conventions used documented. Their presence permits identifying precisely what must be reworked, their absence requires reconstituting by sampling what the annotators were doing.
Is the variability measurable. A sample re-annotated by an expert permits quantifying the discrepancy and therefore deciding whether a partial correction suffices.
And has the acquisition metadata been retained. Its absence prevents distinguishing an annotation error from a date or processing discrepancy.
One important observation follows. A documented corpus, even imperfect, can be repaired, where an undocumented one is re-annotated, which makes documentation decisive retrospectively as much as at the time.
The errors that are not errors
One qualification deserves making, not every observed discrepancy being an error.
Three situations are regularly treated as failures when they are normal.
Disagreement between annotators on ambiguous cases is expected and it is information rather than a defect. Its measurement supplies the achievable performance bound, and its complete disappearance would rather signal an overly coarse convention.
Lower performance on rare classes is structural and not correctable by better work, imbalance being a property of the territory.
And a performance drop on a new geography is documented by research, which makes it a known limit rather than a model failure.
One practical consequence follows for a satellite image segmentation project. Distinguishing an error from a limit avoids spending effort correcting what cannot be corrected, and that distinction belongs at diagnosis rather than after several unsuccessful attempts.
Why these errors persist
One cross-cutting observation deserves closing this inventory because it explains why known errors keep occurring.
Three mechanisms sustain them beyond ignorance.
Schedule pressure systematically favours shortcuts, all six prevention measures delaying the first visible result while producing nothing exhibitable.
The information asymmetry between commissioner and producer means a client cannot distinguish rigorous work from fast work before the consequences appear.
And the absence of a shared norm leaves each project reinventing its conventions, the scientific literature documenting good practice without any obligation imposing it in a commercial context.
That third mechanism is changing. The regulatory requirements the preceding articles flagged, and the rise of institutional buyers, are progressively introducing a method constraint that did not exist.
One consequence follows for a provider. Adopting these practices now amounts to anticipating a requirement that is arriving rather than answering one that exists, a considerably cheaper position than the reverse.
A checklist before production
Ten points condense this inventory into a check applicable before launching a satellite image segmentation production.
The decision the result will inform is named.
The target objects are visible at the available resolution.
The classes are defined by observable and quantified criteria.
The priority rules and the minimum unit are fixed.
The boundary convention is written and illustrated.
The edge case library exists and it is accessible during the work.
A pilot has established a throughput and measured annotator disagreement.
The sampling plan is documented and it covers the difficult configurations.
The geographic split of the evaluation set is decided.
And the metadata to retain with each annotation is listed.
A project ticking those ten has addressed most of the causes listed. A project ticking fewer than half will probably meet several of the families set out, and it is better to know that before committing the annotation budget.
What an audit of an existing project reveals
One practical exercise deserves describing, since it applies this inventory to work already under way.
Four checks establish quickly where a satellite image segmentation project stands, and each takes under an hour.
Ask to see the boundary convention. If it exists as a written illustrated document, the project is in the minority that has addressed its dominant risk. If the answer is a description rather than a document, it has not.
Ask how the evaluation set was separated from training. A geographic answer indicates the method question was considered; a random one indicates the measured performance overstates what deployment will show.
Ask what the inter-annotator disagreement is. A figure indicates the achievable bound is known; its absence indicates no one can say whether the model is close to its ceiling or far from it.
And ask what metadata accompanies each annotation. A list indicates later diagnosis remains possible; a shrug indicates any future problem will be unattributable.
Those four answers characterise a project more reliably than any performance figure, and they can be obtained in a single meeting from a team that has never met the person asking.
Setting these errors out to a client
One commercial dimension deserves stating because naming possible failures looks counterproductive and is not.
Three reasons justify that transparency.
An informed client asks better questions, which produces a better scoped project and fewer disputes at delivery.
A provider able to name what makes a project fail demonstrates an experience no capability presentation replaces, and that demonstration costs one conversation.
And a difficulty announced before it occurs is handled jointly, where the same difficulty discovered at delivery becomes a reproach.
One caveat accompanies that recommendation. Setting out the errors without setting out the arrangement that addresses them installs an anxiety without answering it, which argues for presenting both together systematically.
What this inventory offers a provider
Three commercial consequences emerge from this inventory.
The first is that a scoping engagement has demonstrable value. An interlocutor able to pose the four feasibility questions before commitment spares a client a failure whose cost far exceeds that of the scoping.
The second is that the dominant families, nomenclature, convention and corpus building, belong precisely to an annotation provider’s competence domain, which makes its intervention more decisive than the modelling team’s.
And the third is that these errors are little discussed publicly, which makes setting them out differentiating.
Why the first three families deserve the effort
A closing observation concentrates this satellite image segmentation inventory into one allocation rule.
Nine families were set out and they are not equal. Three of them, scoping, nomenclature and convention, share four properties none of the others has.
They occur first, which means everything downstream inherits them.
They cost hours to address and months to correct, an asymmetry sharper than in any other family.
They belong to human decision rather than to technique, which means no tool, model or budget substitutes for taking them.
And they determine the ceiling rather than the level, which means a project that gets them wrong cannot be rescued by doing everything else well.
That last property is the decisive one. Errors in the later families degrade a result; errors in the first three cap it. A project can absorb a technical error and recover; it cannot absorb an unfixed boundary convention and produce a corpus anyone can defend.
The practical rule follows directly: spend disproportionate attention on what happens before annotation begins, because that is the only period during which the ceiling is still being set.
What to take away
The errors in this field fall into nine families by when they occur, and the first three, scoping, nomenclature and convention, determine the ceiling of everything after.
Three readings emerge. Most of these errors are committed by omission rather than by action, produce no immediate effect and leave no visible trace when corrected, three properties explaining their persistence despite being avoidable. The asymmetry between prevention cost and correction cost is extreme, a written boundary convention costing an hour where reconstituting it after production costs the whole corpus. And six prevention measures representing days on a project spanning months address the causes of most of the failures listed, which makes them a project’s most profitable investment.
For the forward view of this field, the article on
satellite segmentation trends examines the developments. For corpus building, the article on the
satellite segmentation dataset details the method.
To explore delivery arrangements, supported formats and applicable control mechanisms, see our dedicated page on
geospatial data processing. And if you would like a project scoped before committing to it,
let us discuss your project.
