A recent publication states in one sentence what most projects refuse to admit: this variability renders the notion of a single, definitive ground truth untenable.
The consequence is uncomfortable. A corpus is not the transcription of a reality annotators approach more or less well: it is the product of partial agreement between legitimately different readings. Treating variability as a defect to be removed therefore removes information rather than noise.
This article covers the question from a project standpoint: where variability comes from, what is reducible, how to measure it, and what to do with multiple annotations when they exist. It extends the complete guide to medical image segmentation.
The three sources of variability in medical image segmentation
A study on representing inter-annotator uncertainty identifies the origin of the phenomenon with useful concision: medical tasks are prone to inter-rater variability due to multiple factors such as image quality, professional experience and training, or guideline clarity. Those three sources call for different responses, and conflating them leads to investing in the wrong place.Image quality
This is the source no protocol reduces. Where the boundary is not visible, two readers produce different limits because the information is missing, not because they disagree. No training corrects that, and demanding it asks the annotator to invent. The response lies upstream, in acquisition criteria and in an explicit reject class. A project accepting any image quality inherits variability it will later attribute to its annotators.Experience and training
This is the most reducible source, and the one a provider acts on directly. It is addressed through initial training, the commented case library, calibration sessions and control by comparison against an expert reference. Its particularity is producing asymmetric variability: a less trained annotator is not merely more dispersed, they are biased in a direction, which makes it detectable by comparison but invisible in a global peer agreement measure.Guideline clarity
This is the most frequent source and the most poorly diagnosed. Where two competent annotators working on a clear image produce different limits, the cause is almost always a missing or ambiguous rule rather than an individual failing. The symptom is recognisable: the disagreement is systematic rather than random, each annotator being consistent with themselves but applying a different convention. That diagnosis matters because it points towards revising the protocol rather than towards training, and those two responses differ in both cost and effect.Measuring before reducing
The first error of a medical image segmentation project is attempting to reduce variability without having characterised it. The useful measurement never reduces to a global agreement coefficient. It has three parts. A breakdown by structure, since agreement on a compact organ and on an infiltrative lesion are unrelated and their average describes neither. A breakdown by annotator, distinguishing shared difficulty from individual drift. And examination of the cases of maximum disagreement, which hold most of the diagnostic information. That last point deserves emphasis. Sorting cases by increasing agreement and examining the ten worst reveals in an hour what no statistic shows: if the disagreements all concern the same kind of situation, the protocol is at fault; if they are scattered, the cause is individual or tied to image quality. That measurement is made at pilot and repeated per batch afterwards. Measured once at project end, it only serves to record.Label fusion strategies in medical image segmentation
Where several annotations of one study exist, what to do with them must be decided, and several approaches coexist. Majority voting retains, for each voxel, the label given by most annotators. Simple and robust, it treats all annotators as equivalent. Simultaneous truth and performance level estimation proceeds differently: it iteratively estimates a latent ground-truth segmentation while modelling annotator-specific sensitivity and specificity. It therefore implicitly weights annotators by their estimated reliability, which makes it superior where levels are heterogeneous. Averaging the annotations produces a continuous map rather than a binary mask, preserving disagreement information as intermediate values. Random sampling, finally, does not fuse at all: it presents the model with a different annotation at each pass. These methods rest on an assumption worth making explicit. The same source notes that the performance-estimation approach assumes an underlying true segmentation exists but is obscured by individual biases. That assumption is reasonable on a well defined anatomical structure and questionable on a lesion with a conventional boundary, where disagreement does not obscure a truth but reflects genuine ambiguity.What medical image segmentation fusion destroys
A recurring criticism of these methods deserves knowing, because it changes how a corpus is conceived. A study on learning from multi-source annotations puts it this way: the inter-observer variability information, which represents the commonality and variety among multi-source annotations, is overlooked during the fusion process. In other words, producing a single mask from several annotations discards the only information indicating where the corpus is reliable and where it is not. That information has value of its own: it allows training to be weighted, predictions to be calibrated, and a clinical user to be told where the system is uncertain for good reasons. A study comparing fusion methods observes accordingly that the conventional training framework shows overconfidence and underestimation of inter-rater variability, whereas approaches preserving that variability produce better calibrated models. The practical consequence for an annotation project is to deliver, where several annotations exist, both the fused mask and the individual annotations. The storage overhead is negligible and the information lost is irrecoverable.The counterintuitive result
One observation deserves knowing before investing in a sophisticated fusion arrangement. A study on the effect of inter-observer variability notes that training with all labelled information, that is without fusion, performs as well as training with either majority voting or performance level estimation. A second study reaches a consistent conclusion on method selection: no consistent observation was made throughout datasets to determine an overall best label fusion method. Two lessons for a project. First, the choice of fusion method is not the structuring decision it is assumed to be: performance differences between methods are small and dataset-dependent. Second, effort is better spent on the quality and consistency of the annotations themselves than on the sophistication of their aggregation, which is good news for an annotation provider and less good for projects hoping statistical treatment will compensate for a weak protocol.Reducing variability in medical image segmentation
Four levers genuinely act, and they differ in cost and in the source they address. The first is writing the conventions. Every systematic disagreement observed at pilot points to a missing rule, and writing it removes that source for the rest of the project. It is the cheapest and most effective lever, and it acts on guideline clarity. The second is the commented case library, which transmits what no abstract rule transmits. Its value lies in documenting the reason for each decision and not only the decision, which allows generalisation to the next slightly different case. The third is the collective calibration session: having the whole team annotate the same batch, then discussing the discrepancies with the clinical referent present. It transmits in half a day what a document does not, and it repeats periodically since drift returns. The fourth is control by comparison against an expert reference rather than by peer agreement. This point is decisive and often neglected: a consistent group can be consistently wrong, and only comparison against an expert reveals a shared bias.Intra-annotator variability
One dimension is regularly forgotten although it is easier to measure than the previous one. The same annotator, faced twice with the same study weeks apart, does not produce the same annotation. That intra-annotator variability sets a floor: nobody can be more consistent with others than with themselves. Measuring it is simple and cheap: quietly reinject a few already processed studies into the work queue and compare. It yields three pieces of information. A consistency floor against which to situate agreement between annotators. An indicator of individual drift, which grows on long repetitive tasks. And a protocol signal, an annotator disagreeing with themselves almost always indicating an insufficiently defined convention rather than inattention. That last point is the most useful in practice. Where disagreement between two people can always be attributed to their differences, disagreement of one person with themselves cannot, which makes it a particularly reliable protocol diagnostic.How many annotators per study
The question of how many annotations per study recurs on every project and has no uniform answer. Three annotation regimes occur in practice. Single annotation with sampled control is the default regime, economical and sufficient on low-variability structures. Double annotation with arbitration doubles the cost and is justified on structures whose pilot-measured variability is high. Multiple annotation, with three or more annotators, costs more still and is justified only on evaluation corpora or on the most ambiguous structures. The economically rational strategy is differentiated rather than uniform. Applying double annotation everywhere means paying for needless control on easy organs and insufficient control on difficult lesions. The pilot supplies exactly the information needed for that differentiation, and it is among its most profitable contributions. One case deserves its own rule: the evaluation corpus. Since it serves to measure, its quality conditions every conclusion of the project, and multiple annotation is justified there even where it is not on the training corpus.Variability as a deliverable
One consequence of the preceding sections deserves pushing to its practical form: what is measured can be delivered. A corpus accompanied by its variability map is worth more than a corpus delivered alone, and the overhead is marginal once the measurement has been made. Three elements compose that delivery. Individual annotations retained alongside the reference mask. A disagreement map per study, showing where annotators diverged. And a per-structure summary of the agreement levels observed. That information serves three downstream uses. It allows training to be weighted, giving less weight to ambiguous regions. It provides a reference for interpreting model performance, since a model failing where experts diverge does not fail in the same sense as one failing where they agree. And it forms part of the data quality demonstration in a regulated context. Few providers deliver that information, mainly because it exposes variability it seems preferable to keep quiet. That is short-sighted: it will be discovered anyway, and having documented it oneself turns it from an admission into a method.What the client must understand
This question is the principal source of misunderstanding between a medical image segmentation provider and its client, and raising it early avoids late difficulty. Three points benefit from being stated explicitly at scoping. Variability exists between the client’s own experts, and measuring it at pilot establishes that fact neutrally rather than defensively. A quality target above that ceiling is unattainable by construction, whichever provider is engaged. And part of the disagreement will not be reduced, because it comes from the image or from the ambiguity of the object rather than from the work. Raising those three before production rather than during a dispute changes the nature of the discussion entirely. Measured upfront, variability becomes shared project data; discovered late, it looks like an excuse.The effect of automatic assistance
A recent phenomenon alters the medical image segmentation variability landscape and deserves anticipating rather than observing. Automatic pre-annotation mechanically reduces variability between annotators, since they all start from the same mask. That reduction is real and measurable, and it is frequently presented as a quality gain. It is not necessarily one. High agreement obtained by anchoring on a common proposal does not mean annotators see the same thing, but that they all accepted the same suggestion. If that suggestion is biased, the corpus is uniformly biased, and the agreement indicator does not signal it. Two measures distinguish the two situations. Have a fraction of the corpus annotated without assistance and compare the two populations, on structure size and on boundary position. And measure agreement between assisted annotators and an expert reference rather than among themselves, which remains the only way to detect a shared bias. That precaution grows more important as model quality improves, since anchoring increases with the plausibility of the proposal. A heavily assisted project must therefore reinforce its comparison arrangement, not relax it because agreement has improved.The clinical referent’s role
One organisational point largely determines the effectiveness of a medical image segmentation arrangement, and it concerns how expertise is mobilised rather than how much of it. The clinical referent is traditionally called upon to validate annotations, which is the most expensive and least productive of their possible uses. Three other employments of their time yield far more. The first is producing the expert reference against which the team will be compared. A few dozen studies suffice, and that reference serves for the project’s whole duration. The second is arbitrating disagreement cases, provided they are presented prepared, grouped by type and accompanied by the precise question to settle. An arbitration justified in one sentence applies to hundreds of later cases, whereas the same decision taken without explanation applies only to the study concerned. The third is participation in calibration sessions, where their presence turns a discussion between annotators into knowledge transfer. Those three consume an order of magnitude less time than systematic validation, and they reduce variability durably where validation merely corrects case by case.The cost of variability in a medical image segmentation project
Variability carries a cost appearing in several lines of a project and worth identifying as such. The first is the measurement arrangement, consuming annotator time on duplicated cases and expert time on references. The second is double annotation where adopted. The third is arbitration, which mobilises the clinical referent and whose volume depends directly on the level of residual variability. The fourth, the most important and least visible, is protocol revision mid-project, which sometimes requires reworking already delivered batches. That last line alone justifies investing in a serious pilot. A missing convention detected on fifty studies costs a rewrite; detected on five thousand, it costs a rework.Variability across sites and over time
Two forms of variability escape the framing above because they operate at a different scale, and multi-site or long-running projects meet both. Between sites, conventions differ even where the protocol is shared. Departments carry their own habits, and annotators recruited locally inherit them. The symptom is a systematic offset between sites rather than dispersion within each, which a global agreement measure hides completely. Detecting it requires stratifying the measurement by site, which costs nothing once the metadata exists and is impossible to reconstruct if it does not. Over time, conventions drift even within one team. Arbitrations accumulate, interpretations settle, and a batch produced in month twelve is not made under quite the same rules as one from month one. The remedy is a fixed reference subset reannotated periodically, which separates genuine drift from changes in the corpus itself. Both matter beyond quality control. A model trained on a corpus with an undetected site offset learns that offset as if it were signal, and performs differently by site for reasons nobody can explain afterwards. Recording site and date as metadata from the first batch is what keeps that diagnosable, and it costs a field.The most common mistakes
These failures recur often enough across corpus projects that naming them is usually enough to avoid them.- Treating variability as an annotator defect without diagnosing its source.
- Setting a quality target without having measured the inter-expert ceiling.
- Measuring agreement globally rather than per structure and per annotator.
- Controlling by peer agreement without comparison against an expert reference.
- Delivering only the fused mask and discarding individual annotations.
- Investing in a sophisticated fusion method rather than in protocol quality.
- Neglecting intra-annotator variability measurement, simpler and highly informative.
- Applying a uniform multiple annotation regime rather than a differentiated one.
- Treating the evaluation corpus under the same regime as the training corpus.
- Raising the variability question with the client only when a dispute arises.
