The organisers of an international tumour segmentation challenge make an admission rarely found in technical literature. On meningiomas, they write that the annotator-approval model does not address inter-observer variability, that dural tail segmentation has significant inter-observer variability due to close proximity to normal enhancing structures and historically ill-defined definitions, and that ultimately expert clinical judgment on meningioma boundaries will likely never be perfectly aligned between different observers.
That sentence captures the field’s central issue. On an organ, the boundary exists and disagreement reflects imprecision. On a tumour, the boundary is partly a convention, and disagreement reflects genuine ambiguity in the object. No protocol removes that ambiguity; a good protocol reduces and documents it.
This article covers the difficulties specific to segmenting tumour lesions: subregions, exclusion rules, multimodality, multi-observer consensus and what clinical use changes. It extends the complete guide to medical image segmentation.
Why medical image segmentation changes nature on a tumour
Four differences separate lesion medical image segmentation from anatomical segmentation, and they compound.No expected shape
A kidney has an approximately known shape, position and size. An annotator therefore has a powerful prior that corrects their perceptual errors. A tumour offers nothing of the kind: it can appear anywhere, in any shape, at any size. The annotator has only the signal, with no safety net.The infiltrative boundary
Many tumours do not stop, they fade. The limit visible on the image does not correspond to the biological limit of the disease, and clinicians know this. Segmenting therefore means delineating what is visible according to a convention, not delineating the real tumour, and that distinction belongs in the protocol so annotators do not each attempt their own extrapolation.Internal heterogeneity
An advanced lesion is not a homogeneous object. It contains active zones, a central necrosis, sometimes cystic or haemorrhagic components, surrounded by oedema that may itself contain infiltration. Segmenting a tumour therefore rarely means producing a single mask.The deceptive neighbourhood
Normal structures frequently mimic the lesion. Enhancing vessels, ischaemic sequelae, post-operative change, artefacts: the main difficulty is not finding the tumour but avoiding what resembles it.Tumour subregions in medical image segmentation
The field’s reference standards decompose the lesion into distinct components, and that decomposition is the first scoping task of a project. A leading challenge’s annotation protocol defines four labels: the non-enhancing tumour core, corresponding to necrosis and covering all portions of the core that do not enhance while being surrounded by enhancing tumour; the surrounding non-enhancing FLAIR hyperintensity, identifying tissue infiltrated by non-enhancing tumour or oedematous from vasogenic oedema; and the enhancing tumour, marking the portions taking up contrast on post-contrast weighted images. These components differ in both difficulty and clinical value. A glioma study reports contrasting scores between them: 0.78 for the necrotic core, 0.86 for peritumoral oedema and 0.88 for enhancing tissue. That hierarchy is instructive. The enhancing component, the most contrasted, is the best segmented. Necrosis, defined by an absence of enhancement within an enhancing zone, is the hardest because its definition is relational rather than intrinsic. Reporting a global score on a tumour therefore mixes three tasks of very different difficulty, and masks precisely the one causing trouble.Exclusion, the hardest part of the protocol
A tumour medical image segmentation protocol devotes as much space to what must be excluded as to what must be included, and that is what distinguishes it from an anatomical protocol. The same standard is explicit on two decisive exclusions. For FLAIR hyperintensity, prior infarcts or microvascular ischaemic white matter changes exhibiting high FLAIR signal are not labelled as such. For enhancing tumour, adjacent blood vessels, bleeding or intrinsic T1 hyperintensity are not included in that class. Those two rules illustrate the nature of the problem. An untrained annotator will spontaneously include any abnormal signal, producing a plausible and wrong mask. The discrimination requires knowing the entities that mimic the lesion, which is medical knowledge rather than perception. The practical consequence is that a library of commented cases focused specifically on anatomical false positives is more useful than a catalogue of typical tumours. The challenge cited notes having supplied its annotators with written instructions and examples of the most commonly encountered pre-segmentation errors, which is exactly that approach.Multimodality in tumour medical image segmentation
Tumour segmentation is distinctive in mobilising several series of one study simultaneously, each answering a different question. The enhancing component is read on post-contrast weighted images. Oedema and infiltration are read on a fluid-attenuated sequence. Necrosis is inferred by comparing those views. No single series suffices, and the annotator works constantly on superimposed or juxtaposed views. That requirement has three operational consequences. Registration between series must be verified before annotation, an offset of a few millimetres being enough to make comparisons wrong. The tool must support synchronised navigation between series, otherwise productivity collapses. And the protocol must state, for each component, which series the contour is authoritative on, otherwise two annotators working on different views will produce disagreements wrongly attributed to the ontology.Inter-observer variability as an unavoidable ceiling
This point governs every performance target in medical image segmentation, and it is more pronounced on lesions than anywhere else. The admission cited in the introduction deserves taking seriously rather than dismissing. It states that on some structures no protocol will perfectly align experts, and that the only response is a multi-observer consensus process, described by the authors themselves as labour and time intensive. The consequence for a project is twofold. First, measuring that variability belongs in the pilot batch, structure by structure and subregion by subregion. Agreement of 0.90 on the enhancing component and 0.65 on necrosis does not call for the same commitments or the same control arrangements. Second, a contractual performance commitment that ignores that ceiling is unattainable by construction, and expressing it per subregion is both more honest and easier to hold.Multi-observer consensus
Where variability is too great to be reduced by protocol, the answer is building the reference from several annotations rather than one. Three consensus arrangements occur. Statistical fusion, which estimates a latent truth from several annotations while implicitly weighting each annotator’s reliability, is used by reference challenges, notably to combine segmentations from different automatic methods. Sequential reading, where a second annotator corrects the first’s work, is cheaper but introduces anchoring. And the consensus meeting, where annotators discuss divergent cases, is the most expensive and the only one that durably enriches the protocol. The choice is a trade-off. On a large corpus, statistical fusion applied to a subset suffices to establish the ceiling and identify problem cases. On a small high-stakes corpus, the consensus meeting produces both a better reference and a reusable case library, which partly offsets its cost.Automatic pre-segmentation
Reference challenges have adopted a medical image segmentation organisation worth describing, since it is now the standard production mode. A recent protocol describes a five-step sequence whose first step involves automatic pre-segmentation of the available images using approaches based on a self-configuring network, followed by iterative manual refinement then expert approval. That organisation has two merits. It sharply reduces annotation time on well contrasted components. And it homogenises the starting point, which mechanically reduces part of the variability between annotators. It also introduces a well known risk that is particularly acute here. Faced with a plausible looking automatic proposal, an annotator corrects less than they accept, and on an infiltrative boundary where no limit is obvious that tendency is maximal. The countermeasure is having a fraction of the corpus annotated without assistance and comparing the two populations, notably on lesion size and on the extent of ambiguous zones.Multiple lesions
A frequent situation complicates the protocol beyond delineation itself: several lesions on one study. Three questions must be settled in writing. Should every lesion be annotated or only those above a given size, and in the latter case the rule must set a measurable threshold. Does each lesion receive its own identifier, which is indispensable as soon as individual follow-up or counting enters clinical use. And how are confluent lesions handled, where components touch or where oedemas overlap. That last question is addressed explicitly by the field’s standards, which note that perilesional oedema from two separate lesions may overlap and conclude that the entities should be treated separately. Without an equivalent convention, two annotators will produce sometimes one lesion and sometimes two, which distorts counting as much as volumetry. An economic consequence attaches to this. Annotation time for a multi-lesion study does not grow proportionally to their number but faster, identifier management and confluence cases adding a load of their own. A costing built on single-lesion studies systematically underestimates these cases.What clinical use changes
The same tumour is not segmented the same way depending on the decision the mask feeds. In radiotherapy, the target volume determines an irradiated zone and the protocol incorporates margins. The requirement bears on surface, and underestimation has opposite consequences to overestimation, which makes error asymmetry explicit. In treatment follow-up, what matters is comparability between two studies more than absolute accuracy. An annotator systematically slightly generous produces perfectly usable follow-up, whereas an inconsistent annotator produces variation that mimics disease change. In surgical planning, the relationship to neighbouring structures dominates, and precision concentrates on the lesion faces in contact with what must be preserved rather than on its total extent. In research and radiomics, reproducibility dominates further still, since extracted descriptors are sensitive to the exact boundary definition. Establishing that use before writing the protocol avoids over-specifying unnecessary precision or, worse, delivering a precise mask where consistency was what was needed.The post-treatment case
A further difficulty appears after intervention, and the field’s challenges increasingly address it. After surgery or radiotherapy, the image contains new elements: resection cavity, scar tissue change, post-radiation enhancement mimicking recurrence. A recent protocol adds a dedicated label for the resection cavity, explicitly recognising that this structure must be distinguished rather than conflated with the lesion. For an annotation project, two rules follow. The post-treatment protocol must be distinct from the initial one, since the entities to discriminate differ. And a corpus mixing pre and post-treatment studies without distinguishing them produces a contradictory signal, the same appearance requiring different labels depending on context.The clinician’s role
Role allocation is more delicate in tumour medical image segmentation than in anatomical work, and getting it wrong makes a project economically unviable. Three levels stand out clearly. Delineating a component whose definition is visual and well written, enhancing tissue on a given series for instance, is a gesture trained annotators perform efficiently. Discriminating between lesion and mimicking structure requires medical knowledge and is where the protocol and the case library make the difference. And qualification, deciding that an image genuinely shows the pathology sought, belongs exclusively to the clinician. The organisation that works involves the clinician at three moments rather than continuously. Upstream, to define subregions and build the case library including false positives. During production, arbitrating escalated cases, whose volume should be measured rather than merely absorbed. And downstream, validating a stratified sample. That arrangement turns a practitioner time requirement proportional to the corpus into one proportional to the number of difficult decisions, an order of magnitude lower. It is what makes a corpus of several thousand studies possible without committing a radiologist full time, and it is generally the point that decides a project’s feasibility.Quality control of a tumour medical image segmentation
The control arrangement follows the field’s general logic, with three inflections specific to lesions. The first is mandatory stratification by subregion. A global indicator is dominated by the largest component, generally oedema, and masks performance on necrosis or on small lesions, which are often the most clinically decisive. The second is specific control of exclusions. Verifying that a lesion was found is easy; verifying that no mimicking structure was included requires actively searching for false positives, which presupposes targeted control rather than random sampling. The third is tracking variability over time. On a task where the boundary is conventional, individual drift is faster than on an anatomical task. Having the same case reannotated periodically by the same annotator, weeks apart, reveals that drift before it affects the corpus.Other locations
The best documented medical image segmentation standards concern the brain, but the principles set out here transfer with adaptation on three points. The relevant subregions change with location. A liver lesion decomposes differently from a glioma, and a lung lesion raises questions of pleural contact and distinction from atelectasis with no cerebral equivalent. Reusing a decomposition established elsewhere without questioning it produces an unsuitable ontology. The mimicking structures change too, and that is the more important point. Every organ has its own false positives, and the case library must be rebuilt for each location rather than borrowed. Finally, the availability of multiple series varies widely. Where brain imaging routinely provides four sequences, other examinations rest on a single acquisition, depriving the annotator of inter-series comparison and making some distinctions impossible. The protocol must then explicitly abandon subregions rather than demand a discrimination the data does not support.Cost of a tumour medical image segmentation project
Tumour segmentation is the most expensive medical image segmentation task in the domain, and four variables explain the dispersion between projects. The number of subregions comes first, each effectively constituting a distinct task with its own definition and control. Boundary sharpness determines the share of work that cannot be automated and varies considerably between a well circumscribed lesion and diffuse infiltration. The number of series to consult simultaneously bears directly on time per study. And the control intensity required, which follows from clinical use and from the variability measured at pilot. The useful estimation method times a pilot batch deliberately including difficult cases, multiple lesions, ambiguous boundaries, post-treatment studies, rather than cases representative of the average. The gap between a single well defined lesion and a complex case commonly reaches a factor of five.Agreeing what the deliverable is
One scoping point avoids a category of dispute that appears only at delivery: what exactly the mask is meant to be. Three formulations circulate and are not equivalent. The visible extent of the abnormality, which is what an annotator can actually produce from the image. The clinician’s best estimate of the disease extent, which incorporates inference beyond what is visible. And the target volume for a given treatment, which incorporates margins and protocol rules on top of either of the first two. Confusing them is common because all three are called the tumour. A project asking for the first and evaluating against the second will find its corpus systematically judged too small, with no error having been made. Stating in one sentence which of the three is being produced, and having a clinician confirm it, costs nothing and prevents a dispute that surfaces late and is expensive to resolve. A useful test at scoping is to ask what happens to the mask afterwards. If margins are added downstream, the deliverable is the first formulation and should not anticipate them. If the mask is used directly, the rules that would have been margins belong in the protocol.The most common mistakes
These failures recur often enough across lesion projects that naming them is usually enough to avoid them.- Reporting a global tumour segmentation score without breaking it down by subregion.
- Not measuring inter-observer variability before setting a target.
- Writing a protocol listing what to include without detailing what to exclude.
- Not verifying registration between series before starting annotation.
- Omitting which series each component is authoritative on.
- Letting automatic pre-segmentation drift into passive acceptance on ambiguous boundaries.
- Mixing pre and post-treatment studies in one protocol.
- Controlling by random sampling without actively searching for false positives.
- Over-specifying absolute precision where the use requires reproducibility.
- Estimating cost on average cases rather than on a pilot including difficult ones.
