One recent result summarises the state of the field better than a long exposition. On segmenting abdominal regions from CT, a study reports an overall Dice score of 0.82 for an automatic model, approaching the interobserver agreement of 0.88 measured between experts on the same task.
Two lessons sit there. The first is that automatic performance now approaches human performance on some tasks. The second, more important, is that the human reference is not perfect: 0.88 means two experts delineating the same structure do not fully agree. That ceiling governs everything a segmentation project can hope for, and it is too rarely measured before a target is set.
This guide covers the field end to end: what segmentation produces, the modalities and their constraints, delineation conventions, metrics, the place of automatic assistance and how to organise the work. It serves as the entry point to a set of articles each covering one aspect in detail.
What medical image segmentation produces
Segmenting means assigning every pixel or voxel a structure label. The output is a mask superimposable on the study, from which a physical quantity is computed.The difference from detection
Detection says only where a structure is; segmentation says exactly what its extent is. That difference is one of nature rather than degree, since only the second supports computing a volume, an area, a distance to another structure or a quantitative change over time. If the intended clinical use involves a measurement, no lighter primitive suffices.Downstream uses
Four families of use occur in practice, and they do not call for the same rigour. Quantification, organ or lesion volumetry, tolerates boundary imprecision that volume averages out. Follow-up mainly requires reproducibility between two studies, absolute precision mattering less than consistency. Treatment planning, radiotherapy or surgery, demands surface precision since the mask defines a zone to treat or spare. And geometric modelling, for printing or simulation, adds topological validity constraints. Establishing that use before writing the protocol avoids over-specifying work that does not need it, which is the field’s most expensive scoping error.Modalities and their constraints
Each modality imposes its own difficulties on medical image segmentation, and a protocol does not transfer between them.Computed tomography
This is the most standardised modality in the field, with density values on a physically reproducible scale across scanners. That property makes thresholding effective on bone, fat and opacified structures. The trap is assuming standardisation removes the need for conventions: two studies acquired at different injection phases show different enhancement, and the protocol must state which phase the contour is authoritative on.Magnetic resonance imaging
Signal intensity has no absolute value there: it depends on the sequence, the settings and the scanner. Thresholding is therefore inoperative, and interpreting a structure depends on comparing several series. The protocol must state which sequence the contour is authoritative on and how to handle cases where series diverge.Ultrasound
Operator dependent, it produces images whose quality varies widely and where artefacts are frequent. A significant share of acquisitions is unusable, which makes an explicit reject class with written criteria indispensable. It is the modality where image validity is as much a question as delineation.Pathology and microscopy
Scale changes everything there: a digitised slide reaches several gigapixels, which rules out exhaustive segmentation and requires a sampling strategy. Staining variability between laboratories plays the role scanner variability plays elsewhere.Two dimensions or three
This is the most structuring trade-off in a medical image segmentation project, and it is decided at scoping. Two-dimensional segmentation handles each slice independently. It is faster, simpler to tool and sufficient where the measurement sought is planar. Its weakness is longitudinal inconsistency: contours that are perfect on each slice can produce a stair-stepped surface, visible only in sagittal or coronal reconstruction. Three-dimensional segmentation reasons over the whole volume. It guarantees spatial consistency, allows direct volume computation and exploits the context of neighbouring slices. Its cost is markedly higher and it requires suitable tools. A third regime exists and is often the right one: annotating in two dimensions with control in three. The annotator works slice by slice, which remains the most efficient gesture, but validation requires review in all three planes. That arrangement captures most of the three-dimensional benefit for a fraction of the premium.Delineation conventions in medical image segmentation
These are what make a corpus coherent, and their absence produces variability that global metrics do not detect.The anatomical definition
This is the starting point of every convention and the most often rushed. Naming a structure does not define it. For a kidney: is the renal pelvis included, the sinus fat, the artery up to which point, cortical cysts. Those questions look trivial until asked and become sources of massive disagreement the moment production starts. Each definition needs visual examples covering a standard case, an included edge case and an excluded one.The gradual boundary
Many anatomical structures have no sharp limit. An infiltrative lesion fades, perilesional oedema surrounds a tumour, an organ merges with adjacent tissue of similar density. The protocol must say where to stop, through an explicit criterion rather than an appraisal.Partial volume effect
At a structure’s edge, a voxel partly contains two tissues. Should it be included? The convention adopted, including any majority-occupied voxel or only fully contained ones, systematically shifts the measured volume in a consistent direction. That rule must be written, otherwise volumes are not comparable between annotators.Multiple structures and cavities
Does a necrotic lesion include its non-viable centre? Does a hollow organ include its lumen? These questions directly determine the quantity measured, and the answer depends on clinical use. Handling them through examples rather than abstract statements considerably reduces disagreement.Metrics and their traps
The Dice coefficient dominates medical image segmentation, and that dominance is a problem as much as a convenience. It measures overlap between two masks and is dominated by the structure’s core. On a large organ, a boundary wrong by two millimetres all around barely dents the score. On a small structure, a minor offset collapses it. Comparing structures of different sizes by Dice is therefore meaningless: the measured gap reflects geometry as much as difficulty. Two complementary metrics correct that distortion. The 95th percentile Hausdorff distance captures maximum boundary deviation and exposes local errors Dice averages away. Surface Dice measures the proportion of boundary within a given tolerance, which matches clinical perception better. The field’s reference review issues a broader warning: the reliability of performance results reported in medical image segmentation depends largely on the quality of annotations and the clinical ground truth employed, with inter-rater variability emerging as an unavoidable problem, particularly in lesions with ambiguous boundaries and low-contrast anatomical structures.The human ceiling
This point deserves separate treatment because it governs every performance target in medical image segmentation. No reference built from human annotations is ever more consistent than the humans who produced it. If two experts agree at 0.88 on a structure, demanding a Dice of 0.95 from a model means asking it to reproduce one annotator’s idiosyncrasies rather than to learn the anatomy. The practical consequence is that establishing that ceiling belongs in the pilot batch. Having the same subset segmented by two independent experts and measuring their agreement structure by structure costs a few days and prevents contracting an unattainable target. It is also the most useful information for situating a performance honestly: a model at 0.82 against a ceiling of 0.88 is a good model, where the same figure presented alone looks mediocre.Automatic assistance and its limits
General-purpose models have transformed the economics of medical image segmentation, and it is worth being precise about what they actually contribute. On well contrasted organs the gain is considerable and pre-segmentation followed by correction is now the default strategy. On lesions, small structures and ambiguous boundaries the gain shrinks sharply, which limits the expected saving since automation absorbs the easy work and leaves the expensive work. One limitation is particularly instructive. A study on using a general-purpose model to produce annotations more cheaply concludes that although the generated annotations show high per-slice Dice scores compared with ground truth, models trained on them perform significantly worse, the discrepancy arising because the model is designed for two-dimensional segmentation and therefore produces a lack of spatial relationships in three dimensions and poor connectivity between slices. In other words, a pre-annotation can look good slice by slice and produce a degraded corpus, the error showing only in volume. That is one more reason to control in three dimensions what was annotated in two. The familiar anchoring effect adds to this: faced with a plausible looking proposal, an annotator accepts more readily than corrects. The countermeasure is reserving a fraction of the corpus for unassisted work and comparing the two populations.Formats and the technical chain
The technical dimension of a medical image segmentation project is often underestimated at scoping although it governs feasibility and integration cost. The study leaves the imaging department in the clinical imaging standard, carrying the indispensable geometric information: pixel spacing, slice thickness and spacing, position and orientation in the patient reference frame. Most segmentation pipelines then convert to a lighter pivot format, which simplifies handling but discards acquisition parameters. Produced masks are delivered in three forms. The label volume, a file with the same dimensions as the study where each voxel carries its class index, is the most direct to consume for training. The clinical standard’s segmentation object reintegrates the mask into the hospital ecosystem and preserves the link to the source study. Contour formats, historically tied to radiotherapy, are a third route. The costliest error in a segmentation chain almost never concerns drawing quality but geometry: orientation flips, origin offsets, loss of inter-slice spacing. A perfect but misaligned mask is a lost mask. The defence is a round-trip test on ten or so cases before opening production.Organising the work
Role allocation determines the economic viability of a medical image segmentation project. Contouring healthy organs with stable geometry is a perceptual and conventional decision: it delegates effectively to trained annotators, provided initial training is serious and backed by a library of commented cases. Qualifying an abnormality, deciding whether to include an anatomical variant and arbitrating edge cases belong to the clinician. Treating the radiologist as an arbitration point rather than a production workforce is what makes a corpus of several thousand studies sustainable. Sizing it correctly means counting not studies but the decisions that fall to them, which yields a very different order of magnitude. A particularly effective working mode has the clinician intervene upstream to define and qualify, the annotation team handle the contouring, and the clinician return for arbitration on escalated cases then final validation on a sample.Quality control of a medical image segmentation
Three mechanisms combine and their intensity is tuned to the target level. Review in all three planes is the control most specific to the field. A segmentation drawn in axial can be inconsistent in sagittal, and that inconsistency is invisible in the working view. It must be systematic before validation, including where the drawing was done exclusively in one plane. Double reading with arbitration concentrates on the low-agreement structures identified during the pilot, rather than being applied uniformly. Doubling everywhere means spending on easy organs while under-resourcing difficult lesions. Inserting reference cases into the normal flow measures individual drift. On a long repetitive task that drift is real and progressive, and it stays under alert thresholds as long as nobody looks for it. In every case, metrics must be reported structure by structure. A mean Dice across a multi-organ corpus is dominated by the large structures and masks a collapse on the small ones, which are often the most critical.The regulatory frame
A segmentation corpus destined for a medical device enters a documentary regime worth knowing at scoping. Two requirements bear directly on annotation. Traceability, meaning who segmented what, under which protocol version and with which control, becomes a required deliverable rather than a by-product. And demonstrating training data quality, which presupposes retaining broken-down control metrics and the arbitration log. A dataset separation requirement takes a particular form here: the split between training and evaluation must be made at patient level rather than slice or even study level. Slices from one volume distributed across both sets produce overstated performance that no later analysis corrects, and it is a frequent error because slicing makes the fault hard to see.Cost and what governs it
Four variables explain most of the dispersion between medical image segmentation projects. Dimensionality comes first: a thin-slice volume means hundreds of contours where an image produces one. The number of structures multiplies work per study and complicates the protocol. Boundary difficulty determines the share of work that cannot be automated, and it varies by a large factor between a liver and an infiltrative lesion. And the level of control required, which follows from clinical use and from whether the project is regulated. The useful estimation method times the work on a pilot batch structure by structure, on representative cases including difficult ones, rather than applying an average time. The gap between a normal study and a complex pathological one commonly reaches a factor of five, which an average hides entirely.The annotator profile
Medical image segmentation calls on a competence distinct from recognition, and conflating the two explains part of the recruitment difficulty. Recognising that a structure is present is perceptual and acquired quickly. Delineating it reproducibly is a manual skill, comparable to a draughtsman’s: steadiness of hand, consistency of zoom level, sweeping discipline between slices. That skill improves with practice and is measured objectively by an annotator’s agreement with themselves on the same volume weeks apart. Two practical consequences follow. Training must include manual practice with immediate feedback on reference cases, not only transmission of anatomical criteria. And team stability matters more here than on classification tasks, the skill taking weeks to acquire and decaying if the person moves on.What the client must supply
A credible medical image segmentation scoping depends on inputs only the client holds, and their absence explains most estimates that fail in production. Five items matter. The list of structures actually required, with their clinical purpose stated, since that purpose determines the precision level. A representative sample of studies including pathological and difficult cases rather than clean examples. The acquisition parameters, modality, slice thickness, sequences, injection protocol, which determine what is visible and therefore what is annotatable. An identified clinical referent with agreed availability for arbitration. And the existing reference standard where one exists, whether a contouring guideline or a local convention, which is a better starting point for the ontology than building one from scratch. That last point is underused. Radiotherapy departments, in particular, often work from published contouring guidelines that already resolve many of the boundary questions a protocol would otherwise have to settle. Starting from them saves weeks and eases acceptance by the clinicians who will validate the corpus.The most common mistakes
These failures recur often enough across segmentation projects that naming them is usually enough to avoid them.- Setting a performance target without having measured inter-expert agreement.
- Reporting a mean Dice across a multi-structure corpus.
- Comparing structures of different sizes by overlap alone.
- Not writing the partial volume convention, which makes volumes incomparable.
- Segmenting in two dimensions without ever reviewing in three planes.
- Transposing a protocol from one modality to another.
- Expecting volumetric consistency from a two-dimensional pre-annotation.
- Assigning all contouring to overloaded clinicians.
- Over-specifying precision beyond what the clinical use requires.
- Estimating cost on an average time rather than per structure and difficulty level.
- Splitting datasets at slice level rather than patient level.
