The debate between two and three dimensional approaches is often settled on principle when it can be measured. A comparative study across 3,430 brain studies supplies the two figures that structure the trade-off: the three-dimensional, intermediate and two-dimensional approaches gave the highest to lowest Dice scores across all models tested, and the three-dimensional approach requires twenty times more computational memory than the other two.
Better performance, twenty times the cost. The trade-off looks simple and is not, because a second study adds a decisive qualification: for anisotropic volumes where the x and y axes preserve much higher resolution than the z axis, the performance advantages of three-dimensional networks are not obvious, sometimes even worse than some intermediate methods.
In other words, the superiority of the third dimension depends on the geometry of the study. This article covers that trade-off from the angle an annotation project cares about: what 2D and 3D actually mean, how anisotropy decides, and what each regime implies for the annotation gesture, control and cost. It extends the complete guide to medical image segmentation.
What 2D and 3D mean in medical image segmentation
The first source of confusion in medical image segmentation is that these terms cover two distinct realities assumed to be aligned when they need not be.On the model side
The study cited defines the three regimes precisely: three-dimensional models process a patch of the volume as input, with all feature maps and parameter tensors three-dimensional, while two-dimensional models process a slice and output a segmented slice, and intermediate models take five consecutive slices as input channels but remain two-dimensional in the rest of their architecture, outputting the segmentation of the middle slice.On the annotation side
Annotation is described by the gesture and by the control. Annotating in two dimensions means drawing slice by slice in a single plane. Annotating in three dimensions means either having tools that operate on the volume, region growing, inter-slice propagation, sculpting, or at minimum controlling the result in all three planes.Why the two do not coincide
A corpus can be annotated slice by slice and used to train a three-dimensional model, and that is in fact the commonest case. The reverse also occurs: a volumetric tool can produce masks consumed by a two-dimensional model. That independence has a consequence projects discover late. A three-dimensional model learns spatial consistency from the data supplied to it. If that data was annotated slice by slice with no volumetric control, it contains longitudinal inconsistency the model will learn faithfully. The model’s third dimension does not correct the absence of a third dimension in the ground truth.Anisotropy, the decisive variable in medical image segmentation
This is the parameter that settles the trade-off, and it can be read directly from the study metadata. A volume is isotropic when voxel size is rigorously identical in all three directions. It is anisotropic when in-plane resolution far exceeds slice spacing, an extremely common situation in clinical practice, where a ratio of five to ten between the two is frequent. The consequence is physical before it is algorithmic. On a strongly anisotropic volume, two neighbouring slices show markedly different anatomical sections, and the continuity the third dimension is meant to exploit does not exist in the data. That is what the study cited above explains: on these images with high inter-slice discontinuity, directly applying a three-dimensional network may involve irrelevant features and hinder the learning process. For an annotation project the practical rule is simple. On an isotropic or mildly anisotropic study, volumetric annotation and three-plane control are fully justified, longitudinal consistency being both expected and verifiable. On a strongly anisotropic study, demanding perfect inter-slice continuity means asking the annotator to invent information the study does not contain.Intermediate regimes
Between the two extremes, several medical image segmentation arrangements offer part of the volumetric benefit for a fraction of the cost. The multi-slice regime feeds the model several consecutive slices and has it output the segmentation of the central one. It supplies immediate context without the cost of a volumetric architecture, and the field’s empirical study observes that all methods of this type have the potential to improve on a two-dimensional baseline. Multi-view fusion segments independently in the three planes, axial, sagittal and coronal, then combines the results. The same study notes that this approach can obtain results comparable to three-dimensional networks, but that three networks, or four, must then be trained to produce the final result, and that the gain stays modest on anisotropic volumes. Patch-based processing, finally, is less a regime than a practical necessity: available memory ruling out processing a whole volume, most three-dimensional methods operate on patches recomposed afterwards. That constraint has a direct annotation consequence, since a structure crossing several patches must stay consistent at their junctions.What the third dimension actually contributes
Two distinct benefits of three-dimensional medical image segmentation are often conflated, although they do not carry the same weight. The first is the performance gain, real but moderate on comfortably sized datasets. The second is more interesting for an annotation project: the same brain study observes that three-dimensional models maintained higher Dice scores when the training set size was decreased from 3,199 studies down to 60. That result has direct economic reach. A three-dimensional model is more robust to scarcity of annotated data, which means that for a given performance target it requires less of it. In a domain where annotation is the dominant cost line and where annotated studies number in the hundreds rather than thousands, that argument outweighs the raw Dice gain. The scoping consequence is that a trade-off based only on compute cost compares the wrong things. The useful question is not which regime costs least to compute, but which regime reaches the clinical objective on the smallest annotated corpus.What each regime implies for annotation
This is the most directly operational aspect of a medical image segmentation project, and the least covered in the technical literature.The gesture
Slice-by-slice contouring remains the fastest and most controllable gesture, including where the target is a three-dimensional model. Volumetric tools, region growing or connected thresholding, are effective on homogeneous well contrasted structures and become hard to control as soon as the boundary is ambiguous, propagation then leaking into neighbouring tissue. The strategy that gives the best results in practice combines both: automatic volumetric initialisation where it works, slice-by-slice correction on the difficult zones, validation in all three planes.Inter-slice propagation
This is the field’s main productivity lever. Annotating one slice in n and propagating the intermediate contours sharply reduces time, provided the structure changes regularly. The practical rule is to densify the annotated slices where geometry changes fast, organ poles, bifurcations, lesion limits, rather than spacing them uniformly. That propagation technique introduces a risk of its own: interpolated slices look plausible and rarely are at the extremities, where the contour shrinks rapidly. Control must target those zones specifically.Sparse annotation
One intermediate regime is worth knowing: annotating only a fraction of slices, without interpolating, and letting the model learn from that partial supervision. It divides annotation cost and presupposes a suitable training method. It does not suit cases where the measurement sought is a volume, since volume cannot be computed on a sparse corpus, but it is relevant where the objective is to train a model that will subsequently segment fully.Elongated structures as a special case
One family of structures defeats the general reasoning and deserves separate treatment: thin elongated objects crossing many slices, vessels, nerves, ducts, distal airways. Their particularity is that they appear on each slice as a minute section, sometimes a few pixels, whereas the real object is a continuous structure several centimetres long. Three consequences follow. The first is that volumetric overlap is a poor guide: a segmentation can post a respectable score while fragmenting the structure into segments, which destroys its usefulness if the clinical use rests on path continuity. Connectivity then becomes a metric in its own right, reported alongside overlap. The second is that slice-by-slice annotation is particularly painful and error-prone here, a three-pixel section being hard to place consistently. Path-following tools, which propagate along the axis rather than perpendicular to it, are markedly better suited. The third is that these structures suffer more from anisotropy than compact organs do. An oblique vessel crossing a thick-slice volume is not continuously sampled, and demanding perfect continuity amounts to requesting a reconstruction rather than an observation. A medical image segmentation project on this kind of structure must therefore state explicitly whether interpolation along the path is permitted.Quality control of a medical image segmentation by regime
The control arrangement must follow the chosen regime, and the commonest error is controlling in two dimensions a corpus destined for volumetric use. Review in all three planes is the field’s specific control. A segmentation drawn in axial can show step artefacts, holes or thickness variations in sagittal that are invisible in the working view. That control must be systematic wherever the deliverable is volumetric, whatever the annotation plane. Two indicators usefully complete that review. The per-slice area curve, which immediately reveals abnormal discontinuities, an anatomical structure evolving gradually. And a connectivity check, flagging isolated components, generally propagation errors. On strongly anisotropic volumes this control must be interpreted carefully. A stair-stepped appearance in sagittal may reflect the real slice spacing rather than an annotation defect, and conflating the two leads to demanding corrections that degrade fidelity.The annotation plane and what it induces
An apparently minor choice durably influences the quality of a medical image segmentation corpus: which plane the annotator works in. Default practice is the axial plane, because it is the acquisition plane and the one clinicians read first. It is not always the best for delineating, however. A structure oriented along the body axis, such as a long muscle or a vertical vascular segment, is poorly seen across successive axial slices and is delineated far more reliably in sagittal or coronal, where its extent appears in one piece. Two practical consequences follow. The protocol benefits from specifying the recommended working plane per structure rather than imposing axial everywhere, which improves both speed and consistency. And control should focus on the plane not used for drawing, since that is where continuity defects appear. One side effect deserves flagging: an annotator always working in the same plane develops a directional bias, their contours being systematically more precise in one direction than the other. Alternating planes on structures that permit it reduces that bias, and measuring it requires comparing boundary deviations by orientation rather than global overlap alone.Geometry as the principal source of error
A medical image segmentation project mixing two and three dimensional regimes handles conversions, and that is where the costliest errors occur. Three traps recur systematically. Loss of slice spacing during extraction to images, which makes any later volumetric computation impossible. Axis inversion between the study reference frame and the annotation tool’s, which produces a mirrored mask, an error all the more insidious for being invisible on a roughly symmetrical structure. And silent resampling, where a tool normalises the volume without the produced mask being returned to the original frame. The defence is a systematic round-trip test before opening production: annotate a structure of known asymmetric geometry, export, reimport, superimpose on the source study and verify alignment. Ten or so cases suffice and this check prevents complete reworks.A decision grid
Five questions settle the regime at project start, and they can be answered in one meeting. What is the anisotropy of the studies? A high ratio between slice spacing and in-plane resolution points towards a two-dimensional or intermediate regime. Is the measurement sought volumetric? If so, three-plane control becomes mandatory whatever the annotation gesture. How many annotated studies will be available? A limited corpus argues for a three-dimensional regime, more robust to scarcity. Are the structures homogeneous and contrasted? If so, volumetric tools bring real gains; if not, slice-by-slice contouring stays safer. Finally, what compute constraints bear on deployment? A factor of twenty on memory is not neutral in a hospital environment. Those five answers almost always point to an obvious regime, and the exercise frequently reveals that the assumed regime was not the right one.Comparative cost of the three regimes
The three regimes are not budgeted alike, and the lines that vary are not the expected ones. Annotation cost depends far more on the number of slices to process and the difficulty of the boundaries than on the target regime, since the gesture stays largely slice-by-slice in every case. What changes is control cost: three-plane validation adds significant time per study, and that time is incompressible. Compute cost, by contrast, varies sharply, and the documented factor of twenty on memory translates into hardware and training duration. That line generally falls to the client rather than the annotation provider, but it belongs in the overall trade-off, otherwise a regime is chosen that the available infrastructure cannot support. The genuinely discriminating line remains the corpus volume required. If a three-dimensional regime reaches the objective with half as many annotated studies, the annotation saving far exceeds the compute premium, which reverses the apparent trade-off.A regime can evolve
One final remark avoids an unnecessarily final choice in medical image segmentation: the chosen regime does not commit the corpus irreversibly, provided it was thought through. A corpus annotated slice by slice but controlled in three planes, delivered as a label volume with its full geometry, feeds two-dimensional, intermediate or three-dimensional training indifferently. It is the most versatile format and it does not cost appreciably more than a two-dimensional deliverable, the bulk of the premium being the volumetric control. Conversely, a corpus delivered as separate slices without geometric metadata permanently closes the three-dimensional option, and that loss is not recoverable. The prudent rule is therefore to annotate at the lightest regime the task permits, but to deliver in the richest format the data allows. Decoupling working regime from delivery regime is the best way to preserve the options of a machine learning team whose architecture choices will probably change during the life of the project.Running the comparison on your own data
Published medical image segmentation comparisons are useful for framing and misleading if a regime is chosen from them directly, since the result depends on the anatomy, the modality and the anisotropy of the studies in question. The workable method is a short evaluation on the pilot corpus. The same annotated subset trains a two-dimensional, an intermediate and a three-dimensional model, all evaluated on the same held-out studies with the same metrics. Two weeks are usually enough once the corpus exists, and the result is specific to the project rather than to a paper. Two outputs matter beyond the ranking. The performance curve against corpus size, obtained by training each regime on increasing fractions, shows where the annotation budget stops buying performance and which regime reaches the target soonest. And the per-structure breakdown frequently reveals that no single regime wins everywhere, large compact organs favouring one and thin elongated structures another, which argues for a mixed arrangement rather than a single choice imposed across the whole ontology.The most common mistakes
These failures recur often enough across segmentation projects that naming them is usually enough to avoid them.- Choosing the regime without having examined the anisotropy of the studies.
- Expecting a three-dimensional model to correct ground truth annotated without volumetric consistency.
- Controlling in two dimensions a corpus destined for a volume measurement.
- Demanding perfect inter-slice continuity on a strongly anisotropic volume.
- Interpolating at fixed intervals rather than densifying at geometry changes.
- Not specifically controlling structure extremities, where interpolation fails.
- Losing slice spacing during extraction to images.
- Not testing the conversion round trip on an asymmetric structure.
- Deciding on compute cost alone without considering the corpus volume required.
- Using volumetric propagation tools on structures with ambiguous boundaries.
- Imposing the axial plane everywhere when some structures delineate better elsewhere.
- Delivering separate slices without geometric metadata, closing the volumetric option.
