The choice of a medical image segmentation format looks like a minor technical decision and in fact determines pipeline performance, the fidelity of the information transmitted, and the corpus’s ability to return to the hospital ecosystem.
A working group in the field states the trade-off plainly: interoperability of the clinical standard’s segmentation format has generally been good, but the performance of its implementations has generally proved several orders of magnitude slower than research formats for uses such as consuming segmentation data in machine learning.
Interoperability on one side, performance on the other, and a ratio counted in orders of magnitude. This article examines the families of formats, what each actually carries, conversion losses and how to choose. It extends the complete guide to medical image segmentation.
Three representation families in medical image segmentation
A medical image segmentation mask can be encoded three ways, and their differences are not merely technical. The first is the label volume: a file with the same dimensions as the study, where each voxel carries the index of the class it belongs to. It is the most direct representation and the most used in research. The second is the clinical standard’s segmentation object, which encodes the mask within the same ecosystem as the source study and preserves the link to it. The third is contour representation, historically tied to radiotherapy, where the structure is described by a series of planar polygons rather than by voxels. These three families are not three encodings of the same object. They carry different information, and converting between them entails losses better known before choosing.Research formats
These formats encode a volume of values with the geometric parameters needed to place it in space: voxel dimensions, origin, orientation. Their strengths are compactness, reading speed and the simplicity of the libraries handling them. They are the default choice in machine learning pipelines and they explain the performance gap noted in the introduction. Their limitation is that they carry the geometry and nothing else. No structure name, no link to the source study, no information about who produced the mask or under which protocol. That information exists, but it lives outside the file, in a naming convention or an annex file, and it is lost as soon as the file circulates alone.The clinical standard’s segmentation format
This format was designed for a different purpose: making a mask an object of the hospital ecosystem in the same way as the study that produced it. It therefore natively carries what the research formats leave outside. The identifier of the source study, which ties the mask to its patient and its acquisition. The description of the structures, potentially with a standardised terminology code. And provenance information: the equipment or software that produced the mask, and when. It also permits something the label volume forbids, discussed further below: overlapping structures. Its cost is the one the introduction names. Heavier files, slower implementations, and a library ecosystem markedly less convenient than that of the research formats. For an entity feeding a training loop that reads the corpus repeatedly, that difference is not theoretical.Contour representation
The third family follows a different logic. Rather than labelling voxels, it describes the boundary as a series of closed polygons, one per slice. Its historical domain is radiotherapy, where treatment planning systems work natively with contours. It has a real advantage for that use: the boundary is described with sub-voxel precision, since a polygon vertex is not constrained to the grid. Its weakness is topological. Representing a structure with a cavity, or one fragmented into several components on the same slice, depends on conventions that differ between implementations. A hollow structure can come back filled after conversion with nothing flagging it.Orientation, the number one trap of medical image segmentation formats
If one point were to be retained from this article, it would be this one, because it is the error that costs the most and shows the least. Orientation conventions do not coincide between families or between libraries. The direction of the axes and the position of the origin can be expressed differently, and a conversion performed without attention produces a mask that is mirrored, flipped or offset relative to the study it is meant to overlay. A one hundred and eighty degree rotation is spectacular and immediately caught. A left-right mirror on a roughly symmetrical structure is not, and it can traverse an entire project without being seen. The mask looks plausible, the metrics are computed against a reference carrying the same error, and everything agrees until the day someone superimposes the result on the source study in a clinical viewer. Two measures prevent it. The first is the round-trip test described below. The second is a systematic superimposition of the final result on the source study in a tool independent of the annotation chain, which is the only check that catches an error affecting the whole pipeline uniformly.Weight and performance in medical image segmentation
The trade-off announced in the introduction deserves detailing, since it drives an architectural choice. Research formats are compact and fast to read, which matters when a training loop traverses the corpus tens of times. Storage and reading time are then negligible. The clinical standard’s format is heavier and slower, sometimes by several orders of magnitude according to the source cited. On a corpus of a few hundred studies that difference is imperceptible; on a corpus of tens of thousands read repeatedly, it becomes a real constraint on training time. The consequence is a two-regime architecture, which is what mature projects adopt. The reference corpus is stored in the clinical format, which preserves all the information and allows return to the hospital ecosystem. A converted copy in a research format feeds the training loop. The conversion is validated once and re-executed on demand, and the reference remains the rich version.The metadata accompanying a mask
Whatever medical image segmentation format is chosen, part of the information does not live in the mask itself and must be transported alongside. Five elements are indispensable. The correspondence between label index and structure name, which is what makes the file readable. The protocol version under which the mask was produced, without which a later disagreement cannot be arbitrated. The identity of the annotator and of the reviewer, required in a regulated context. The assistance configuration where one was used. And the geometry of the source study, which allows the correspondence to be verified rather than assumed. The clinical standard’s format natively carries part of this, which is one of its principal arguments. Research formats carry none of it, and a project using them must therefore organise an annex file and keep it synchronised, which is a discipline more than a technique. That last remark deserves emphasis. Encoding the information in the file makes it inseparable; keeping it alongside makes it losable. Most corpora that become unusable after a few years do so not because the masks were lost but because what they mean was.Choosing a medical image segmentation format by destination
The choice follows from what the corpus must feed, and four situations cover most cases. Where the corpus serves exclusively to train a model, within one team and with no return to the clinical world, a research format is the right answer. Its performance is decisive and the metadata it lacks can live in an annex file the team controls. Where the corpus must return to the hospital ecosystem, to be viewed by clinicians or archived alongside the studies, the clinical format is the only workable answer. Converting afterwards is possible and it requires reconstituting information that was never captured. Where the corpus feeds a regulated medical device, the traceability the clinical format carries natively becomes an argument in itself, since it makes the demonstration of data provenance simpler. A fourth case deserves mention: a corpus destined for radiotherapy. There the contour representation matches the target systems and its sub-voxel precision on the boundary matters, since the mask defines a treated volume.The round-trip test, an indispensable control
A simple validation procedure protects against most of the failures described above, and it costs a day. The principle is to send a mask through the full path it will take in production: annotation, export, conversion, storage, reimport, and superimposition on the source study. Any deviation observed at the end is a chain defect, whatever its origin. Three precautions make that test effective. Use a structure of known asymmetric geometry, since a symmetrical structure hides mirror errors. Verify the computed volume at each stage, since a spacing loss shows there before it shows visually. And perform the superimposition in a tool independent of those used in the chain, since an error shared by the whole chain remains invisible from within it. The cost of this validation is measured in hours. The cost of not doing it is measured in reprocessed corpora, and it is the failure most frequently discovered too late.Conversion and its losses
Moving between families of medical image segmentation formats is never neutral, and knowing the losses avoids suffering them. From label volume to contours, the loss concerns topology. A mask with a hole or an isolated component translates poorly into polygons, and cavity representation conventions differ between implementations. A hollow structure can come back filled after conversion with no alert. From contours to label volume, the loss concerns sub-voxel precision. A polygon describes a continuous boundary, and rasterisation returns it to a grid, introducing a systematic volume difference depending on the inclusion rule adopted. That difference is small in relative terms and sufficient to make two measurements non-comparable. Between research formats and the clinical format, the loss concerns metadata. The clinical format transports the source study identifier, the structure descriptions and sometimes terminology coding; research formats retain only the geometry. A downward conversion is therefore irreversible on that plane. The practical consequence is to keep the richest representation produced and generate the others on demand, rather than converting and discarding the original. Storage costs less than reconstitution.The case of multiple structures
A difficulty specific to medical image segmentation arises as soon as several structures coexist on one study. The label volume imposes a choice: each voxel belongs to one class and one only. Overlapping structures are therefore impossible to represent, which is a problem as soon as the ontology contains nested objects, a lesion within an organ, or genuinely overlapping ones, oedemas from two adjacent lesions. Two workarounds exist. The first produces one binary file per structure, which permits overlap at the price of multiplied file count and storage. The second establishes a priority order, each voxel being assigned to the highest priority class, which loses information but keeps a single file. The clinical format avoids the problem since it natively permits several segments over the same voxels. It is one of its strongest arguments and it is rarely cited among them. That question must be settled at scoping rather than at delivery. Discovering that the chosen format cannot represent the ontology requires either changing format or renegotiating the ontology, and both are expensive once production has started.Structure nomenclature
One aspect of medical image segmentation formats is routinely treated as a detail although it conditions the corpus’s usability: how structures are named. Three levels of rigour occur. The commonest rests on free strings, each project inventing its names, which works internally and makes aggregation between corpora impossible. The second rests on a controlled list specific to the project, documented and stable, which suffices for most uses. The third relies on standard terminology coding, which the clinical format can transport, and which makes a structure identifiable without knowledge of the project. That third level takes modest initial effort and has two benefits projects underestimate. It makes the corpus aggregable with others, which has growing value as public datasets multiply. And it removes an entire class of ambiguity, since a free name can designate different anatomical extents depending on the specialty using it. The practical rule is to decide that level at scoping rather than coming to it later. Adding terminology coding to a delivered corpus means reopening every file, whereas carrying it from production costs one column in the protocol.What to agree with the receiving team
The format is the point of contact between the provider and the team that will consume the corpus, and misunderstandings there are frequent. Five points benefit from being written before production. The exact delivery format and its version. The orientation and geometry convention, with a jointly validated example file. The file structure, one per structure or a multiclass volume. Metadata transport, annex file or carrying format. And the validation procedure, which must be executed on a reduced batch before production opens. That last point is the one most often skipped and the most protective. Having the receiving team validate a batch of ten studies, in their own environment and with their own tools, surfaces incompatibilities while they cost a day rather than a corpus.Compression and storage
One practical dimension of medical image segmentation is rarely discussed at scoping and becomes a constraint on large corpora: how much space the masks occupy and how that space is managed. A label volume has the same dimensions as the study but a very small range of values, which makes it exceptionally compressible. Lossless compression typically reduces it by an order of magnitude, and it should be applied by default since nothing is traded for it. Two points deserve care nonetheless. Compression must be lossless, without exception; a lossy scheme applied to a label volume does not blur the boundary, it invents classes, which is a corruption rather than a degradation. And the compression scheme must be one the receiving team’s libraries read natively, otherwise the space saved is paid back in integration friction. A separate consideration applies to the source studies. A corpus is not only its masks, and keeping the studies alongside is what allows the annotations to be reviewed, corrected or re-exported later. Projects that archive masks without their studies find, some years on, that they hold labels nobody can verify.Versioning the corpus
A medical image segmentation corpus is rarely produced once and left alone. It grows, it is corrected, and its protocol evolves, which raises a question of versioning that formats do not answer on their own. Three practices keep this manageable. Assign a version to the corpus as a whole and record what changed between versions, in plain terms rather than as a file diff. Never modify a delivered mask in place; produce a new version and keep the previous one, since a model trained on version one cannot be evaluated against version two without knowing what moved. And record, per study, the corpus version it belongs to, which makes any later analysis attributable. None of this is technically demanding and all of it is nearly impossible to reconstruct retroactively. A project that starts versioning after its third delivery has already lost the ability to explain differences between its first two.The most common mistakes
These failures recur often enough across corpus projects that naming them is usually enough to avoid them.- Choosing the format from habit rather than from the corpus’s destination.
- Not testing the conversion round trip on an asymmetric structure.
- Verifying superimposition only within the chain that produced the mask.
- Storing structure names in a file naming convention alone.
- Not recording the protocol version alongside the masks.
- Discovering that the format cannot represent overlap after production starts.
- Converting to a lighter format and discarding the richer original.
- Losing inter-slice spacing during extraction to images.
- Delivering without having a reduced batch validated by the receiving team.
- Using the clinical format inside a training loop, at the cost of degraded performance.
