The only systematic study devoted to labelling instructions in biomedical image analysis reaches a result most projects ignore in practice. Covering 14,040 images annotated by 156 annotators from four professional companies and 708 crowdworkers from a participatory platform, using instructions with different information density levels, it concludes that including exemplary images significantly boosts annotation performance compared to text-only descriptions, while solely extending text descriptions does not.
Lengthening the text achieves nothing. Adding visual examples changes everything. That result contradicts dominant practice, which answers every disagreement with an additional paragraph in the protocol.
This article covers building an annotation protocol and the consensus mechanisms accompanying it. It extends the complete guide to medical image segmentation.
Why lengthening the text fails
The result cited deserves understanding rather than merely applying, since its logic carries over to other decisions. A textual definition describes a category by its properties. Faced with a concrete case, the annotator must check whether those properties apply, an operation already presupposing that each term can be interpreted. Adding properties adds interpretations to make, which does not improve the decision and may slow it. A visual example works differently. It shows a decision taken on a real case, transmitting simultaneously the rule and how to apply it. The annotator no longer reasons by deduction but by analogy, which is faster and more reliable on perceptual tasks. The practical consequence is direct. Faced with an observed disagreement, the instinct is to refine the rule in writing; the effective move is to freeze the disputed case as an example, with the decision taken and its reason. That substitution is the principal quality lever of a medical image segmentation protocol, and it costs less than the writing it replaces.What a medical image segmentation protocol must contain
Four components suffice, and their order of importance is not the assumed one.Definition by example
Each structure needs at least three images: a standard case, an included edge case and an excluded one. That triplet transmits more than a page of description, and it forms the core of the protocol rather than its annex. One constraint attaches to it: those examples must come from the project’s corpus rather than an anatomical atlas. An example drawn from ideal images does not prepare for real ones, and it induces a confidence production will contradict.Exclusion rules
A protocol listing what to include without detailing what to leave out leaves the hardest question open. On most medical tasks, the difficulty is not finding the structure but avoiding what resembles it. Those rules also benefit from illustration. An image of a mimicking structure, with an explanation of what distinguishes it, beats a list of differential criteria.Edge cases
Some questions follow from no observation and must be settled arbitrarily: inclusion of an internal sub-structure, handling of a cavity, the limit between two touching pieces, what to do with a pathological fusion. Those decisions must appear explicitly, with their reason. A rule without its reason does not generalise to the next slightly different case, whereas a justified rule lets the annotator reason by analogy.What to do when in doubt
The most often absent component of a medical image segmentation protocol is also the most useful: what the annotator does when no rule applies. Three courses are possible and must be chosen explicitly. Decide by judgement and flag the case, which preserves pace and feeds the case library. Suspend and escalate immediately, which guarantees consistency at the cost of throughput. Or apply a documented default rule, which suits frequent cases of low stakes. Without that instruction, each annotator invents their own course, and the corpus contains a mixture of untraced individual decisions.The case library as the project’s central asset
This component deserves treating as a project deliverable rather than a by-product of training. Its contents differ from what teams assemble spontaneously. A catalogue of exemplary cases teaches little, since those are the cases nobody gets wrong. What has value is the difficult material: mimicking structures with an explanation of what distinguishes them, edge cases with the decision and its reason, confluence situations with the convention applied, and the typical errors of whichever assistance model is used. Two practices keep it useful. It is built by arbitration rather than by prior writing, since the ambiguities that matter are those production reveals rather than those anticipated. And every entry records the reason, not just the decision. That library is the asset with the longest life in a medical project. Models are retrained and protocols revised, but the accumulated record of why each difficult decision was taken carries across both, and it is what lets a new annotator reach production level in weeks rather than months.The protocol as a living document
A medical image segmentation protocol written once before production and never reopened is an incomplete protocol, whatever the quality of its initial drafting. The reason is structural. The ambiguities that matter surface only on contact with the real corpus, and no anticipation exhausts them. A project must therefore plan a revision cycle, tightened at the start and spaced out afterwards. A contractual consequence follows and benefits from being raised early. If the protocol evolves, early batches were produced under slightly different rules from later ones. Two answers exist: rework earlier batches at every revision, which is expensive, or accept the heterogeneity while documenting it, which is acceptable as long as it is traced. The choice depends on the scale of the revision. A clarification affecting only rare cases does not justify rework; a convention change on a frequent structure does. Settling that at each revision, rather than coming to it at delivery, is what separates a project that is steered from one that is endured.Calibration sessions in medical image segmentation
One arrangement transmits in half a day what a document does not, and it is little used. The principle is to have the whole team annotate the same small batch independently, then examine the discrepancies collectively with the clinical referent present. Every divergence becomes a precise question, and its resolution feeds the case library. Three benefits compound. Ambiguous conventions surface immediately, where they would take weeks to manifest in production. Annotators hear the clinician’s reasoning, which lets them reproduce it on neighbouring cases. And the team calibrates mutually, which reduces variability more effectively than individual correction does. These sessions benefit from periodic repetition, since drift returns. Their useful frequency is high at the start, then falls without ever disappearing entirely on long projects.The consensus mechanism
A protocol must specify not only the rules but how new rules are established, otherwise every arbitration becomes a negotiation. Three consensus arrangements occur. Decision by a single clinical referent is the fastest and most consistent, and its limits are that person’s availability and dependence on their particular judgement. A consensus meeting between several experts produces more robust rules and costs more. Weighted voting, rarer, suits cases where experts come from different specialties with divergent conventions. Whatever the arrangement, two requirements apply. The decision must be recorded with its reason and its date, which makes it citable and generalisable. And it must be circulated to the whole team, not only to the annotator who raised the case, otherwise the same question will recur. That last point is the most often neglected. An arbitration given individually by message improves one study; the same arbitration recorded and circulated improves every subsequent study.Qualifying medical image segmentation annotators
The study cited in the introduction supplies a second result, less surprising but worth stating: professional annotators constantly outperform the participatory platform’s crowdworkers. That finding is worth stating because it is sometimes contested on the grounds that medical tasks could be decomposed into simple micro-tasks. The study concerns biomedical annotation tasks specifically, and the gap there is constant. Its practical consequence concerns qualification. A pre-test should not cover drawing quality on a standard case, which most candidates pass, but the difficult decisions: an edge case, a mimicking structure, a situation where no rule applies. That test reveals in minutes what several days of production would otherwise reveal. It benefits from being completed by a self-consistency measure, having the same case processed twice a few days apart. An annotator who disagrees with themselves on a well defined task will not disagree less with others.Versioning the protocol
A medical image segmentation protocol that evolves must be versioned, and that requirement goes beyond documentary good practice. Three elements must be recorded at each revision. The version number and its date, carried in the metadata of every annotated batch. The list of changes, expressed as rules rather than as a file diff. And the decision taken regarding earlier batches, rework or documented acceptance of the heterogeneity. That traceability has three uses. It allows an observed heterogeneity in the corpus to be explained rather than merely noted. It forms part of the data quality demonstration in a regulated context. And it allows, where an annotation is contested, verification of which rule applied when it was produced, a question with no answer absent versioning.The protocol in the client relationship
One aspect of a medical image segmentation protocol is rarely considered and yet bears heavily on the commercial relationship: the protocol is also what makes a disagreement arbitrable. Without a written protocol, a dispute over an annotation is settled by opinion, and the client’s opinion prevails by construction. With a written protocol validated by the clinical referent, the same dispute is settled by checking whether the rule was applied. Two cases then arise, and both are preferable to the first: either the rule was applied and the disagreement concerns the rule, which calls for a shared revision; or it was not and rework is legitimate. That property has a methodological consequence. The protocol must be formally validated by the client before production, and that validation benefits from bearing on the examples rather than the text, since the examples are what the annotator will actually apply. A provider proposing that validation unprompted, and accepting that the case library stays accessible to the client throughout the project, turns a potential friction point into a demonstration of method. The cost is nil and the effect on trust is disproportionate.What must be documented
One observation from the same research field deserves relaying, since it identifies a collective failing of the domain. Work on dataset documentation notes that aspects such as hardware and data acquisition details are commonly documented, while information regarding data annotation practices, annotation error quantification, or data limitations are not consistently reported, which risks having considerable implications for the abilities of data users to detect potential sources of bias. In other words, projects scrupulously document the equipment and neglect the people and the rules. It is an imbalance easy to correct and it constitutes a differentiator for a provider, since annotation documentation is precisely what they control. Four elements should accompany any delivered corpus: the protocol version with its rules, the profile and number of annotators, the broken-down control metrics, and the corpus’s known limitations. That last point demands an honesty few practise and which rarely turns against the practitioner.Existing medical image segmentation guidelines
One question precedes the writing of any medical image segmentation protocol and is rarely asked: whether to start from scratch. In several domains, contouring recommendations already exist, published by learned societies or established by international consensus. Radiotherapy is the most developed example, with guides detailing the limits of many structures and illustrating the difficult cases. Adopting an existing guide has three advantages. It was built by consensus among experts, giving it a legitimacy an internal protocol will never have. It resolves upfront part of the boundary questions a project would otherwise settle itself. And it eases acceptance by the clinicians who will validate the corpus, since they recognise a reference from their own discipline. Two reservations accompany it. Those guides are designed for clinical practice rather than volume annotation, so they leave open the questions specific to a corpus, notably handling non-conforming cases and what to do in doubt. And they correspond to a precise use, generally therapeutic, which may differ from the project’s. The effective approach is therefore to rely on the existing guide for the anatomical definitions and complete it with the rules specific to production. Writing a protocol without having checked whether a reference guide exists is avoidable expense.Who writes the medical image segmentation protocol
One organisational question recurs on every project and its answer is counterintuitive. The protocol is written neither by the clinician alone, who produces a document that is correct and unusable because it assumes implicit knowledge, nor by the annotation team alone, which produces a usable and sometimes wrong one. The arrangement that works has the annotation team draft from the clinician’s explanations, then has the clinician validate. That direction of drafting has a useful property: what the annotator did not understand does not appear in the text, and that absence is visible on review. The reverse is not. The clinical referent therefore intervenes at three moments rather than continuously: initial definition, validation of the drafting, and arbitration of escalated cases. That distribution is what makes a project of several thousand studies sustainable without committing a practitioner full time.The protocol of a growing corpus
One final point concerns projects whose corpus extends over time, the commonest situation in medical work. Three evolutions occur and each tests the protocol differently. Adding structures requires extending the ontology without renumbering what exists, which presupposes the full nomenclature was fixed at the outset rather than built as needs arose. Adding sources introduces images acquired differently, which can invalidate rules written for one scanner or one acquisition protocol. And team turnover transfers knowledge through the case library rather than through the presence of the original people, which tests the real quality of that library. Those three share a property: they reveal what the protocol left implicit. A rule that worked because the whole team shared a context stops working as soon as that context changes. The remedy is simple to state and demanding to hold. Write for a reader who was not there, and verify that property periodically by having someone outside the project apply the protocol on a few cases. The questions they ask identify exactly the implicit assumptions, and that check costs half a day for a benefit appearing months later.Testing the protocol before production
A medical image segmentation protocol is a hypothesis about what annotators will understand, and hypotheses are testable cheaply before they become expensive. The test is simple. Give the protocol, with nothing else, to two people who did not write it, and have them annotate the same ten studies chosen to include difficult cases. Then measure their agreement and, more importantly, list the questions they asked. Those questions are the output that matters. Each one identifies a place where the protocol left something implicit, and each is cheaper to fix now than after a thousand studies. Agreement between the two testers is a secondary signal: high agreement on a badly specified protocol simply means both invented the same interpretation, which is luck rather than evidence. One refinement improves the test considerably. Have one tester be someone with clinical background and the other without. The questions differ, and the gap between them shows exactly which parts of the protocol rely on knowledge the production team may not have. That difference is invisible when the protocol is only reviewed by people who already know the domain.The most common mistakes
These failures recur often enough across corpus projects that naming them is usually enough to avoid them.- Answering a disagreement with an additional paragraph of text.
- Illustrating the protocol with ideal images rather than ones from the corpus.
- Listing what to include without detailing what to exclude.
- Recording a rule without its reason, which prevents generalisation.
- Not stating what to do when in doubt.
- Writing the protocol once and never reopening it.
- Giving an arbitration individually without recording or circulating it.
- Qualifying an annotator on a standard case rather than on difficult ones.
- Not versioning the protocol or linking it to the batches produced.
- Documenting the acquisition equipment and neglecting annotation practices.
