A labeling instruction is judged not on its precision but on its reproducibility. Two people reading it must produce the same thing, and that criterion eliminates most documents drafted with the greatest care.
This article sets out that drafting. It extends the article on security requirements.
What makes an instruction inapplicable
Five failings recur in data labeling references. A definition that describes the object without saying what to do with it. A term admitting two readings depending on the reader. An appeal to common sense rather than an explicit decision. An accumulation of rules no memory retains. And examples bearing only on the obvious cases. One important practical consequence follows. The third failing is the most widespread, an instruction asking for discernment to be exercised saying nothing verifiable and leaving the rule to each person’s judgement.What defining a category requires
Five elements make a data labeling category usable. What it includes, stated positively. What it excludes, stated just as explicitly. Its boundary with the nearest neighbouring category. A typical example and a close counter-example. And the conduct to adopt where membership stays doubtful. One important observation follows for a data labeling project. The second element is almost always missing, a purely positive definition leaving each person to decide alone what falls outside the category, which produces different boundaries depending on the operator.What the example transmits
Five properties distinguish an example from a plain written formulation. It admits only one possible reading. It is consulted in a second in the course of the work. It shows the limiting case instead of attempting to describe it. It compares directly with the work under way. And it passes from one person to another with no loss. One practical consequence follows. Those five properties make a few examples more effective than several pages, an annotated counter-example standing as a rule with no wording interposing itself.What the vocabulary used decides
Five precautions make a formulation reproducible. Using one single term to designate one same thing. Avoiding adjectives whose threshold is not defined. Preferring an imperative instruction over a description. Writing thresholds as figures rather than as appraisals. And banning formulations that presuppose a shared judgement. One important practical consequence follows for a data labeling project. The second precaution alone eliminates a large share of divergences, an adjective such as significant or visible carrying a different threshold in each reader, which turns an apparently clear rule into a source of dispersion.What the document’s structure presupposes
Five parts compose a usable data labeling reference. A reminder of the final use the corpus will serve. The definition of each category with its examples. The description rules, illustrated rather than described. The catalogue of limiting cases already settled. And the conduct to adopt before a case absent from the catalogue. One observation follows. The first part is often neglected and changes the quality of adjudications, an operator who understands their work’s destination settling new cases far better.What testing an instruction establishes
Five observations follow from a reference tested before production. Two operators handle the same data without conferring. The divergences observed designate the sentences to rewrite. The questions asked designate the points purely absent. The time spent reveals the rules too costly to apply. And the unforeseen cases feed the catalogue from the outset. One important practical consequence follows for a data labeling project. The third observation is worth as much as the second, each question signalling a gap the drafting had left without any disagreement revealing its existence.What the case catalogue becomes
Five properties characterise this part of the reference. It is empty at the start and grows richer every week. It grows fast at first and then progressively stabilises. Each entry definitively settles a case already met. It is consulted by case rather than by theme. And it eventually constitutes the very heart of the document. One important practical consequence follows for a data labeling project. The second property gives a valuable progress indicator, a catalogue still growing at the same pace after several weeks signalling insufficient initial scoping rather than a particularly varied corpus.What revising the reference imposes
Five precautions accompany any modification mid-project. Dating the version and keeping all the previous ones. Explicitly flagging what has just changed. Stating whether the earlier batches are concerned. Notifying every operator simultaneously. And verifying the change’s application on the next batch. One important observation follows for a data labeling project. The third precaution avoids a costly mixture, a reference modified without saying whether earlier work must be reworked producing a corpus governed by two superimposed conventions, which reproduces exactly the defect the revision meant to correct.What operator reports bring
Five conditions make this circuit effective. A simple way of flagging a case without interrupting the work. A short response time on the cases reported. A decision recorded in writing rather than an oral answer. An immediate circulation of that decision to the whole team. And a systematic addition to the catalogue of settled cases. One important observation follows for a data labeling project. The second condition determines all the rest, an operator waiting several days for an answer settling the matter alone meanwhile, which produces a precedent the later decision will contradict without anyone reworking what has already been done.Three errors proper to drafting
Three data labeling defects belong to the reference itself. A rule formulated with no example illustrating it. A case settled orally and never recorded in writing. And a document whose length discourages any consultation. Those three defects are not visible in the document, they reveal themselves in the dispersion of the data produced, and their common point is to let a rule exist without it being genuinely transmitted.What the client must supply here
Five elements come from the client rather than from the provider. The business definition of each category of the reference. The cases they already know to be ambiguous in their own field. The precise use the corpus is destined for. The vocabulary used by their internal teams. And a contact able to settle a new case quickly. One important practical consequence follows for a data labeling project. The last element conditions the pace of the whole project, a case reported with no identified recipient staying in suspense while production continues, which makes the availability of an arbiter more determining than the quality of the initial document.What training adds to the document
Five contributions do not pass through the written text. The commented handling of a few items in front of the operator. The comparison of their work with a reviewer’s. The immediate correction of the very first habits taken up. The discovery of limiting cases directly in real conditions. And an explanation of the reasoning behind each demanding rule. One important observation follows for a data labeling project. The third contribution alone justifies this accompaniment, a habit corrected on the first day costing minutes whereas the same habit discovered three weeks later will have produced a bias across several thousand elements.What this chapter shares with the previous ones
Three findings recur from the start of this course. An example transmits a convention better than a paragraph. A divergence between operators designates the reference rather than a fault. And an unwritten rule is replaced by as many rules as there are operators. One important practical consequence follows for a data labeling project. Those three findings all converge on the same work, the drafting of the reference appearing in hindsight as the point where most of what the other chapters then tried to measure or correct was actually decided.What this subject has that is particular
Four traits distinguish this question from the course’s others. It costs a few days and decides several months of work. It is judged on the dispersion produced and not on its own quality. It goes on being written throughout production. And nobody explicitly asks for it in a brief. One important observation follows for a data labeling project. The last trait explains its frequent absence, a client ordering described data and not a reference, which leaves the provider responsible for work nothing in the order made compulsory.What the reference becomes after the project
Five uses outlive it once the corpus is delivered. It permits extending the corpus without breaking its consistency. It explains to a third party what the delivered data means. It serves as a basis for a neighbouring project in the same field. It documents decisions nobody remembers any longer. And it considerably speeds up training a new team. One important practical consequence follows for a data labeling project. The fourth use reveals itself two or three years later, a renewed team taking up a corpus whose conventions survive only in this document, which makes it the item whose value grows as the project’s memory fades.What the reference changes about the project’s economics
Five effects carry through to the budget in data labeling. Throughput improves as soon as cases stop interrupting the work. The rework rate falls as the rules stabilise. Control concentrates on the genuinely ambiguous points. Training an additional operator becomes markedly shorter. And the client’s own arbitration time falls sharply. One important observation follows for a data labeling project. The last effect interests the client more than the others, a mature reference dividing the number of requests they receive, which frees precisely the resource whose availability conditions the whole project.What the tool can carry of the reference
Five functions bring the rule closer to the hand in data labeling. A reminder of the convention displayed during the work. Direct access to the catalogue from the production screen. A case report sent without leaving the item being handled. A technical constraint forbidding a non-compliant entry. And a dropdown list limiting the categories available. One important practical consequence follows for a data labeling project. The fourth function advantageously replaces a written rule, a threshold imposed by the tool dispensing with any memorisation, which shows that part of the reference gains from becoming a constraint rather than a sentence.What the reference cannot resolve
Five difficulties persist whatever the document’s quality. A genuinely undecidable case on the data stays undecidable. Fatigue at a session’s end degrades how the rules are applied. A badly defined final use makes every adjudication arbitrary. A category badly chosen at the outset stays so to the end. And a disagreement between business experts is not settled in writing. One important observation follows for a data labeling project. The first difficulty deserves being owned rather than denied, an honest reference being able to prescribe flagging a case rather than settling it, which is worth more than a rule invented to avoid admitting that the data says nothing.What the first week must produce
Five results justify a cautious start. A dozen limiting cases settled and recorded. A first agreement rate measured between two operators. A list of the wordings rereading has revealed to be ambiguous. A corrected version of the reference circulated to the team. And a realistic estimate of the throughput attainable. One important observation follows for a data labeling project. Those five results are worth more than a volume produced quickly, a week devoted to stabilising the rules sparing rework across everything that follows, which makes a slow start markedly more profitable than an energetic one.What the provider brings here
Four contributions distinguish a professional data labeling reference. A first version drafted before production rather than afterwards. A catalogue fed by the operators’ daily reports. A test of the reference conducted on a batch before fixing it. And handing the document to the client with the delivered corpus. One important practical consequence follows. The last contribution is neglected and costs dearly, a corpus delivered without its reference becoming unusable for anyone wishing to extend it later, which turns an internal working document into a deliverable in its own right.What this chapter presupposes of the provider
Five practices distinguish a living reference from a dead document. Writing a first version before the client asks for one. Recording each adjudication rather than passing it on orally. Rereading the document when dispersion rises. Delivering it with the corpus without waiting to be asked. And flagging to the client the points their own need leaves undefined. One important practical consequence follows. The second practice demands a discipline nothing imposes, an adjudication given in three words in conversation settling the case of the day without settling anything for what follows, which explains why so many projects answer the same question several times over.Approaching the drafting of a reference
Five questions scope this drafting on a data labeling project. Is the final use recalled. It guides the adjudications. Are the exclusions written. They are often missing. Are the rules illustrated. Text alone does not suffice. Is the unforeseen case handled. That is the missing point. And does the document fit in few pages. Otherwise it will not be read. Those five answers determine the reference’s applicability. Asking them before production avoids a document nobody will open twice.The question that frames the drafting
One question determines the useful form of a data labeling reference. Will the document serve to train or to settle during the work. Training presupposes continuous text, explaining the final use and the logic of the rules, read once at the start and then rarely reopened. Settling presupposes a catalogue consultable in seconds, organised by case rather than by theme, opened several times a day. That question belongs to the moment the document is used, it is asked before writing the first line, and it explains why a single reference regularly fails at both uses.Three decisions before drafting
Three decisions commit a data labeling reference. Separating the rules to memorise from the catalogue meant for consultation. Writing each category’s exclusions alongside its definition. And providing from the outset the conduct to adopt before an unforeseen case. Those three decisions cost one morning, they precede the first item handled, and their absence produces a document the operators will stop opening by the second week.Three checks on a reference
Three controls qualify a data labeling instruction. The presence of written exclusions for each category. The existence of a conduct to adopt before an unforeseen case. And the date of the last revision made to the catalogue. Those three controls are each asked in one question, they require no technical competence, and their absence indicates a document that will produce divergences nobody will be able to explain.Three questions before fixing a rule
Three questions precede entering a rule into the reference. Would two people read it in exactly the same way. Would one example suffice to replace it entirely. And will an operator be able to apply it without having to think. Those three questions are asked in a minute per rule, they require no particular competence, and their absence produces an accurate document nobody manages to apply in the same way.What this chapter teaches
One cross-cutting observation deserves closing this examination. An instruction is judged on what it settles, not on what it describes. Three findings compose it. An appeal to discernment says nothing verifiable. A definition without exclusions lets each person fix their own boundary. And a counter-example stands as a rule with no wording interposing itself. That finding extends the preceding chapters, most divergences tracing back to a sentence nobody had judged ambiguous.What this chapter leaves to the next
A well written instruction meets uses it had not foreseen. Three questions stay open in data labeling. What a generative model demands as human work. Why preference sometimes replaces description. And how to judge output for which no single answer exists. Those three questions belong to labeling for generative AI, which constitutes the subject of the following chapter.Why exclusions are the hardest part to get written
One point about how references get drafted belongs at the close. Clients write what a category contains and stop there. Three reasons follow in data labeling. The inclusions are what they think about when defining the need. The exclusions only become visible once someone gets them wrong. And listing what something is not feels like stating the obvious. One important practical consequence follows for a data labeling provider. Asking for the exclusions explicitly is worth the awkward conversation, since every boundary left unwritten becomes a decision an operator makes alone, and the client discovers their own definition only when shown two plausible readings of it side by side.What a good reference gives the client back
Four returns come from the drafting exercise itself. Their own need stated more precisely than before. The ambiguities in their field made explicit. A document their own teams can use internally. And a record of decisions their organisation had never fixed. One important observation follows for a data labeling project. The last return is the least expected and often the most valued, a client discovering that a distinction their teams had argued over for years finally appears in writing, which makes the reference useful well beyond the corpus it was drafted for.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Describing the object instead of saying what to do with it.
- Appealing to common sense rather than settling explicitly.
- Defining a category without writing what it excludes.
- Illustrating only the obvious cases.
- Omitting the conduct to adopt before an unforeseen case.
- Drafting a document too long to be consulted.
- Neglecting the reminder of the corpus’s final use.
- Leaving a case settled orally without writing it down.
- Fixing the reference at the start without keeping it alive.
- Circulating a corrected version without flagging what changed.
