Not all description errors are equal. Some are corrected in a day, others are discovered months later and force the whole corpus to be reworked. What separates them rarely comes down to their frequency.
This article sets out that ranking. It extends the article on synthetic data.
What makes an error expensive
Five factors determine the price of a data labeling defect. The precise moment it becomes visible in the project. The exact share of the corpus it affects simultaneously. Its capacity to repeat itself in a perfectly systematic way. The difficulty of spotting it through a plain ordinary control. And the work needed to correct it once identified. One important practical consequence follows. The first factor dominates all the others, one same defect costing a few hours if spotted on the first batch and several months if discovered in real use.What omission costs
Five characteristics make this the most expensive data labeling defect. It appears nowhere in the deliverable finally produced. No control bearing on the existing alone detects it. It teaches the trained system to ignore what it should see. Measuring it presupposes going over the same data entirely again. And correcting it requires reopening the whole corpus. One important observation follows for a data labeling project. The second characteristic explains its persistence, an arrangement verifying the correctness of what appears in the corpus staying perfectly silent on what does not appear there.What a mistaken convention produces
Five effects follow a rule badly defined at the outset. It is applied in a perfectly consistent way throughout. Every agreement indicator comes to validate it without reserve. It affects the whole corpus and not a mere fraction. It reveals itself only at the moment of the system’s real use. And correcting it demands an entirely complete rework. One practical consequence follows. The second effect makes it formidable, a corpus faithfully applying a bad convention obtaining better scores than a corpus hesitating over the right one.What systematic bias brings about
Five properties distinguish this data labeling defect from ordinary noise. It always leans in exactly the same direction. It never offsets itself between operators. It passes wholly into the system finally trained. It goes entirely unnoticed as long as one looks at a plain average. And it is corrected very badly once the corpus is produced. One observation follows. The fourth property explains how it survives control, a constant offset staying invisible in a global rate whereas a breakdown by operator makes it appear at once.What a selection defect produces
Five consequences follow a corpus badly chosen at the outset. The system learns a world very different from the one it will meet. The indicators nonetheless stay excellent on the corpus itself. The failure shows itself on the very first real data. No control bearing on the description could have signalled it. And correcting it requires an entirely complete new acquisition. One important practical consequence follows for a data labeling project. The fourth consequence places this defect beyond the annotation provider’s reach, a perfectly described corpus staying unusable if the starting data did not represent the use, which makes that responsibility entirely the client’s.What the rare category reveals
Five observations concern the least represented classes. They weigh very little in a global quality rate. They suffer most from a definition that stayed vague. They nonetheless often concentrate the use’s real interest. They improve very badly through a plain increase in volume. And their failure ends up passing for an acceptable result. One important practical consequence follows for a data labeling project. The third observation creates the central paradox, a corpus excellent on common situations failing precisely on those for which the system had been ordered.What the moment of discovery changes
Five stages mark the growing cost of one same defect. Spotted at the trial batch, it is corrected before any production. Spotted on the first batch, it costs the rework of that one batch. Spotted mid-project, it costs the rework of the earlier batches. Spotted at the moment of delivery, it costs a full renegotiation. And spotted in use, it costs the corpus as well as the time already spent. One important observation follows for a data labeling project. Those five stages separate orders of magnitude rather than percentages, which justifies devoting to the first control an effort disproportionate to the volume it examines.What drift mid-project produces
Five observations characterise this particular defect. The corpus stays perfectly consistent within each batch. The first batches and the last apply two conventions. No control bearing on a single isolated batch detects it. It appears only when batches distant in time are compared. And correcting it requires choosing which of the two rules to keep. One important practical consequence follows for a data labeling project. The third observation explains its frequency, a control arrangement verifying each batch as it is produced successively validating two incompatible conventions without ever setting them against each other.What correcting a defect demands
Five pieces of work compose a serious rework. The precise identification of the population genuinely affected. The correction of the reference before any effective rework. The rework of all the batches produced under the old rule. A new measurement verifying that the defect has indeed gone. And a written record of what was modified as well as the reasons. One important observation follows for a data labeling project. The first piece of work decides the cost of all the others, a badly delimited population leading either to reworking far more than necessary or to leaving part of the defect in place, which makes that delimitation more determining than the rework itself.What the client must watch here
Five signals announce an expensive defect in the making. A high agreement rate accompanied by no measure of omission. A report presenting only one single global figure. A complete absence of reported cases for several weeks. A reference that has stayed fixed since the project’s start. And rare classes whose results are never presented separately. One important practical consequence follows for a data labeling project. The third signal reads against intuition, an absence of questions rarely signalling a perfect reference and more often operators settling matters alone, which prepares exactly the kind of defect this chapter describes.What this chapter shares with the previous ones
Three findings recur from the start of this course. A global figure always masks a failure on one population. The dominant defect stays absent from the deliverable itself. And scoping decides far more than execution. One important observation follows for a data labeling project. Those three findings designate the same arrangement, a measurement designed to verify what appears in the corpus being unable by construction to say anything about what should have been there, which explains why the most expensive defects survive rigorous controls.What this subject has that is particular
Four traits distinguish this question from the course’s others. It is not handled at the moment it occurs. It ranks defects by cost rather than by frequency. It designates defects the ordinary control validates. And it bears on what the corpus does not contain. One important practical consequence follows for a data labeling project. The second trait reverses the usual order of priorities, a rare and structuring defect deserving more attention than a frequent and scattered one, which contradicts the intuition governing most control arrangements.What a documentation defect costs
Five consequences follow a corpus delivered without its reference. Nobody knows any longer which convention had been applied. A later extension produces a corpus that ends up inconsistent. The limiting cases already settled all have to be rediscovered. A disagreement bearing on quality can no longer be adjudicated. And the corpus loses part of its value with each year that passes. One important observation follows for a data labeling project. This defect degrades no data at all and yet costs a great deal, a perfectly described corpus becoming unusable as soon as nobody knows any more what its categories exactly covered.What the first batch must look for
Five examinations are worth conducting before any volume production. A complete rework of a sample in order to measure omission. A breakdown of the gaps by operator and by category. An entirely separate examination of the least represented classes. A confrontation of the limiting cases met with the written rules. And a verification that the data really matches the foreseen use. One important practical consequence follows for a data labeling project. Those five examinations each bear on one of the defects described in this chapter, which makes the first batch the only moment when all of them are still corrected for a negligible cost.What a format defect brings about
Five observations concern this purely technical defect. It affects in no way the correctness of the descriptions produced. It nonetheless prevents any immediate use of the corpus. It is discovered at the client’s very first reading. It is most often corrected by a plain automatic conversion. And it sometimes costs several weeks when essential information is missing. One important observation follows for a data labeling project. The last observation distinguishes two very different situations, a structural conversion being settled in hours whereas information never collected forces the corpus to be reopened, which makes validating the format as urgent as validating the convention.What these defects really cost
Five items compose the expenditure of a late rework. The description work entirely redone on the population concerned. The quality control wholly applied a second time. The training time definitively lost on the defective corpus. The delay taken by everything that depended directly on this corpus. And the confidence lastingly damaged between the two parties. One important observation follows for a data labeling project. The fourth item often exceeds all the others combined, a technical team having waited six months for a usable corpus losing more in calendar terms than the cost of the rework itself would suggest.What one operator’s defect produces
Five observations concern a constant individual gap. It affects only the share of the corpus that person handled. It stays perfectly invisible in a rate calculated on the whole. It appears instead immediately as soon as one breaks down by person. It is corrected by a rework targeted on those batches alone. And it almost always reveals an instruction that was misunderstood. One important practical consequence follows for a data labeling project. The last observation changes how this defect is handled, a constant individual gap designating a point of the reference that person read differently, which calls for a written clarification rather than a reminder addressed to them alone.What this ranking changes about the organisation
Five consequences follow a reading by cost rather than by frequency. The first batch mobilises an effort disproportionate to its volume. The control looks first for the absent before verifying the present. The breakdowns now precede the reading of the global rate. Rare classes are systematically given a separate examination. And comparing the older batches with the most recent becomes periodic. One important observation follows for a data labeling project. Those five consequences cost almost nothing once installed, most of the effort lying in deciding to apply them rather than in conducting them, which makes them accessible to projects of any size.What the provider brings here
Four contributions prevent the most expensive defects. A control conducted from the first batch rather than at delivery. A measure of omission published from the project’s start. A periodic comparison between batches distant in time. And a flagging of every rare class handled uncertainly. One important practical consequence follows. The third contribution appears in no brief and avoids the most discreet defect, a convention drift showing itself only when two batches produced weeks apart are set side by side.What this chapter presupposes of the provider
Five practices distinguish honest production from reassuring production. Looking for omission rather than waiting for the client to ask. Flagging a drift before the client observes it. Comparing batches distant in time with nothing requiring it. Isolating rare classes even where they degrade the result. And proposing a rework rather than delivering a visibly doubtful corpus. One important practical consequence follows. The last practice costs immediately and protects lastingly, a corpus delivered with a defect known to the provider alone always ending up discovered, which turns an owned rework into a dispute nothing defuses.Approaching a corpus’s defects
Five questions scope the defects of a data labeling corpus. Is omission measured. Otherwise it stays invisible. Was the convention tested. It affects the whole corpus. Are the rates broken down. A bias always hides there. Are rare classes isolated. They carry the real stake. And did the control happen early. The cost depends on the moment. Those five answers determine the risk a corpus carries. Asking them early avoids a rework nothing made inevitable.The question that ranks a defect
One question determines the cost of a data labeling defect. Is the defect spread at random or carried by a rule. Spread at random, an inattention or an isolated hesitation, it dilutes in the volume and is corrected by a targeted rework. Carried by a rule, a misunderstood convention or an operator constant in their gap, it affects everything belonging to that rule and is corrected badly. That question belongs to the defect’s structure and not to its frequency, it is asked from the first control, and it separates a rework of a few hours from one of several weeks.Three decisions that avoid the most expensive defects
Three decisions protect a data labeling project. Measuring omission from the first batch rather than at the project’s end. Testing the convention on a sample before fixing it. And breaking down every rate by operator and by category. Those three decisions cost a few days, they precede volume production, and their absence exposes the project precisely to the three defects nothing recovers afterwards.Three checks on a delivered corpus
Three controls qualify a data labeling corpus. The presence of an omission rate measured separately. The breakdown of the results by operator and by category. And a comparison between the first batches and the last. Those three controls are each asked in one question, they require no technical competence, and their absence leaves in place precisely the defects no partial rework will correct.Three questions before a corpus received
Three questions precede accepting a deliverable. What omission rate was measured, and on which sample. Do the results vary by operator or by category. And do the first batches resemble the last. Those three questions are asked in a few minutes, they require no technical competence, and their absence leads to accepting a corpus whose defects will reveal themselves when nothing can correct them.What this chapter teaches
One cross-cutting observation deserves closing this examination. A defect’s cost lies in the moment of its discovery. Three findings compose it. Omission appears in no control of the existing. A bad convention applied faithfully obtains good scores. And a systematic bias stays invisible in an average. That finding extends the preceding chapters, the most expensive defects being precisely those the usual arrangement does not look for.What this chapter leaves to the next
These defects occur in work that people carry out. Three questions stay open in data labeling. What working conditions change about the quality produced. What an ordering party owes those who describe their data. And how these questions arise within a subcontracting chain. Those three questions belong to the ethics of labeling, which constitutes the subject of the following chapter.Why the early control is the hardest to defend
One point about how projects get scheduled belongs at the close. A heavy control on the first batch always looks like a poor use of time. Three reasons follow in data labeling. The batch is small and the effort seems out of proportion to it. Nothing has gone wrong yet, so there is nothing visible to justify it. And the schedule pressure is highest exactly at the start. One important practical consequence follows for a data labeling provider. Holding that control in place is worth the argument it costs, since every defect in this chapter is cheap on the first batch and expensive on the last, and no client has ever regretted the days spent there once a corpus reached its intended use intact.What each party can see here
Four asymmetries shape who notices which defect. The provider sees the operators’ hesitations first. The client alone knows whether the data matches their use. The provider can compare batches across the whole project. And only the client observes what fails in production. One important observation follows for a data labeling project. The second and fourth asymmetries sit entirely on the client’s side, which means the two defects with no possible remedy late, a corpus that never represented the use and a failure only real usage reveals, depend on information the provider cannot obtain by working harder.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Measuring correctness without ever measuring omission.
- Fixing a convention without having tested it on a batch.
- Reading a global rate with no breakdown at all.
- Judging a corpus on its most frequent classes.
- Postponing the first control to the project’s end.
- Confusing a systematic bias with random noise.
- Treating a rare class as an ordinary one.
- Accepting a high agreement as proof of quality.
- Correcting a defect without reworking the earlier batches.
- Neglecting a gap because it bears on few elements.
