A quality assurance arrangement is judged on one thing alone: what it makes visible and what it lets through. Most of them measure with precision what already exists in the deliverable and stay silent on what does not appear there.
This article sets out that arrangement. It extends the article on the cost of labeling.
What the control must establish
Five questions structure a data labeling arrangement. The proportion of elements plainly present and yet not described. The proportion of elements described without existing anywhere. The correctness of the categories effectively assigned. The agreement obtained between two operators on the same data. And the consistency of the treatment given to limiting cases. One important practical consequence follows. The first question is the most expensive to measure and the most useful, omission appearing only if someone goes over the same data again from the start.What double description brings
Five pieces of information follow from one data labeling batch handled twice. The rate of elements seen by only one of the two operators involved. The degree of agreement obtained on the elements seen by both. The bias proper to each of the operators mobilised. The list of cases where the instruction was read differently. And an estimate of the omission rate across the whole corpus. One important observation follows for a data labeling project. The fourth piece is often worth more than the others, each divergence designating a point of the reference to rewrite, which turns a control measurement into a scoping tool.What the global figure masks
Five breakdowns reveal what a single rate always conceals. By category, isolating the rare categories in particular. By operator, to separate bias from dispersion. By batch, to spot one data source in particular. By difficulty, separating common cases from limiting ones. And by period, to detect any drift over time. One practical consequence follows. Those five breakdowns cost a few minutes of computation and change the conclusion, an excellent global rate being able to cover a complete failure on a whole population.What the two opposite errors cost
Five differences separate the missed element from the one wrongly described. The missed element stays perfectly invisible in the deliverable. The element in excess is spotted instead at a single glance. The first teaches the system to ignore what it should see. The second teaches it to see what does not exist. And their relative weight depends entirely on the foreseen use. One important practical consequence follows for a data labeling project. The last difference directs the whole arrangement, an alert detection suffering above all from the first defect whereas a count suffers as much from both, which makes control coverage depend on the need rather than on any general rule.What the audit sample requires
Five properties make a data labeling control defensible. A genuinely random draw rather than a considered selection. Coverage of every category, including the rarest. A representation of difficult cases proportionate to the corpus. A size sufficient for the rates calculated to mean something. And a reference description produced independently. One observation follows. The first property conditions all the others, a sample proposed by the party being evaluated necessarily holding the data they handle best.What the report must contain
Five elements compose a usable data labeling report. The fraction of the corpus that was genuinely and effectively controlled. The draw method finally adopted for that sample. The rates broken down by category as well as by operator. The omission rate measured in a wholly separate way. And the list of convention divergences effectively recorded. One important practical consequence follows for a data labeling project. The second element is almost always missing and conditions the reading of the others, a rate calculated on a chosen sample not having the same reach as one calculated on a random draw.What the frequency of control changes
Five effects follow the rhythm at which the measurement is conducted. A control conducted at the very end of a project permits no useful correction. A control conducted on the first batch still directs the reference. A regular control detects drifts before they take hold. A control made wholly permanent becomes a disproportionate burden. And a control spaced too far apart lets whole batches pile up for rework. One important observation follows for a data labeling project. The second effect is worth several times the third, a measurement conducted on the first data correcting a convention before it is applied to the whole corpus, which makes the control’s calendar as determining as its coverage.What the agreement rate reveals about the team
Five readings follow from an agreement tracked over time. A low agreement from the very outset signals an incomplete reference. A rising agreement signals training that is at last producing its effects. A falling agreement signals a drift or else a different batch. A high agreement with a high omission rate signals a problem of method. And a perfect agreement on a difficult batch signals an over-permissive instruction. One important practical consequence follows for a data labeling project. The fourth reading is the most useful and the least obvious, two operators being able to agree perfectly on what they see while missing the very same elements, which shows that a high agreement never guarantees exhaustiveness.What the measurement really costs
Five items compose the control expenditure. The double description of a fraction of the corpus. The time spent computing as well as reading the indicators. The rereading of the cases flagged by the measurement. The rework of the elements effectively set aside. And drafting the report finally handed to the client. One important observation follows for a data labeling project. The first item explains nearly the whole gap between two arrangements, measuring omission presupposing that the same data is handled twice whereas every other indicator is calculated on the existing deliverable, which makes that single decision the principal budgetary choice of the control.What the measurement will never say
Five questions escape every indicator. Whether the convention adopted genuinely suited the foreseen use. Whether the corpus covers the situations the system will encounter. Whether the categories adopted correspond to the client’s real need. Whether the threshold adopted has not excluded what mattered. And whether the starting data was the right data. One important practical consequence follows for a data labeling project. Those five questions belong to scoping and not to execution, a corpus perfectly compliant with an unsuitable convention obtaining excellent rates and staying unusable, which shows that no measurement recovers a decision taken before the first item of data.Three errors proper to measurement
Three data labeling defects belong to the arrangement itself. An agreement measured with no omission counted at all. A global rate published with not the slightest breakdown. And an audit sample proposed by the party being evaluated. Those three defects produce perfectly presentable reports, they reveal themselves at the moment the corpus is used, and their common point is to bear on the way of counting rather than on the work counted.What the client must specify here
Five pieces of information come from the client rather than from the provider. The type of error that costs them most dearly in their use. The categories whose correctness matters more than the others. The precise acceptance threshold beyond which a batch is refused. The control coverage they wish to fund. And the party who will actually draw the audit sample. One important observation follows for a data labeling project. The first piece redirects the whole arrangement, a client indifferent to elements in excess and sensitive to omissions justifying an expensive double description rather than a quick rereading, which makes that answer more structuring than the budget announced.What the operator must know here
Five kinds of knowledge help the team work alongside the measurement. The exact meaning of the indicators calculated on their work. The precise threshold below which an element is reworked. The difference between a divergence of convention and an error. The value of flagging a case rather than waiting for the control. And the very purpose of the arrangement, which is never to sanction. One important practical consequence follows for a data labeling project. The third kind changes the relation to control, an operator who understands that a divergence designates a vague instruction rather than a personal fault accepting rereading far more readily, which makes the measurement useful instead of making it threatening.What this chapter shares with the previous ones
Three findings recur from the start of this course. A global figure always masks a failure on one population. The dominant defect stays absent from the deliverable itself. And a divergence between operators designates the reference rather than a fault. One important observation follows for a data labeling project. The third finding finds its most direct use here, each divergence recorded during the measurement supplying a free correction of the reference, which makes the control a scoping arrangement as much as a verification one.What this subject has that is particular
Four traits distinguish this question from the course’s others. It produces the figure on which everyone will decide. It is conducted after the work and not during it. It can be well conducted and stay silent on what matters. And it inspires a confidence its reach does not always justify. One important practical consequence follows for a data labeling project. The last trait explains this chapter’s place, a published figure acquiring an authority its method of calculation does not necessarily deserve, which makes reading a report critically as important as producing one.What comparison between corpora presupposes
Five conditions make two data labeling corpora comparable. One same description convention applied to both. One same threshold deciding what is described and what is not. One same rule for partly observable elements. One same draw method for the control samples. And one same definition of the indicators published. One important observation follows for a data labeling project. Those five conditions are rarely met and never verified, two corpora produced by different teams showing rates of comparable appearance without anything authorising them to be set side by side, which makes comparison between providers far more fragile than it looks.What the contract must provide for here
Five clauses deserve writing down before the start. The indicators finally adopted as well as their method of calculation. The acceptance threshold as applied batch by batch. The fraction of the corpus submitted to control. The party who draws the audit sample. And the treatment of rework caused by a change of reference. One important practical consequence follows for a data labeling project. The fourth clause avoids the most awkward disagreement, a dispute arising afterwards over the sample’s representativeness being settled by no rule at all if nothing was written, which leaves each party suspecting the other without either having acted badly.What this chapter presupposes of the provider
Five practices distinguish an honest control from a decorative one. Publishing the least flattering rate rather than the most presentable. Accepting that the client draws the audit sample themselves. Breaking down the rates even when the breakdown hurts the result. Flagging a drift before the client observes it. And maintaining the control coverage when the schedule tightens. One important practical consequence follows. The last practice is the first to be sacrificed under pressure, a control lightened to hold a deadline producing a deliverable that passes acceptance, which simply shifts the problem to a moment when nobody will be able to attribute it to that decision.What the first batch must establish
Five results justify a control from the very start. The initial agreement rate obtained between two operators. The full list of cases the reference did not settle. The omission rate measured on a sample entirely gone over again. The dispersion observed between the very first batches produced. And the real time the control itself comes to consume. One important observation follows for a data labeling project. The last result serves to size all the rest of the project, a control whose real cost proves double the estimate forcing a choice between reducing its coverage and revising the budget, which is better settled on the first batch than on the last.What the provider brings here
Four contributions distinguish a serious data labeling control. An omission rate measured and then published rather than left unsaid. Rates broken down by category, by operator as well as by batch. A sample drawn at random and not proposed by the producer. And an exploitation of the divergences to correct the reference. One important practical consequence follows. The first contribution apparently hurts the provider, a published omission rate necessarily making a poorer showing than a high agreement, whereas it designates the one measurement the client cannot obtain otherwise and grounds a confidence no flattering rate builds.Approaching a quality arrangement
Five questions scope a data labeling arrangement. Is omission measured. Otherwise the report is incomplete. Are the rates broken down. Otherwise they mask the failures. Who draws the sample. That decides its reach. What fraction is covered. It conditions the reliability. And are divergences exploited. They correct the reference. Those five answers determine the arrangement’s value. Asking them before setting it up avoids a reassuring control that describes nothing.The question that frames the arrangement
One question determines the control coverage in data labeling. Does the use suffer more from a missed element or from one wrongly described. The missed element, an alert not triggered or an incomplete inventory, requires measuring omission before anything else and paying the price of a double description. The element in excess, an untimely alarm or an inflated count, is spotted on rereading and costs markedly less to control. That question belongs to the consequences of the defect and not to its frequency, it is asked before the first control, and it decides the share of budget devoted to measurement.Three decisions before measuring
Three decisions commit a data labeling arrangement. Measuring omission separately rather than relying on agreement alone. Breaking down the rates by category from the very first report. And having the sample drawn by a party other than the producer. Those three decisions cost a few hours, they precede the first control, and their absence produces reassuring reports that describe nothing.Three checks on a quality report
Three controls qualify a data labeling report. The presence of an omission rate measured in a wholly separate way. The breakdown of the rates by category as well as by operator. And an indication of the draw method finally adopted for the sample. Those three controls are each asked in one question, they require no technical competence, and their absence indicates a report whose main figure does not describe what one believes.Three questions before an announced rate
Three questions precede any conclusion drawn from a quality figure. On which population was this rate calculated. Was omission counted, or only agreement. And who chose the data in the sample. Those three questions are asked in a minute, they require no technical competence, and their absence leads to deciding on a figure whose reach nobody knows.What this chapter teaches
One cross-cutting observation deserves closing this examination. An arrangement never measures anything but what it can see. Three findings compose it. Omission appears only by going over the same data again. An excellent global rate can cover a complete failure. And each divergence designates a point of the reference to rewrite. That finding extends the preceding chapters, measurement becoming a scoping tool as much as a verification one.What this chapter leaves to the next
A control arrangement presupposes access to the data it examines. Three questions stay open in data labeling. What sensitive data imposes as precautions. Why a certification does not on its own guarantee security. And how to verify what a provider asserts on this point. Those three questions belong to security requirements, which constitute the subject of the following chapter.Why the omission measurement gets dropped first
One point about what survives budget pressure belongs at the close. When a control arrangement is trimmed, omission is the first thing cut. Three reasons follow in data labeling. It costs more than every other indicator put together. Its absence changes nothing visible in the report. And no client has ever refused a delivery for lacking it. One important practical consequence follows for a data labeling provider. Defending it as the last item to cut rather than the first is worth the argument, since every other indicator describes the work that was done and this one alone describes the work that was not, and a corpus judged solely on what it contains has never been judged at all.What the client keeps from a good report
Four uses outlive the delivery itself. A basis for comparing the next corpus with this one. A record of which categories proved hardest to describe. An explanation of why certain conventions were adopted. And a starting point for whoever extends the work later. One important observation follows for a data labeling project. The second use is the least anticipated and the most valuable, a record of where operators disagreed telling a future team exactly where their own reference will need to be clearer, which spares them rediscovering the same difficulties one by one.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Judging a corpus on a global rate with no breakdown.
- Measuring agreement without measuring omission.
- Letting the producer choose the audit sample.
- Confusing a divergence of convention with an error.
- Neglecting the rare categories in the calculation.
- Controlling at the end of a project rather than on the first batch.
- Ignoring divergences rather than exploiting them.
- Concluding from a figure without knowing its calculation population.
- Entrusting the control to whoever produced the work.
- Lightening the arrangement because the first batches were good.
