A model learns what it is shown. Data labeling is the work that decides what it will see, and that work is settled far more in the decisions taken before the first image than in the hand movement of the person annotating.This article opens a course devoted to this activity and its conditions.
What this activity covers
Five operations compose a data labeling project.Defining what must be described and at what level.Drafting the reference that settles the ambiguous cases.The production itself, conducted by trained operators.The control measuring what the result is really worth.And delivery in a format the client’s chain can read.One important practical consequence follows. Only the third operation is visible from the outside, which leads to comparing services on their least determining part.What the modality changes
Five families of data call for distinct labeling methods.The still image, where the object is designated by a simple geometry.Video, which adds continuity between the frames.Text, where the boundary between two segments is open to debate.Audio, where the reference becomes purely temporal.And three-dimensional data, which describes a whole scene.One important observation follows for a data labeling project. Those five families share the same underlying difficulties, the convention to write and the omission to measure recurring everywhere, which makes experience gained on one modality largely transferable to the others.What the reference determines
Five points are settled before any data labeling production.The exact scope of what must be described.The classes adopted as well as their mutual boundary.The treatment of cases where the data stays ambiguous.The threshold below which an element is not described.And the conduct to adopt before an unforeseen case.One practical consequence follows. The last point is almost always missing and weighs heaviest, a reference silent on new cases letting every operator invent their own rule.What the operator really brings
Five competences separate a trained annotator from an executant.The fast reading of an instruction as well as its memorisation.The immediate recognition of cases already settled.The reflex of flagging a new case rather than inventing.The regularity of the hand kept across thousands of elements.And an awareness of their own drop in working pace.One important practical consequence follows for a data labeling project. The third competence protects the consistency of the whole corpus, an operator settling an unforeseen case alone creating a parallel convention nobody knows about, which produces work that is consistent per person and inconsistent overall.What the measurement must establish
Five indicators describe the quality of a labeled corpus.The proportion of elements present and not described.The proportion of elements described without existing.The correctness of the categories effectively assigned.The agreement between two operators on the same data.And the consistency of the treatment of limiting cases.One observation follows. The first indicator is the most expensive to produce and the most useful, omission appearing only if someone goes back over the same data, which explains its frequent absence from reports.What the cost covers
Five items compose the price of data labeling work.The initial scoping, the reference and first adjudications included.Training the operators in that precise reference.Production itself, the only item the rate makes visible.The control as well as the rework it triggers.And delivery, conversion and documentation included.One important practical consequence follows. Four items out of five disappear behind the unit rate, which lets two quotations showing the same price cover very different services.What the corpus must cover
Five axes structure a labeled data set.The common situations, which serve as a reference.The difficult cases, where the convention is really tested.The rare categories, systematically under-represented.The least favourable of the acquisition conditions.And data holding no element to describe at all.One important practical consequence follows for a data labeling project. The last axis is almost always neglected, a corpus composed only of rich data producing a system inclined to find everywhere what it was taught to look for, which is corrected by deliberately including empty examples.What the delivery format requires
Five elements usefully accompany a delivered corpus.The format used as well as the convention associated with it.The correspondence between the codes and the categories.The reference actually applied during production.The treatment given to data with no element to describe.And the quality measurements obtained on a sample.One important observation follows for a data labeling project. The third element is almost always neglected and costs dearly, a corpus delivered without its reference becoming unusable for anyone wishing to extend it later, which turns an internal document into a deliverable in its own right.What automation shifts
Four effects accompany a proposal produced by a model.The volume handled manually falls on common cases.Verification comes to partly replace description.Elements the tool ignores remain almost always ignored.And the model’s convention silently imposes itself on the corpus.One important practical consequence follows for a data labeling project. The third effect worsens this activity’s principal defect, an operator checking proposals ceasing to look for what no proposal signals, which concentrates omissions on the population that already suffered most from them.Three errors proper to this activity
Three data labeling defects belong to its very nature.A convention left implicit because it seemed obvious.An omitted element, invisible in the deliverable as in the measurement.And a corpus delivered without the document explaining what it holds.Those three defects are not visible on an isolated example, they reveal themselves at the moment the corpus is used, and their common point is to bear on what surrounds the data rather than on the data itself.What the client must supply
Four elements come from the client rather than from the provider.The final use for which the corpus is destined.The business definition of each category of the reference.The cases they already know to be ambiguous in their field.And a sample of data taken at random rather than chosen.One important practical consequence follows for a data labeling project. The last element decides the whole costing, a carefully selected sample showing clean data whereas the real corpus holds far heavier cases, which produces an accurate estimate for data that will never be handled.What organising the work decides
Five choices weigh as much as the operators’ competence.The length of the sessions and the rhythm of the breaks.The distribution of the batches between the people.The circuit permitting a new case to be reported.The response time on the cases reported.And the circulation of decisions to the whole team.One important observation follows for a data labeling project. The fourth choice conditions all the others, an operator waiting several days for an answer settling the matter alone meanwhile, which produces a precedent the later decision will contradict without anyone reworking what has already been done.What this activity has that is particular
Four traits distinguish data labeling from other services.Its central movement is learnt in an hour and mastered slowly.Its quality cannot be read from the deliverable it produces.Its low unit cost pushes towards handling very large volumes.And its value is decided before the work begins.One important practical consequence follows for a data labeling project. Those four traits combine unfavourably, an easy activity conducted on large volumes with no written scoping producing a massive and barely visible inconsistency, which explains why the most disappointing corpora are often the simplest to produce.What the control must aim at
Five arrangements compose a useful control.A double description of a fraction of the corpus.A sorting of gaps between convention divergence and error.A rereading targeted on the cases near the threshold.A tracking of indicators batch by batch rather than at the end.And an exploitation of divergences to correct the reference.One important observation follows for a data labeling project. The last arrangement transforms the nature of the control, each divergence recorded supplying a free correction of the reference, which makes it a scoping tool as much as a verification one.What security imposes here
Four requirements accompany sensitive data.Access restricted to the people assigned to the project alone.Traceability of consultations as well as of exports.A confidentiality undertaking signed by each operator.And a documented destruction of the data at the project’s end.One important practical consequence follows for a data labeling project. The fourth requirement is often neglected and badly controlled, a copy forgotten on a workstation sufficing to empty the other three of their reach, which makes the exit procedure as important as the entry one.What scale changes
Four effects accompany a project of large scope.The scoping is spread across a far greater volume.Dispersion between operators grows with their number.Control must tighten to offset that dispersion.And training a reinforcement becomes a frequent operation.One important observation follows for a data labeling project. The last effect makes the reference determining, a mature document permitting an operator to be brought in within days where a vague one imposes weeks, which decides the real capacity to absorb a rise in load.What the corpus becomes after the project
Four uses outlive it once delivery is done.It serves as a basis for a successor model.It permits evaluating a system nobody had foreseen.It documents conventions nobody remembers any longer.And it can be extended if its consistency has been preserved.One important practical consequence follows for a data labeling project. The last use depends entirely on decisions taken at the outset, a corpus produced without a written reference no longer being extendable without breaking its consistency, which turns an initial saving into a far greater expense a few years later.What this course will handle
Four families of questions occupy the following chapters.The data modalities and the methods they call for.The organisation of work, of teams and of competences.The operational aspects, cost, quality, instructions and security.And the developments, automation, synthetic data and trends.One important observation follows for a data labeling project. Those four families all return to decisions taken before production, which explains why an activity reputed to be simple occupies fifteen chapters while its central movement is learnt in an hour.What the provider brings here
Four contributions distinguish a serious data labeling service.A reference written before production rather than afterwards.A measurement of omission published rather than left unsaid.A deliberate coverage of the corpus’s difficult cases.And documentation handed over with the delivered data.One important practical consequence follows. The second contribution apparently hurts the provider, a published omission rate necessarily making a poorer showing than a flattering figure, whereas it designates the one measurement the client cannot obtain otherwise and grounds a confidence no favourable indicator builds.What this chapter presupposes of the provider
Four practices distinguish a serious data labeling provider.Writing a reference before the client asks for one.Publishing the least flattering indicator rather than the most presentable.Refusing a commitment on a volume never sampled.And delivering the documentation without waiting to be asked.One important observation follows. Those four practices cost little and get noticed only on the second project, a client who has already worked with a less rigorous provider recognising the difference at once, which makes this rigour more commercial than it looks.What the first batch must establish
Four results justify a trial batch before production.The average number of elements to describe per unit of data.The agreement rate obtained between two operators.The list of cases the reference did not settle.And the real time spent on a representative sample.One important observation follows for a data labeling project. The third result is worth more than the others at the start, each case reported becoming a written rule that will not arise again, which makes the first batch an exercise in drafting the reference as much as a throughput measurement.Approaching a labeling project
Five questions scope a data labeling project.What exactly must be described here. The scope decides the rest.Is the reference written down. Otherwise everyone will invent it.Is the omission measured. It stays invisible otherwise.What format does the chain expect. It is asked well beforehand.And will the corpus serve several distinct uses. That changes everything.Those five answers determine the value of the corpus produced. Asking them before production avoids work nothing will permit verifying.The question that frames the project
One question determines the scale of a data labeling project.Will the corpus serve one precise use or several successive ones.One precise use permits the most economical level of description the need tolerates, the corpus disappearing with the model it will have served.Several uses justify a richer description, each later employment being able to impoverish it towards what it needs whereas the reverse is impossible.That question belongs to the client’s ambition and not to the immediate need, it is asked before the first item of data, and it separates a one-off saving from a reusable asset.Three decisions before producing
Three decisions commit a data labeling project.Writing a reference that settles rather than a document that describes.Providing an arrangement able to measure omission.And obtaining from the client the format their processing chain expects.Those three decisions cost a few days, they precede the first item handled, and their absence produces a corpus whose value will stay unknown.Three checks on an existing corpus
Three controls qualify an already labeled data set.The presence of the reference applied during production.The existence of an omission measurement distinct from the other indicators.And an indication of the agreement rate between two operators.Those three controls are each asked in one question, they require no technical competence, and their absence indicates a corpus whose consistency rests on habits rather than on a rule.Three questions before costing
Three questions precede any data labeling quotation.How many elements does a single unit of data hold on average.On which sample exactly was that count carried out.And what fraction of the corpus will the control cover.Those three questions are asked in a few minutes, they call for examining a real sample, and their absence leads to committing to a scope neither the client nor the provider has verified.What this chapter teaches
One cross-cutting observation deserves closing this opening.The visible part of the work is the least determining.Three findings compose it.Only production is visible from outside a service.A silent reference lets every operator invent their own rule.And omission reveals itself only by going back over the same data.That finding opens the fourteen chapters that follow, each returning to an aspect this activity’s apparent simplicity causes to be neglected.What this guide establishes
Four principles run through this whole course.What is decided before production determines what it will reach.A defect invisible in the deliverable costs more than a visible one.A global measurement always masks a failure on one population.And a corpus without documentation loses its value with the team’s memory.One important practical consequence follows. Those four principles bear on scoping and measurement rather than on execution, which makes them insensitive to technical developments and explains why they apply identically to every modality handled in the following chapters.What this chapter leaves to the next
Each type of data poses these questions in its own way.Three questions stay open in data labeling.What an image demands that text does not.Why audio and video move the reference towards time.And what stays common to every modality.Those three questions belong to the panorama by modality, which constitutes the subject of the following chapter.Why the simplicity of this work is misleading
One point about how this activity is perceived belongs at the close.Clients rarely believe this work needs discussing before it starts.Three reasons follow in data labeling.The task looks obvious when described in one sentence.The unit price is low enough to seem not worth the meeting.And nothing in a delivered corpus displays what went wrong.One important practical consequence follows for a data labeling provider. Insisting on the scoping conversation anyway is what separates the two outcomes, since a client who skipped it will judge the corpus by its appearance and discover the problem only when their model underperforms, and by then the cause is indistinguishable from a dozen others.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Comparing two services on unit price alone.
- Starting production with no written reference.
- Measuring quality without measuring omission.
- Discovering the expected format after production.
- Neglecting the conduct to adopt before an unforeseen case.
- Assuming two operators will describe the same thing.
- Confusing volume produced with the corpus’s value.
- Delivering a corpus without its documentation.
- Estimating a volume without having counted on a sample.
- Treating scoping as a preliminary formality.
