A labeling budget is spent on data that was chosen. Random sampling, which remains the common practice, devotes most of that budget to describing situations the model already handles correctly.
This article sets out that selection. It extends the article on labeling for generative AI.
What random sampling produces
Five effects accompany a selection with no criterion in data labeling. The corpus composition reproduces exactly that of the available data. The most common situations occupy the majority of the volume. Rare cases appear only in proportion to their frequency. The budget is largely spent on cases already mastered. And the model’s progress slows as the volume grows. One important practical consequence follows. The third effect severely limits the result, a class present in one case out of a thousand obtaining only a handful of examples in an otherwise considerable corpus.What guided selection changes
Five criteria direct the choice of data to describe in data labeling. The model’s uncertainty bearing on a precise item of data. The disagreement between several models bearing on the same item. The proximity to cases previously handled badly. The distance from the data already described. And membership of a visibly under-represented category. One important observation follows for a data labeling project. The first criterion dominates current uses, a high uncertainty designating precisely the data whose description will teach the model most.What this approach gains
Five benefits accompany a reasoned selection of the data. The same performance level reached with far less data. A markedly faster progression in the first iterations. A markedly increased coverage of the field’s difficult cases. An early detection of badly defined categories. And a budget entirely concentrated on what is genuinely missing. One practical consequence follows. The fourth benefit gets little notice and is worth a great deal, the data the model finds uncertain often revealing a category boundary the reference had settled badly.What this approach loses
Five counterparts accompany a guided selection. The corpus stops representing the real distribution of the data. Difficult cases accumulate at the expense of general variety. The selection inherits every blind spot of the current model. The evaluation set must stay randomly drawn. And implementation demands a genuine iterative infrastructure. One observation follows. The third counterpart is the gravest, a model unaware of a category never finding it uncertain and therefore never asking to be shown it.What uncertainty really signals
Five situations produce a high uncertainty in the model. A case genuinely difficult for any human observer. A category the reference defined particularly badly. A degraded item whose poor quality prevents any decision. A situation absent from the initial training corpus. And a plain calibration defect affecting the model used. One important practical consequence follows for a data labeling project. Those five situations call for opposite responses, the first justifying more annotation while the third justifies discarding the item, which makes examining the selected cases more instructive than the criterion itself.What the iterative cycle presupposes
Five stages compose a complete data labeling loop. A first batch drawn at random serving as an initial base. A training run producing a first intermediate model. A selection of the data by the criterion finally adopted. A description of that batch entrusted to the operators. And an evaluation on an independent set before the next iteration. One important practical consequence follows for a data labeling project. The last stage is easily neglected and invalidates all the rest, a cycle conducted with no independent measurement no longer permitting anyone to know whether the selection improves or degrades the result.What the initial batch determines
Five properties make a first batch usable. A random draw preserving the real distribution. Coverage of every category in the reference. A volume sufficient to train a first model. A deliberate inclusion of the most difficult situations. And a quality controlled before any training run. One important practical consequence follows for a data labeling project. The second property conditions everything that follows, a category absent from the initial batch staying invisible to the model that will guide the selection, which condemns it never to appear in the final corpus.What this approach changes about the organisation
Five constraints weigh on how the project runs. Production is organised in batches rather than continuously. Waiting periods separate two successive batches. The team must stay available between two iterations. The total volume is not known at the start. And coordination with the technical team becomes daily. One important observation follows for a data labeling project. The third constraint is neglected at costing and is expensive, a trained team having to be kept on during the training periods without producing, which cancels part of the gain the volume reduction had promised.What blind spots require providing for
Five arrangements compensate for what the model does not ask for. A share of random draw kept in every batch. A periodic examination of what the selection never retains. A deliberate inclusion of the rarest categories. An evaluation set covering situations absent from the corpus. And a usage feedback reporting the failures observed in production. One important observation follows for a data labeling project. The last arrangement is the only one entirely escaping the model’s limits, a failure observed in real use signalling a situation neither the selection nor the evaluation had envisaged, which makes it the most valuable source for the next iteration.Three errors proper to guided selection
Three data labeling defects belong to this approach. An evaluation set built by the same criterion as the training. A selection launched from an initial model that is too weak. And a complete absence of random draw in the successive batches. Those three defects produce an apparent and flattering progression, they reveal themselves when the system meets real situations, and their common point is to let the model decide alone what it must learn.What the client must supply here
Five elements come from the client rather than from the provider. Access to the model producing the uncertainty scores. An evaluation set built and kept apart. An iteration rhythm compatible with their training cycles. The categories they know to be under-represented. And feedback on the failures observed in real use. One important practical consequence follows for a data labeling project. The third element conditions the whole organisation, a training run taking several days imposing interruptions the annotation team must absorb, which makes coordinating the calendars more determining than the choice of selection criterion.What this chapter shares with the previous ones
Three findings recur from the start of this course. Automatic help reinforces what the model already knows how to do. A corpus that does not test the difficulty reassures wrongly. And the most expensive defect stays the one nothing signals. One important observation follows for a data labeling project. The first finding takes its most paradoxical form here, a selection guided by the model explicitly targeting its weaknesses while staying blind to those it is unaware of, which reproduces the pre-annotation mechanism inside an arrangement designed to avoid it.What this subject has that is particular
Four traits distinguish this question from the course’s others. It decides what will be described rather than how to describe it. It entrusts a scoping decision to an intermediate model. It produces a real gain and a non-representative corpus. And it imposes an organisation in cycles rather than continuous. One important practical consequence follows for a data labeling project. The third trait deserves owning from the outset, a corpus optimised for training no longer serving to measure, which requires building and funding two distinct sets rather than one.What the first loop must establish
Five measurements direct the rest of the project. The gain obtained compared with a random batch of the same size. The proportion of selected data that proves unusable. The distribution of categories within the selected batch. The real time separating two successive iterations. And the stability of the criterion from one iteration to the next. One important observation follows for a data labeling project. The first measurement settles matters on its own, a guided selection bringing nothing over a random batch of the same volume signalling that the initial model was too weak or the corpus too homogeneous, which avoids committing to several cycles with no benefit.What this approach changes about the economics
Five effects modify the structure of the budget. The total volume of data described falls markedly. The unit cost rises since the remaining cases are difficult. Keeping the team between batches adds to the expenditure. Setting up the loop occupies part of the start-up. And the evaluation set constitutes an additional expense. One important practical consequence follows for a data labeling project. The second effect sharply reduces the apparent gain, halving the volume not halving the budget since each remaining item demands more time, which makes a gain announced in volume misleading once again.What this chapter presupposes of the provider
Five practices distinguish an honest guided selection. Comparing the first selected batch with an equivalent random one. Flagging that part of the retained data is unusable. Keeping a random draw even when nobody asks for it. Warning that the resulting corpus no longer serves to evaluate. And saying when this approach will bring nothing on this project. One important practical consequence follows. The last practice runs against immediate commercial interest, a guided selection selling better than an ordinary draw, whereas it serves no purpose on a homogeneous corpus and costs the cycle-based organisation it imposes.What this approach does not replace
Five pieces of work stay intact whatever selection is adopted. Drafting the reference and adjudicating the limiting cases. Measuring omission on a sample gone over again. Building a representative evaluation set. The quality control applied to each batch produced. And the documentation handed over with the final corpus. One important observation follows for a data labeling project. Those five pieces of work already represented the project’s most expensive share, which explains why a reduction in the volume described does not produce the budget fall the announced gain had promised.What the selection criterion must follow
Five observations direct the choice of criterion used. Uncertainty suits cases where the categories are well defined. Disagreement between models suits cases where several are available. Distance from described data suits a new field. Targeting a category suits cases where it has been identified. And a mixed criterion suits most real projects. One important practical consequence follows for a data labeling project. The third observation handles precisely the blind spot described above, a selection favouring what least resembles the already described data going after situations uncertainty did not signal, which partly compensates for the model’s blindness.What stopping the loop requires deciding
Five signals indicate that a further iteration will bring nothing. A performance gain that has fallen below the measurement noise. A selection retaining data close to the previous batches. A growing proportion of unusable data being retained. A budget reaching the limit set at the outset. And a performance level judged sufficient for the use. One important observation follows for a data labeling project. The last signal is neglected in favour of the others, a technical team pursuing improvement well after the real use has stopped benefiting from it, which consumes a budget another piece of work would have used better.What batch size decides
Five effects accompany the sizing of an iteration. A small batch permits a fast correction of the criterion. A small batch multiplies the team’s waiting periods. A large batch better amortises the training time. A large batch risks carrying an already outdated criterion. And the optimal size depends on the model’s training time. One important practical consequence follows for a data labeling project. The last effect sizes the batch better than any general rule, a batch having to occupy the team while the model trains, which makes that technical constraint determining for the daily organisation of the work.What the provider brings here
Four contributions distinguish a well conducted guided selection. A human examination of the selected data before it is handled. A share of random draw kept in every iteration. A flagging of the categories the model never asks for. And a measure of the gain published iteration after iteration. One important practical consequence follows. The first contribution is almost always neglected and is worth a great deal, a selected batch sometimes revealing that part of the uncertain data is simply illegible, which calls for sorting rather than description and avoids paying for pointless work.Approaching a guided selection
Five questions scope such a selection in data labeling. Does the evaluation set stay random. Otherwise nothing is measurable. Does the criterion suit the need. Uncertainty is not always enough. Are the blind spots handled. The model does not signal them. Is the loop genuinely iterative. A single pass brings nothing. And is the initial batch sufficient. A weak model selects badly. Those five answers determine the approach’s usefulness. Asking them before launching avoids an unbalanced corpus with no measurable gain.The question that frames the selection
One question determines this approach’s interest in data labeling. Must the corpus represent the world or correct a model. Representing the world, serving as a reference or an evaluation, requires a random draw no guided selection can stand in for. Correcting a model, filling in what it handles badly, justifies a guided selection and accepts the imbalance it produces. That question belongs to the corpus’s destination, it is asked before the first iteration, and it explains why one same project needs both at once.Three decisions before guiding the selection
Three decisions commit a guided selection in data labeling. Building an evaluation set drawn at random and keeping it apart. Covering every category in the initial batch. And providing for a share of random draw in each iteration. Those three decisions cost a few days, they precede the first loop, and their absence produces an unbalanced corpus whose contribution nobody will be able to measure.Three checks on a loop under way
Three controls qualify a guided selection in data labeling. The independence of the evaluation set from the selection criterion. The presence of a random share in each selected batch. And the measurement of the gain at each iteration rather than at the end. Those three controls are each asked in one question, they require no technical competence, and their absence indicates a loop nobody can affirm is improving anything at all.Three questions before launching a loop
Three questions precede adopting a guided selection. Will the evaluation set be built independently of the criterion. What delay separates two iterations in practice. And what share of random draw will be maintained. Those three questions are asked before the first batch, they require no technical competence, and their absence leads to committing to an arrangement whose real contribution nobody will measure.What this chapter teaches
One cross-cutting observation deserves closing this examination. The model can only ask for what it already knows it handles badly. Three findings compose it. A random draw spends most of the budget on mastered cases. A high uncertainty designates the most instructive data. And a category that is ignored will never be judged uncertain. That finding extends the preceding chapters, the selection reproducing the mechanism already described for pre-annotation.What this chapter leaves to the next
Selecting presupposes existing data one can also manufacture. Three questions stay open in data labeling. What artificially produced data genuinely replaces. Why it does not dispense with a human description. And in which cases it brings more than a real acquisition. Those three questions belong to synthetic data, which constitutes the subject of the following chapter.Why the volume gain gets quoted and the cost gain does not
One point about how this approach is presented belongs at the close. Guided selection is always announced as a percentage of volume saved. Three reasons follow in data labeling. The volume reduction is real and easy to demonstrate. The rise in unit cost only appears once production starts. And the waiting periods between batches are nobody’s line item. One important practical consequence follows for a data labeling provider. Quoting the expected budget rather than the volume is the only honest framing, since a client told their corpus will shrink by half expects their invoice to do the same, and discovers otherwise at a point where the arrangement is already in place.What the two teams owe each other here
Four commitments make an iterative loop workable. The technical team announces its real training turnaround. The labeling team keeps capacity available across the gaps. Both agree who examines the selected batch before work starts. And neither treats a disappointing iteration as the other’s failure. One important observation follows for a data labeling project. The last commitment decides whether the loop survives its first poor result, an iteration bringing no gain being ordinary information about the corpus rather than evidence of bad work, which only holds if both sides agreed on that reading beforehand.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Selecting the evaluation set by the same criterion as the training.
- Conducting a single pass rather than an iterative loop.
- Launching the selection from an initial model that is too weak.
- Neglecting the categories the model ignores entirely.
- Confusing the model’s uncertainty with the case’s real difficulty.
- Forgetting that the resulting corpus no longer represents the distribution.
- Measuring the gain with no independent evaluation set.
- Applying this approach to a corpus too small to benefit.
- Selecting only difficult cases until saturation.
- Assuming a gain observed on one field transfers to another.
