Describing what a piece of data holds and judging what a model has produced are two distinct kinds of work. The second admits no single answer, which moves the reference from a description towards a judgement and alters everything the preceding chapters established.
This article sets out that shift. It extends the article on labeling instructions.
What these uses demand
Five kinds of human work feed a generative model in data labeling. Writing answers that then serve as reference examples. Ranking two given outputs by order of preference. Scoring an output against several distinct criteria. Flagging content that is problematic or else dangerous. And correcting an output so as to make it acceptable. One important practical consequence follows. The second kind dominates current uses, a comparison between two answers being made far faster and more reliably than an absolute score.What preference changes
Five differences separate a ranking from a description in data labeling. There exists no single right answer to be found. Two evaluators can diverge without either being wrong. The judgement depends on criteria that are sometimes contradictory. One evaluator’s own consistency becomes the central indicator. And the agreement measure loses part of its usual meaning here. One important observation follows for a data labeling project. The fourth difference replaces the arrangement described in the preceding chapters, an evaluator contradicting themselves on identical cases signalling a problem no disagreement between people would reveal.What the reference must fix here
Five points are settled before any evaluation of generated output. The precise criteria on which an output is judged. Their order of priority where they conflict with one another. The treatment given to outputs that are equally satisfactory. The conduct to adopt before manifestly problematic content. And the conduct to adopt when both answers are bad. One practical consequence follows. The second point is almost always missing, an accurate but badly phrased output having to be compared with an agreeable but approximate one with no rule settling the matter.What evaluation biases produce
Five inclinations systematically distort a ranking. The spontaneous preference given to the longer answer. The preference for a confident tone regardless of the content. The preference given to careful formatting. The tendency to choose systematically the first answer presented. And the growing indulgence over a session that runs a little long. One important practical consequence follows for a data labeling project. Those five inclinations all act in the same direction and do not offset one another, which produces a signal oriented towards form rather than accuracy and teaches the model exactly what one was trying to avoid.What the measurement becomes here
Five indicators describe the quality of such data labeling work. One evaluator’s consistency when faced twice with the same exact case. The agreement obtained between evaluators on the clearest cases. The proportion of comparisons finally declared equivalent. The average time spent on a single comparison. And the distribution of preferences by the criterion invoked. One observation follows. The first indicator becomes central, an evaluator inconsistent with themselves producing a signal nothing distinguishes from noise, which makes that measurement more useful than interpersonal agreement.What the arrangement must correct
Five measures limit the biases described above. A strictly random alternation of the order of the two outputs. The presentation of pairs of comparable length. The reinsertion of already evaluated cases to measure consistency. A strict limitation on the length of each session. And an obligation to justify each choice briefly. One important observation follows for a data labeling project. The last measure acts most effectively, an evaluator having to name the criterion that decided their choice discovering for themselves that they preferred an answer for its length, which corrects a bias no outside control would have detected.What these uses require of evaluators
Five competences separate this work from ordinary description. The attentive reading of two sometimes long outputs. The capacity to apply several criteria simultaneously. Resistance to the preference for the longer answer. The detection of a false but well phrased assertion. And a sufficient knowledge of the field being handled. One important practical consequence follows for a data labeling project. The fourth competence separates a useful evaluator from a fast one, a confident and false answer being preferred by default to a hesitant and accurate one.What writing examples requires
Five constraints frame the production of reference answers. A verifiable accuracy across every fact stated. A tone and format matching the expected behaviour. A length strictly proportionate to the question asked. An explicit refusal where the request must not be met. And enough originality not to copy an existing source. One important practical consequence follows for a data labeling project. The first constraint makes this work far slower than it looks, a writer having to verify every assertion they produce, which brings the throughput closer to documented writing than to plain description.What flagging content imposes
Five precautions frame this kind of task. A written and precise definition of what must be flagged. An exposure limited in duration for each evaluator. An informed consent before any assignment to this task. A possibility of withdrawing from the task with no consequence. And support available for the people concerned. One important observation follows for a data labeling project. Those five precautions belong to the employer’s responsibility as much as to the quality of the work, repeated exposure to difficult content having effects neither the reference nor the control measures, which makes this point non-negotiable regardless of the budget.What correcting an output demands
Five requirements frame this kind of task. A modification as minimal as possible rather than a full rewrite. A reason for correction recorded for each intervention. Respect for the tone as well as the original format. A systematic check of the facts added during the correction. And a distinction between factual error and stylistic preference. One important practical consequence follows for a data labeling project. The first requirement determines the usefulness of the signal produced, a full rewrite no longer saying anything about what was wrong in the original output, which turns an exploitable correction into a plain new example.Three errors proper to generative uses
Three data labeling defects belong to this kind of work. A measurement arrangement carried over unchanged from ordinary description. An implicit ban on declaring two outputs equivalent. And a specialised evaluation entrusted to generalist evaluators. Those three defects produce a preference corpus of complete appearance, they reveal themselves in the behaviour of the resulting model, and their common point is to carry into a judgement task habits designed for a description task.What the client must settle here
Five decisions belong to the client rather than to the provider. The precise criteria on which an output will be judged. Their order of priority where they conflict with one another. The behaviour expected before a problematic request. The level of expertise genuinely required of evaluators. And the degree of tolerance granted to a legitimate divergence. One important observation follows for a data labeling project. The second decision is almost always neglected and shapes the model more than all the others, an arbitration between accuracy and agreeableness left implicit being settled differently by each evaluator, which produces an average signal corresponding to no intention at all.What this chapter shares with the previous ones
Three findings recur from the start of this course. An unwritten rule is replaced by as many rules as there are people. An indicator never measures anything but what it can see. And the most expensive defect stays the one nothing signals. One important practical consequence follows for a data labeling project. The second finding takes a new form here, a high agreement rate possibly reflecting shared biases rather than grounded judgement, which makes that indicator misleading exactly where it seemed most natural to use.What this subject has that is particular
Four traits distinguish this question from the course’s others. It bears on a judgement rather than on an observation. It deprives the usual arrangement of its principal indicator. It sometimes exposes evaluators to difficult content. And it shapes a behaviour rather than a detection capacity. One important practical consequence follows for a data labeling project. The last trait raises the stakes of this work, a preference corpus determining how a system will answer millions of people, which justifies a higher demand for method than ordinary description requires.What the first batch must establish
Five measurements direct the rest of the project. The consistency rate of an evaluator measured on repeated cases. The proportion of pairs judged equivalent. The real time spent on a single comparison. The list of conflicts between criteria effectively met. And the correlation between preference expressed and answer length. One important observation follows for a data labeling project. The last measurement is calculated in minutes and reveals what matters, a preference correlated with length signalling that the arrangement captures form instead of substance, which invalidates the corpus well before the model reveals the effect.What legitimate divergence requires providing for
Five arrangements handle a disagreement that is not an error. A written distinction between acceptable divergence and error. A disagreement threshold beyond which a case goes up for adjudication. A retention of both judgements rather than a forced arbitration. A periodic analysis bearing on the most disputed cases. And a revision of the criteria as soon as the disagreement recurs. One important practical consequence follows for a data labeling project. The third arrangement surprises and is worth more than the alternative, a retained disagreement signalling an area where the expected behaviour stays undetermined, which informs more than a preference settled arbitrarily to produce a single figure.What these uses do not change
Five findings from this course stay intact here. The reference must be written before the very first evaluation. An unforeseen case goes up rather than being settled alone. A global measurement always masks a failure on one population. Documentation entirely conditions the corpus’s future value. And scoping weighs more than speed of execution. One important observation follows for a data labeling project. Those five findings hold the more clearly as the work becomes subjective, the absence of a right answer making the written convention not less necessary but entirely determining.What this work is becoming
Five developments are taking shape for these uses. A growing share of the evaluation entrusted directly to models. A human refocusing on the cases those models settle badly. An increasingly marked requirement for business expertise. Growing attention paid to the evaluators’ working conditions. And a documentation of criteria that has become a required item. One important practical consequence follows for a data labeling project. The first development reproduces exactly the mechanism described about pre-annotation, a model evaluating according to its own biases confirming what it already knows how to do, which makes human control more necessary as automation advances.What the provider brings here
Four contributions distinguish a serious generative service. A reference of ranked criteria rather than a plain list. An arrangement limiting length and position biases. A measure of individual consistency published with the corpus. And evaluators chosen for their real knowledge of the field. One important practical consequence follows. The second contribution is not visible in the deliverable and changes everything it holds, a corpus produced without order alternation carrying a systematic preference for the first answer, which teaches the model a rule nobody meant to pass on.What this chapter presupposes of the provider
Five practices distinguish serious generative work. Flagging to the client the criteria conflicts their reference leaves open. Setting up order alternation without waiting to be asked. Publishing individual consistency rather than agreement alone. Limiting exposure to difficult content even without an obligation. And refusing a specialised evaluation for want of competent evaluators. One important practical consequence follows. The first practice sends back to the client work they believed finished, a reference listing criteria without ranking them leaving the most important arbitration to evaluators who have no standing to make it.What these uses change about the economics
Five effects distinguish the budget of a generative project. Throughput falls markedly, a comparison requiring two full readings. The level of expertise required appreciably raises the hourly cost. Reinserting repeated cases adds apparently unproductive volume. The limited length of sessions reduces daily production. And scoping the criteria occupies a quite unusual place. One important observation follows for a data labeling project. The third effect is easily removed and paid for afterwards, reinserting cases costing a few percent of the volume while supplying the only indicator able to qualify the corpus produced.Approaching a generative project
Five questions scope such a data labeling project. Are the criteria written. Otherwise each person invents their own. Is their priority fixed. Conflicts are frequent. Is consistency measured. It replaces agreement. Does the field require expertise. Often yes. And is equivalence permitted. Otherwise it will be forced. Those five answers determine the value of the signal produced. Asking them before production avoids a preference corpus nothing explains.The question that frames the project
One question determines the nature of the work in generative data labeling. Is one seeking to establish what is accurate or what is preferable. What is accurate, a verifiable factual answer, is handled as an ordinary description and bears the usual agreement indicators. What is preferable, a tone, a length or a balance between criteria, belongs to judgement and requires an entirely different arrangement. That question belongs to what the model must learn, it is asked before any task design, and it separates two projects nothing permits conducting in the same way.Three decisions before launching an evaluation
Three decisions commit a generative data labeling project. Writing the criteria and their order of priority before the first pair. Explicitly permitting a declaration of equivalence. And providing for the reinsertion of already seen cases to measure consistency. Those three decisions cost half a day, they precede any production, and their absence produces a signal nobody will be able to interpret afterwards.Three checks on a preference corpus
Three controls qualify such a data labeling corpus. The presence of a measure of individual consistency. The existence of written criteria with their order of priority. And the proportion of comparisons declared equivalent. Those three controls are each asked in one question, they require no technical competence, and their absence indicates a preference corpus whose logic nobody knows.Three questions before costing
Three questions precede a generative data labeling quotation. Which criteria and which order of priority will apply. What level of expertise does the field require of evaluators. And does the task expose people to difficult content. Those three questions are asked before any costing, they require no technical competence, and their absence leads to committing to a throughput that corresponds to nothing.What this chapter teaches
One cross-cutting observation deserves closing this examination. The reference moves from a description to a judgement. Three findings compose it. Two evaluators can diverge without either being wrong. Individual consistency replaces interpersonal agreement. And a confident, false answer is preferred by default. That finding extends the preceding chapters, the measurement arrangement having to be entirely rethought.What this chapter leaves to the next
All this discussion presupposes knowing which data to evaluate. Three questions stay open in data labeling. How to choose the data that will teach the model most. Why a random draw wastes part of the budget. And what a selection guided by the model loses. Those three questions belong to active learning, which constitutes the subject of the following chapter.Why the criteria conversation gets pushed back
One point about how these projects are scoped belongs at the close. Clients supply a list of criteria and rarely rank them. Three reasons follow in data labeling. Ranking them means admitting that some matter less than others. Every criterion has an internal owner who wants it first. And the conflict only becomes concrete once real pairs appear. One important practical consequence follows for a data labeling provider. Sending the ranking question back rather than absorbing it is the only defensible position, since an unranked list means each evaluator resolves the organisation’s internal disagreement on its behalf, thousands of times, in a direction nobody chose.What the evaluators carry here
Four demands weigh on people doing this work. Sustained attention across two long readings per decision. Repeated judgement with no correct answer to rest on. Occasional exposure to content that is hard to read. And an output whose effect they will never observe. One important observation follows for a data labeling project. The second demand is the least visible and the most tiring, a task offering no resolution wearing differently from one where an answer eventually confirms the work, which makes session limits and rotation more necessary here than anywhere else in this course.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Applying the agreement indicators designed for description.
- Omitting the priority order between contradictory criteria.
- Forbidding two outputs to be declared equivalent.
- Entrusting a specialised evaluation to generalist evaluators.
- Neglecting the spontaneous preference for longer answers.
- Measuring agreement without measuring individual consistency.
- Treating a legitimate divergence as an error.
- Omitting the conduct to adopt when both answers are bad.
- Costing this work at the throughput of a simple description.
- Assuming an evaluator detects a well phrased factual error.
