Manufacturing data rather than acquiring it solves an availability problem by introducing another. A produced item arrives with its description already made, which removes the human work and replaces it with a question nothing settles: does this data resemble what the system will encounter.
This article sets out that practice. It extends the article on active learning.
What produced data brings
Five benefits accompany this approach in data labeling. An exact description supplied with no human intervention at all. A volume available with no acquisition limit whatever. A coverage of situations that are rare or else dangerous. A balance of categories decided rather than endured. And a complete absence of any personal data. One important practical consequence follows. The third benefit alone justifies this approach in several fields, certain situations being neither waited for nor provoked in the real world.What produced data does not replace
Five limits accompany this approach in data labeling. The gap between produced data and genuinely acquired data. The absence of all the real world’s acquisition defects. The repetition of patterns the generator reproduces endlessly. The difficulty of producing what nobody has imagined. And the impossibility of evaluating a system on this data alone. One important observation follows for a data labeling project. The last limit is the most constraining, an evaluation set having to come from the real world whatever the synthetic share of the training corpus.What the gap from reality produces
Five differences separate a produced item from a genuinely acquired one. A sharpness markedly superior to that of real sensors. A lighting as well as a framing that are far too regular. An absence of the artefacts proper to each piece of equipment used. A variability markedly below that of the observed world. And a distribution of situations decided by a parameter setting. One practical consequence follows. Those five differences combine into a gap the system learns without knowing it, which produces excellent results on produced data and a failure on the first real data.What measuring the gap demands
Five comparisons reveal the distance between the two sets. The performance of one same model measured on each of the two corpora. The capacity of a simple classifier to distinguish their origin. The distribution of categories in one as well as in the other. The acquisition characteristics present on one side only. And the types of errors effectively committed on each set. One important practical consequence follows for a data labeling project. The second comparison gives the most direct measure, a simple model distinguishing produced data from real data effortlessly signalling a gap the main system will learn just as easily.What combining the two permits
Five ways of mixing generation and acquisition in data labeling. An initial training on produced data then a tuning on real data. An enrichment of the rare categories alone through generation. A production targeted on the sole dangerous situations to observe. A proportionate mixture within one same training corpus. And an evaluation conducted exclusively on real data. One observation follows. The first way gives the best results in most fields, a produced base supplying the general regularities while a real tuning corrects the gap.What human work becomes here
Five tasks persist despite the automatic description. Defining the reference the generator will have to apply. Verifying that the produced data really does resemble reality. Describing the evaluation set, necessarily real. Analysing the situations the generator never covers. And describing the real data serving for the tuning. One important practical consequence follows for a data labeling project. Those five tasks already represented a project’s most expensive share, which explains why an approach removing the description entirely reduces the budget only to a limited extent.What the generator requires beforehand
Five pieces of work precede any production of data. Modelling the objects as well as the scenes to represent. Setting the parameters of the variations the generation will have to explore. Defining the situations the final corpus must cover. Reproducing the acquisition defects proper to the real world. And validating the rendering by comparison with acquired data. One important observation follows for a data labeling project. Those five pieces of work shift the cost rather than removing it, a credible generator demanding weeks of design, which makes this approach profitable on large volumes and expensive below that.What the fields concerned have in common
Five situations make this approach particularly relevant. A rare event acquisition will never encounter. A dangerous situation impossible to provoke for real. An environment whose access is expensive or else restricted. A need bearing on particularly sensitive personal data. And a geometry perfectly known from the construction itself. One important practical consequence follows for a data labeling project. The last situation explains certain fields’ lead, a produced scene supplying the exact distances and positions no human description would obtain, which makes generation superior to manual work rather than simply cheaper.What the final tuning demands
Five properties make this last stage effective. A modest real volume but very carefully described. A coverage of the most frequent situations met in use. A description quality above that of the produced corpus. An inclusion of the acquisition defects proper to the equipment used. And a measure of the gain obtained before and then after this stage. One important observation follows for a data labeling project. The third property is wrongly neglected, a tuning bearing on little data tolerating no approximation since each element there weighs infinitely more than in a corpus produced in volume.Three errors proper to generation
Three data labeling defects belong to this approach. An evaluation conducted on the produced data itself. A gap from reality never measured and assumed negligible. And a generator reproducing the situations already well covered. Those three defects produce results announced as excellent, they reveal themselves on first contact with acquired data, and their common point is to confuse mastery of a constructed world with mastery of the observed one.What the client must supply here
Five elements come from the client rather than from the provider. A sample of real data serving as a reference for the rendering. A description of the situations their use really meets day to day. The precise characteristics of the acquisition equipment used. The rare situations they absolutely want covered. And feedback on the failures observed during real use. One important practical consequence follows for a data labeling project. The first element conditions the generator’s whole rendering, a production designed with no real reference producing a plausible but different world, which is corrected with difficulty once the volume has been produced.What this chapter shares with the previous ones
Three findings recur from the start of this course. A corpus that does not test the difficulty reassures wrongly. The most expensive defect stays the one nothing signals. And a measurement never says anything beyond what its population holds. One important observation follows for a data labeling project. The third finding takes its clearest form here, an excellent result calculated on produced data faithfully describing performance on a world that does not exist, which recalls that a figure is read first through the set on which it was obtained.What this subject has that is particular
Four traits distinguish this question from the course’s others. It removes the description work rather than improving it. It replaces a cost problem with a validity problem. It produces an exact description by construction. And it makes the evaluation set more important than the corpus. One important practical consequence follows for a data labeling project. The third trait explains this approach’s appeal, a description with no possible error removing at a stroke all the agreement and omission concerns of the preceding chapters, which shifts the difficulty entirely without reducing it.What the first trial must establish
Five measurements direct the rest of the project. The performance obtained on the real evaluation set. The ease with which a model distinguishes the two origins. The gain brought by a tuning on acquired data. The list of situations the generator cannot produce. And the real time devoted to designing the generator. One important observation follows for a data labeling project. The third measurement settles what matters, a tuning bringing a considerable gain signalling a large gap the produced volume did not compensate, which indicates where to direct the effort before increasing production.What this approach changes about the economics
Five effects modify the structure of the budget. The description cost tends towards zero on the produced share. Designing the generator occupies a large part of the start-up. Producing volume then costs almost nothing at all. The evaluation set stays entirely at the project’s expense. And the final tuning adds a carefully described real corpus. One important practical consequence follows for a data labeling project. The second effect fixes the break-even point, designing a generator costing several weeks whatever the volume targeted, which makes this approach interesting beyond a certain volume and a loss below it.What this chapter presupposes of the provider
Five practices distinguish an honest mixed project. Publishing the results obtained on real rather than produced data. Measuring the gap and reporting it even where it hurts the project. Flagging the situations the generator will never cover. Costing the generator design before announcing a saving. And saying when the volume targeted does not justify this approach. One important practical consequence follows. The first practice immediately separates two kinds of claim, a result announced without stating its calculation set being precisely the one no client should accept, which rejoins the breakdown requirement set out in the chapter on quality.What this approach does not replace
Five pieces of work stay intact despite the automatic description. Defining the categories and their boundaries. Building a real and described evaluation set. Analysing the situations the use really meets. Describing the data serving for the final tuning. And the documentation handed over with the produced corpus. One important observation follows for a data labeling project. The first piece of work stays entirely human and conditions the generator itself, a badly defined category producing a million examples faithfully applying a mistaken convention, which worsens the defect instead of easing it.What production from a model changes
Five observations concern data produced by another system. The description is no longer exact but merely probable. The generating system’s biases pass into the produced corpus. The variety narrows as the generation repeats itself. A human verification becomes necessary again on a sample. And the data’s origin deserves tracing right through to delivery. One important practical consequence follows for a data labeling project. The second observation clearly distinguishes this data from a parametric generation, a description issued from a model faithfully reproducing its errors, which brings this practice closer to the pre-annotation examined earlier than to a genuine synthetic production.What this practice is becoming
Five developments are taking shape for these uses. The rendering quality improves and narrows the gap from reality. Production from models is generalising rapidly. Requirements for tracing the origin are tightening. Fields with known geometry stay the best served. And the value of described real data rises accordingly. One important observation follows for a data labeling project. The last development deserves noting, a carefully described real corpus becoming the rare reference against abundant produced volumes, which reverses the value relation between the two sets.What the proportion of the two sets decides
Five observations direct the balance between production and acquisition. A majority produced share suits fields with a known geometry. A majority real share suits highly variable fields. Rare categories justify a targeted production whatever the balance. The evaluation set stays real in every case. And the final tuning always bears on acquired data. One important practical consequence follows for a data labeling project. The third observation permits a balance varying by category rather than a single ratio, a frequent class doing without production while a rare one benefits strongly, which makes the balance finer than a plain global proportion.What the provider brings here
Four contributions distinguish a well conducted mixed project. The careful description of the real evaluation set. A measure of the gap published rather than assumed negligible. A flagging of the situations the generator does not cover. And a description of the real data serving for the final tuning. One important practical consequence follows. The first contribution looks modest and becomes the most determining, a small but perfectly described evaluation set constituting the only reference permitting anyone to know whether the produced corpus serves any purpose.Approaching a mixed project
Five questions scope such a data labeling project. Does the evaluation stay real. That is non-negotiable. Is the gap from reality measured. Otherwise it shows late. Which situations justify generation. The rare and the dangerous. Is a real tuning provided for. It corrects the gap. And does the generator cover the unforeseen. Almost never. Those five answers determine the value of the corpus obtained. Asking them before producing avoids a system excellent in simulation and failing elsewhere.The question that frames the use of generation
One question determines this approach’s relevance in data labeling. Does the problem come from the volume available or the variety met. A volume problem, abundant data that is expensive to describe, is handled by generation as long as the gap from reality stays measurable and correctable. A variety problem, situations acquisition never produces, is handled by generation provided somebody knows how to describe them well enough to model them. That question belongs to what the corpus lacks, it is asked before any generator design, and it explains why one same technique serves two objectives with no common measure.Three decisions before generating
Three decisions commit a mixed data labeling project. Building an evaluation set drawn from the real world before any generation. Measuring the gap between the two sets from the first batch produced. And providing for a final tuning on genuinely acquired data. Those three decisions cost a few days, they precede volume production, and their absence produces a system excellent in simulation and unverifiable elsewhere.Three checks on a mixed corpus
Three controls qualify such a data labeling corpus. The real origin of the evaluation set finally used. The presence of a measure of the gap between the two sets. And the existence of a tuning conducted on acquired data. Those three controls are each asked in one question, they require no technical competence, and their absence indicates an announced result nothing attaches to the real world.Three questions before generating
Three questions precede adopting this approach. Where will the data serving for the final evaluation come from. How will the gap from reality be measured, and when. And is a tuning on acquired data provided for in the budget. Those three questions are asked before any design, they require no technical competence, and their absence leads to funding a volume whose contribution nobody will be able to estimate.What this chapter teaches
One cross-cutting observation deserves closing this examination. Produced data carries the exact description of an approximate world. Three findings compose it. The gap from reality is learnt with nothing signalling it. An evaluation set must stay real whatever the method. And a generator does not produce what nobody has imagined. That finding extends the preceding chapters, the difficulty moving from description towards representativeness.What this chapter leaves to the next
A corpus produced or acquired carries the same defects as any corpus. Three questions stay open in data labeling. Which defects cost most once the system is in use. Why some of them stay invisible until the last moment. And how to spot them before they show themselves. Those three questions belong to the most expensive errors, which constitute the subject of the following chapter.Why the simulation figure travels further than it should
One point about how these results get reported belongs at the close. A performance figure obtained on produced data circulates as if it were general. Three reasons follow in data labeling. It is available early, before any real evaluation exists. It is almost always the higher of the two numbers. And nothing in the figure itself records where it came from. One important practical consequence follows for a data labeling provider. Naming the evaluation set every time the figure is quoted is the only reliable safeguard, since a number repeated three times without its origin becomes the project’s official performance, and correcting it later reads as a regression rather than a clarification.What a real evaluation set is worth here
Four returns come from that small described corpus. The only measurement that speaks about the intended use. A reference for judging every future generator version. A basis for deciding when the produced volume stops helping. And a record of what the real world actually contains. One important observation follows for a data labeling project. The third return is the least anticipated, a real evaluation set showing when additional generated volume no longer moves the result, which turns an open-ended production into one with a defensible stopping point.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Evaluating a system on artificially produced data.
- Neglecting to measure the gap between production and reality.
- Assuming an unlimited volume compensates a distribution gap.
- Omitting the final tuning on real data.
- Generating only the situations already well covered.
- Confusing an absence of personal data with an absence of bias.
- Ignoring the artefacts the real world always adds.
- Expecting the generator to produce the unforeseen.
- Comparing a result in simulation with a result in use.
- Assuming the gain transfers from one field to another.
