Describing an object in an image, an intention in a sentence and a speaker in a recording pass for distinct trades. They in fact share the same difficulties, and what separates them belongs less to the data than to the way ambiguity shows itself there.
This article compares those modalities. It extends the data labeling guide.
What the still image demands
Five characteristics distinguish this modality in data labeling. The element is designated by a geometry drawn on the image. The object’s boundary is often visible, sometimes only assumed. The control is done by eye alone, and immediately at that. The number of elements per image varies very widely. And the ambiguity bears on what the box must hold. One important practical consequence follows. The third characteristic explains this modality’s popularity, a reviewer judging an image in a second whereas the other modalities require reading, listening or navigating.What video adds
Four requirements separate video from a plain series of images. An element keeps the same identity from one frame to the next. The description must stay stable on an object that stayed still. Appearance as well as disappearance are marked explicitly. And consistency is judged on the sequence taken whole. One important observation follows for a data labeling project. The first requirement carries most of the cost, identity being information no single frame permits verifying, which moves the control from the image to the trajectory.What text imposes
Five characteristics distinguish this modality in data labeling. The element is designated by a plain interval of characters. The segment’s exact boundary is constantly open to debate. The meaning depends on a context that is sometimes far away. Two readers will understand the same sentence differently. And the control requires rereading everything, hence time. One practical consequence follows. The fourth characteristic produces a structurally lower agreement rate than in imagery, which makes double description more necessary here rather than less.What audio shifts
Five characteristics distinguish this modality in data labeling. The reference becomes temporal rather than spatial. A sound segment’s boundary stays blurred by nature. The quality of the recording conditions the whole task. Listening takes at least the full length of the extract. And going back over it becomes a constant operation. One observation follows. The fourth characteristic fixes a floor nothing crosses, an hour of recording demanding at least an hour of listening and often several, which makes throughput far less compressible than in imagery.What these modalities share
Five difficulties recur identically everywhere. A convention must be written before any production. An omitted element stays invisible in the deliverable. Two operators end up diverging on the limiting cases. A global measurement masks a failure on one population. And a corpus without documentation becomes unusable. One important practical consequence follows for a data labeling project. Those five difficulties bear on scoping and on measurement rather than on the hand movement, which explains why a team rigorous on one modality stays so on another whereas a merely fast team transfers nothing.What three-dimensional data requires
Five difficulties accompany this modality in data labeling. No single view shows the object in its entirety. The density of information falls rapidly with distance. Dimensions become inferred rather than measured. The frame of reference used must be documented. And the operator’s competence builds slowly. One important practical consequence follows for a data labeling project. The last difficulty sets this modality apart from all the others, reading a spatial representation being acquired only through practice, which makes replacing an operator markedly more expensive.What the control costs by modality
Five gaps separate the verification arrangements. An image is reread in a small fraction of the production time. Video requires covering whole trajectories. Text demands a rereading as long as the description. Audio imposes listening again to the segments in full. And the third dimension calls for several successive viewpoints. One important observation follows for a data labeling project. Those five gaps explain the differences in rate better than the difficulty of the hand movement, a control costing nearly as much as production on some modalities whereas it represents a fraction of it on imagery.What multimodal work adds as difficulty
Four requirements appear when several modalities coexist. The temporal references must correspond exactly between sources. One same entity must carry the same identifier absolutely everywhere. The category references must stay mutually compatible. And a divergence between modalities must be settled by a rule. One important practical consequence follows for a data labeling project. The last requirement is discovered late and costs dearly, a recording contradicting what the image shows placing the operator before an adjudication nothing had foreseen, which produces as many decisions as there are people involved.What automation changes by modality
Five situations present themselves differently. Imagery benefits from reliable proposals on common objects. Video profits from interpolation between two key frames. Text receives proposals that are plausible and yet sometimes false. Audio gains a transcription to correct rather than to produce. And the third dimension remains today largely manual. One important observation follows for a data labeling project. The third situation demands the greatest vigilance, a plausible textual proposal being validated unexamined far more readily than a visibly misplaced box, which makes control more necessary precisely where automation appears most convincing.What training requires by modality
Five durations separate the learning curves. Imagery is mastered in days once the reference stays simple. Video demands learning to follow an object rather than to describe it. Text presupposes ease of reading and an understanding of the field. Audio calls for an ear already trained to recording conditions. And the third dimension is learnt in whole weeks rather than days. One important practical consequence follows for a data labeling project. Those five durations decide the capacity to absorb a rise in load, a modality slow to learn forbidding a team to be reinforced quickly, which makes the calendar depend on recruitment well before it depends on volume.Three errors proper to the choice of modality
Three data labeling defects belong to this choice. A rate transferred from one modality to another with no measurement. A control arrangement designed for imagery and applied to text. And a team trained on one modality then assigned to another. Those three defects are committed before production, they are observed when deadlines or quality depart from forecast, and their common point is to assume a continuity between kinds of work that present none.What the client must specify here
Four pieces of information come from the client rather than from the provider. The modality that genuinely carries the information their use needs. The conditions under which their data was acquired. The degree of ambiguity they already observe in their field. And the expected correspondence between sources where there are several. One important observation follows for a data labeling project. The second piece conditions the whole throughput, a noisy recording or a badly exposed image doubling the time spent without anything in the announced volume foreshadowing it, which makes examining a real sample indispensable before any commitment.What this chapter shares with the previous one
Three findings from the pillar chapter hold on every modality. The convention must be written before production begins. Omission stays the least visible and most expensive defect. And a corpus without documentation loses its value over time. One important practical consequence follows for a data labeling project. Those three findings hold the more clearly as the modality makes control difficult, a defect invisible on an image being far more so within an hour-long recording, which makes scoping more determining as verification becomes expensive.What the delivery format changes here
Five requirements vary with the modality handled. Imagery demands a fully documented coordinate system. Video requires a stable identifier as well as a frame number. Text calls for unambiguous character positions. Audio presupposes precise temporal references and a reference point. And the third dimension imposes describing the spatial frame. One important observation follows for a data labeling project. The third requirement produces silent errors, a one-character offset or a different encoding displacing every segment of a text without any file becoming invalid, which reproduces on text what coordinate conversion produces on imagery.What this subject has that is particular
Four traits distinguish this question from the course’s others. It is settled before a team has even been assembled. It decides competences that do not all transfer. It fixes time floors nothing will cross. And it passes for a plain detail of implementation. One important practical consequence follows for a data labeling project. The last trait explains the frequency of unpleasant surprises, a client choosing their modality according to the data they hold rather than to what handling it will take, which makes the real cost surface once acquisition has already been paid for.What the first batch must establish here
Four measurements justify a trial per modality. The real production time on a representative sample. The control time set against that production time. The agreement rate obtained between two operators. And the list of cases the reference did not settle. One important observation follows for a data labeling project. The second measurement often surprises and changes the quotation, a ratio between control and production varying tenfold across modalities, which makes that single measurement more informative than any throughput estimate.What the working tool changes by modality
Five functions determine the throughput obtained. Imagery demands fast drawing as well as a fluid zoom. Video requires frame-by-frame navigation forwards and backwards. Text calls for a selection precise down to the character. Audio presupposes fine control of playback speed. And the third dimension imposes fast switching between views. One important practical consequence follows for a data labeling project. The fourth function is the most neglected and the most profitable, slowed playback permitting a boundary to be placed that normal speed makes undecidable, which improves agreement between operators without changing the reference at all.What the corpus must cover by modality
Five situations deserve deliberate inclusion. Images taken under unfavourable lighting conditions. Sequences where several elements cross one another repeatedly. Texts written in a register unusual for the field. Recordings holding background noise or else varied accents. And three-dimensional scenes with uneven relief. One important observation follows for a data labeling project. Those five situations are almost always missing from the samples supplied, a client spontaneously showing their cleanest data, which produces a reassuring corpus and a system caught out at the first real conditions.What these modalities are becoming together
Four developments are bringing long-distinct kinds of work closer. The same tools now handle several types of data. References are written according to a common structure. Quality indicators are formulated in the same way. And teams move from one modality to another more easily. One important observation follows for a data labeling project. Those four developments remove neither the throughput gaps nor the time floors, which explains why a convergence of methods can coexist lastingly with rates that stay, for their part, very different from one modality to the next.What the provider brings here
Four contributions distinguish a serious multimodal service. A costing established modality by modality rather than in one block. A control arrangement suited to each of them separately. A written rule for contradictions between sources. And a team trained on the modality concerned rather than on annotation in general. One important practical consequence follows. The last contribution is badly verified and weighs heavily, a provider competent in imagery possibly proving inexperienced in audio or in three dimensions, which justifies asking for a trial batch on the modality genuinely concerned.What this chapter presupposes of the provider
Four practices distinguish an honest position on modalities. Announcing that a modality has never been handled rather than stretching the truth. Costing the control separately instead of drowning it in the rate. Refusing a deadline a listening floor makes impossible. And proposing a trial batch on the modality genuinely concerned. One important practical consequence follows. The third practice protects both parties, a commitment made on a duration of audio shorter than the listening time needed being holdable only by degrading the work, which makes refusal preferable to an acceptance both parties will pay for.Approaching a multimodal project
Five questions scope it in data labeling. Which of the modalities carries the decisive information. The others follow. Is control possible by eye alone. Otherwise it costs dearly. Is the ambiguity spatial or else semantic. The method differs. Is the throughput compressible. Audio is not. And does competence transfer. Often yes, except in three dimensions. Those five answers determine the project’s structure. Asking them before production avoids applying one modality’s methods to another.The question that frames the choice of modality
One question determines the structure of a data labeling project. Is the need’s ambiguity spatial or semantic. A spatial ambiguity, where the object ends and the background begins, is settled by a geometric convention and verified by eye in a second. A semantic ambiguity, what this sentence expresses or what this sound means, is settled by a definition and verified by doing the work again. That question belongs to the nature of the expected disagreement, it is asked before the team is assembled, and it decides the share of budget devoted to control.Three decisions before choosing a modality
Three decisions commit a multimodal data labeling project. Identifying which of the modalities genuinely carries the decisive information. Costing the control separately for each of them. And writing the rule applying when two sources contradict each other. Those three decisions cost half a day, they precede any production, and their absence leads to handling data of which one part will bring nothing.Three checks before costing a modality
Three controls direct a data labeling costing. The real control time set against the production time observed. The agreement rate obtained between two operators on that modality. And the possible existence of an incompressible time floor. Those three controls are measured on a trial batch, they require only a day, and their absence leads to transferring a rate from one modality to another that bears no resemblance to it.Three questions before committing
Three questions precede a multimodal data labeling quotation. Which modality carries the information the use genuinely depends on. Under exactly what conditions was the data acquired. And has the provider already produced on that precise modality. Those three questions are asked before costing, they call for examining a real sample, and their absence leads to entrusting work whose difficulty nobody has measured.What this chapter teaches
One cross-cutting observation deserves closing this examination. What changes between modalities is the form of the ambiguity. Three findings compose it. Imagery makes control immediate, the other modalities do not. Text produces a structurally higher disagreement. And audio imposes a time floor nothing crosses. That finding extends the pillar chapter, the underlying difficulties staying identical beneath very different appearances.What this chapter leaves to the next
Choosing the modality leads to choosing who will do the work. Three questions stay open in data labeling. What an in-house team brings that a provider does not. Why volume and duration change that calculation. And what each option really costs once everything is counted. Those three questions belong to the choice between in-house and outsourced work, which constitutes the subject of the following chapter.Why the audio floor is worth defending
One point about deadlines on this modality belongs at the close. Clients routinely ask for audio turnaround that arithmetic forbids. Three reasons follow in data labeling. Duration of recording is the only figure they have to reason from. Imagery has taught them that throughput can be pushed. And a provider who accepts the deadline appears simply more capable. One important practical consequence follows for a data labeling provider. Explaining the floor rather than accepting the deadline is the only tenable position, since an hour of recording cannot be described without an hour of listening, and a team that agrees to compress that is not working faster but listening less.What the corpus keeps across modalities
Four elements stay identical whichever data is handled. A written reference that settles rather than describes. An omission measurement distinct from the other indicators. A deliberate coverage of the difficult situations. And documentation delivered alongside the data itself. One important observation follows for a data labeling project. Those four elements travel unchanged from one modality to the next, which explains why an experienced team adapts to a new type of data faster than its unfamiliarity with the material would suggest.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Applying to one modality the methods proven on another.
- Costing audio by duration without counting the re-listening.
- Neglecting double description on ambiguous text.
- Treating video as a series of independent images.
- Entrusting three-dimensional data to an untrained team.
- Assuming a high agreement rate in imagery will hold in text.
- Omitting to document the frame in three dimensions.
- Controlling text by sampling as light as in imagery.
- Ignoring the effect of recording quality on throughput.
- Comparing throughputs between modalities without relating them to control.
