The preceding articles set out the methods. This one examines what no method fully resolves: the configurations where the information is not in the images and where convention replaces observation.
It extends the article on quality control in video annotation.
The three families of difficulty
Three distinct causes produce the failures of video annotation. Occlusion, where the object is present and hidden by something else. Image degradation, where the object is visible and badly resolved, motion blur and compression notably. And identity ambiguity, where the object is perfectly visible and indistinguishable from another. One important practical consequence follows. Those three families call for different responses, the first a convention, the second an acquisition choice and the third a documented abandonment, which makes their confusion expensive in video annotation.The degrees of occlusion
Four situations are distinguishable. They do not call for the same treatment. Light partial occlusion, where the object stays identifiable and locatable within a margin of error. Heavy partial occlusion, where only a fraction stays visible and where localisation becomes an estimate. Brief total occlusion, where the object disappears for a few frames and where continuity makes the link obvious. And prolonged total occlusion, where the link rests on inference rather than on observation. One important observation follows. Only the fourth situation is genuinely undecidable, the first three belonging to a convention to be written rather than to a limit of the data.Annotating a partially hidden object
Three video annotation conventions coexist. Their choice changes the corpus. Annotating the visible part alone, which produces a box faithful to the observation and a size that varies with the masking. Annotating the complete estimated extent, which produces a stable size and a degree of invention. And annotating both, a visible box and an estimated box, which doubles the work and removes the ambiguity. One practical consequence follows for a video annotation project. The first convention produces a size variation the automatic checks flag as suspect, which requires accompanying that choice with an occlusion attribute so the check distinguishes a masking from an error.Motion blur
This degradation is specific to video annotation. It is often wrongly treated as a defect. An object moving rapidly spreads across several pixels during the exposure. Three consequences follow. The outline becomes indeterminate, which makes precise delineation impossible and disagreement between annotators inevitable. The position itself becomes ambiguous, the object occupying a streak rather than a point. And the appearance changes, a blurred texture no longer resembling the same texture sharp. One important observation follows. These frames will be present in production, which precludes discarding them from the corpus, and the convention must state where to place the box on a streak, at the start, the centre or the end.Re-identification after occlusion
This operation concentrates the difficulty of tracking. It rests on three cues. Trajectory continuity, an object reappearing where its movement predicted. Appearance similarity, colour and shape bringing two observations together. And context, an object reappearing in a configuration consistent with its surroundings. Three situations weaken those cues. Duration, a long occlusion making the trajectory prediction imprecise. Change of appearance, an object reappearing at another angle or under other lighting. And the presence of similar objects, which multiplies the compatible candidates. One important practical consequence follows for a video annotation project. Those three weakenings compound in dense scenes, which makes re-identification hardest exactly where it would be most useful.What compression destroys
This invisible degradation deserves setting out. It affects the small objects in a video annotation corpus. Video compression reduces volume by removing information judged barely perceptible. Four measurable effects follow. Small objects lose their texture and become patches of colour. Outlines produce blocky artefacts, which displaces the apparent boundary. Intermediate frames carry less information than keyframes, which makes quality variable within one sequence. And rapid movement triggers more aggressive compression, which degrades precisely the difficult moments. One observation follows. The last effect is the most penalising and the least known, compression and motion blur compounding on the same frames, which produces a degradation far greater than either would produce alone.Objects at the resolution limit
A fourth difficulty accompanies the preceding three and it is purely quantitative. A distant object occupies too few pixels to be reliably qualified. Four consequences follow. The class becomes uncertain, a silhouette of a few pixels potentially belonging to several categories. Delineation varies strongly between annotators, one pixel of difference representing a large share of the object. Detection itself becomes inconsistent, one object appearing and disappearing according to local contrast. And the cost per object rises while its value for training falls. One important practical consequence follows for a video annotation project. Those four consequences justify fixing a minimum size below which an object is not annotated, a figure that saves work and removes a major source of inconsistency.The undecidable class
This category must exist in any video annotation nomenclature. Its absence produces a silent damage. Some configurations do not carry the information needed for a decision. Three benefits accompany this class in video annotation. It avoids an arbitrary choice, which introduces noise into the corpus with nothing signalling it. It measures the genuinely undecidable share, information that bounds the achievable performance. And it locates the configurations to address, each marked case pointing at a missing convention or a limit of the data. One important observation follows. An abnormally low rate on this class indicates that annotators have stopped using it rather than an improvement, which makes it an indicator to interpret in both directions.What acquisition can correct
Four upstream decisions reduce these difficulties. They escape annotation itself. Exposure time, whose reduction lessens blur at the price of a darker image. Capture rate, a higher rate reducing the displacement between two frames and easing pairing. Compression bitrate, whose increase preserves small objects at the price of volume. And camera placement, a higher angle reducing mutual occlusions between objects. One practical consequence follows. The last decision is the most effective and the least expensive, a change of angle removing part of the occlusions rather than making them more legible, which places it ahead of any consideration of processing.The most difficult configurations
Five situations compound several of the difficulties set out and they deserve identifying at scoping. The crowd, where density, similarity and mutual occlusions compound. The crossing of two identical objects, a configuration where the information is missing by nature. The night scene, where sensor noise adds to compression and low contrast. Passing behind a wide obstacle, where the occlusion duration exceeds the reliable linking threshold. And the moving camera at speed, where blur affects the whole image rather than the moving objects alone. One observation follows. Those five configurations represent a modest fraction of the duration and a large share of the disagreement, which justifies sampling them specifically at control rather than hoping a random draw meets them.What assistance does not resolve
Three limits bound what assistance models contribute on these configurations. A model proposes a plausible link with no more information than an annotator has, which produces a confident and unfounded decision. Validation bias operates more strongly here than elsewhere, a proposal on a difficult case being precisely the one that should be examined. And an error on these configurations propagates, a switch at an occlusion contaminating everything that follows. One important observation follows for a video annotation project. Those three limits lead to a simple rule, assistance handles the easy segments and the human handles the occlusions and the crossings, a division that concentrates effort where the automatic fails.What multimodality brings
One complementary source lifts part of these difficulties and it deserves situating. A second sensor observes the same scene on a different principle. Three contributions follow. Occlusion is not the same from each viewpoint, an object hidden from one camera potentially being visible to another. Motion blur does not affect a range sensor as it affects a camera. And depth separates two objects superimposed in projection, which lifts part of the crossing ambiguities. Three limits accompany it. Registration between sensors introduces its own error source. Temporal synchronisation becomes a requirement, two sensors sampling at different rates. And the annotation cost grows, each modality adding a view to handle or verify. One observation follows for a video annotation project. These contributions justify a second modality where occlusions dominate the failures and not where it is added on principle, a distinction matching the finding the preceding series established on source fusion.What the convention must provide for
Six convention decisions cover most of these difficulties. They are written before production. The visibility threshold beyond which an object ceases to be annotated. The handling of an occlusion, interruption, suspension or interpolation through. The duration beyond which a link is no longer attempted. The position of the box on a blurred object. What to do when two candidates are equally plausible. And the attribute permitting an uncertain decision to be marked. One practical consequence follows for a video annotation project. The fifth decision is the most often omitted and the most decisive, the absence of a rule leading each annotator to settle by intuition and producing an inconsistency no check catches.What the annotator must be able to do
Five capabilities condition the correct handling of these configurations. Going back freely, a linking decision being taken by comparing before and after. Marking an uncertain decision without interrupting their work, failing which doubt resolves into an arbitrary choice. Consulting the case library from the interface, rather than memorising a long convention. Escalating an uncovered case to a lead, with the sequence and instant concerned. And seeing the already annotated trajectory, which makes a linking inconsistency immediately visible. One practical consequence follows for a video annotation project. The second capability is the most decisive and the most often absent, an annotator with no means of expressing doubt producing a decision nothing distinguishes from a founded one.Measuring what is undecidable
One operation quantifies that share and it bounds any contractual requirement. Two annotators handle a sequence containing occlusions and their links are compared. Three usable pieces of information follow. The agreement rate on links after occlusion, generally lower than agreement on detections. The occlusion duration beyond which that agreement collapses, which supplies the linking threshold to adopt. And the list of disagreement cases, the raw material of the reference. One important practical consequence follows. The second piece turns an arbitrary threshold into a measured value, an operation taking a few hours and replacing a contestable convention with a finding.What these difficulties cost
Four effects on the load accompany these configurations and they can be costed. Throughput falls sharply, a difficult case requiring examination where an ordinary one is handled in one gesture. Disagreement rises, which requires more frequent double annotation on these sequences. Arbitration becomes necessary, an edge case escalating to a lead rather than resolving on the spot. And rework after correction bears on whole segments, a switch resolved late forcing a review of everything after. One practical consequence follows for a video annotation project. Those four effects make the cost per minute of a dense sequence unrelated to that of an ordinary one, a gap the pilot batch must measure by deliberately including a difficult sequence.The first batch and the difficult cases
Four sequences compose a pilot batch that genuinely tests the conventions. An ordinary sequence, which supplies the reference throughput. A prolonged occlusion, which tests the linking threshold adopted. A crossing of similar objects, which tests the rule applying to equivalent candidates. And a sequence in degraded conditions, blur or low light, which verifies that the placement convention holds. One observation follows. Those four sequences represent a few minutes, they produce most of the disagreements that would otherwise surface in production, and their absence from the pilot batch explains most of the gaps between an announced throughput and an observed one.Framing a tenable requirement
A three-step method avoids an unachievable commitment on these configurations. Separate the requirement by configuration type, performance on ordinary sequences and on difficult ones not being comparable. Compare each requirement with the disagreement measured between annotators on the same configuration type. And declare the undecidable share explicitly, which bounds what a system can reach independently of its quality. One important observation follows. The third step is the one providers most readily omit and the one that best protects them, an undeclared undecidable share becoming at delivery an imputed defect rather than a property of the data.Explaining these limits to a client
Four framings make this conversation productive rather than defensive. Show the configuration rather than describe it, an image of two identical objects in contact being worth any explanation. Distinguish what belongs to acquisition, to convention choice and to the irreducible, which allocates responsibility without evading it. Propose the upstream correction where one exists, a camera repositioning being a constructive answer rather than an admission of defeat. And quantify the share concerned, a difficulty affecting a small fraction of the duration not justifying abandoning the project. One observation follows for a video annotation provider. The first framing is the most effective, a client understanding immediately an undecidability they see where they readily contest one they are told about.Approaching these difficulties in a project
Five questions scope these video annotation difficulties. They are asked before production. What proportion of the duration presents occlusions. That answer sizes the principal difficulty. Is motion blur present at working speeds. That answer requires a placement convention. Are the objects visually distinguishable from one another. A negative answer makes re-identification structurally limited. Can the camera be repositioned. That answer opens the most effective correction. And is an undecidable class provided. Its absence turns a limit into noise. Those five answers determine the conventions to write and the tenable requirement. Asking them before starting avoids committing to a continuity the data does not carry.The question that precedes the others
One question determines whether these difficulties are blocking or marginal. What share of the duration actually presents these configurations. A low answer places these difficulties at the margin, handling by convention and an undecidable class sufficing to absorb them. A high answer places them at the centre, which justifies an upstream correction, a higher rate or a second modality. That question is answered by watching a few minutes, it precedes any discussion of method, and it avoids sizing a whole project on a difficulty affecting only a fraction of it.What this chapter teaches
One cross-cutting observation deserves closing this examination. These difficulties fall into three categories of which only one belongs to annotation. Three findings compose it. Part belongs to acquisition, blur, compression and camera angle correcting upstream or not at all. Part belongs to convention, occlusions and thresholds requiring a written rule rather than additional effort. And part is irreducible, two identical objects in prolonged contact not carrying the information that separates them. That finding matches the one the preceding series established in other domains: distinguishing an error from a limit avoids spending effort correcting what cannot be corrected, and that distinction belongs at diagnosis rather than after several attempts.The three questions to ask about the footage
Three questions put to a client about their own material reveal more than any technical discussion. They are asked while watching a video annotation sample together. Where does the camera sit, and could it sit higher. That answer opens the cheapest correction available and it is almost never proposed by an annotation provider, which makes asking it distinctive. What is the exposure and the compression setting. Those two are usually chosen for storage or for human viewing rather than for machine use, and both are adjustable at no cost. And can the two objects the client cares most about be told apart by a person watching. A negative answer settles the feasibility question before any method is discussed. One practical consequence follows for a video annotation provider. Those three questions concern the acquisition rather than the annotation, they cost nothing to ask, and they position the provider as someone examining the whole chain rather than accepting whatever arrives at their end of it.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Treating motion blur as a defect and discarding the frames concerned.
- Annotating the visible part alone with no associated occlusion attribute.
- Not fixing the duration threshold beyond which a link is no longer attempted.
- Omitting the rule applying where two candidates are equally plausible.
- Neglecting the undecidable class.
- Reading a fall in the undecidable rate as progress.
- Expecting a tracking model to settle an undecidable configuration.
- Ignoring the compounding of compression and blur on rapid movement frames.
- Requiring an identity continuity the occlusion duration makes impossible.
- Neglecting camera repositioning, the most effective correction.
