A video corpus can be correct frame by frame and unusable as a whole. That possibility defines the particular difficulty of video quality control: the gravest defects are visible nowhere in an isolated frame.
This article sets out how to detect them. It extends the article on video annotation tools.
Why control differs here
Four properties distinguish controlling a video annotation corpus from controlling an image corpus.
The characteristic defects concern relations between frames rather than their content.
The volume precludes exhaustive verification, one annotated minute containing thousands of objects.
The redundancy between neighbouring frames makes random sampling uninformative, two consecutive frames carrying almost the same information.
And continuity supplies automatically verifiable constraints an image corpus does not.
One important practical consequence follows. Those four properties render inoperative an arrangement designed for image annotation, and they simultaneously open up more powerful automatic checks.
The defects specific to video
Six failure modes do not exist in image annotation.
The identity switch, two objects exchanging their identifier.
Fragmentation, a single track cut into several by missed links.
Box drift, a localisation progressively shifting with no identifiable point error.
Instability, an object detected then missed then redetected across consecutive frames.
Attribute carry-forward, a value filled at a keyframe and propagated while it changes.
And inconsistency at task boundaries, a track cut by an administrative division receiving two identities.
One important observation follows. Those six defects share a property, each leaves every frame individually correct, which makes them undetectable by any control bearing on isolated frames.
The automatic checks
Six verifications exploit temporal continuity.
They run with no intervention.
The position jump, a displacement incompatible with physical speed signalling a switch.
The abrupt size variation, which signals a change of object rather than a movement.
The distribution of track durations, an excess of very brief tracks revealing fragmentation.
Identifier uniqueness per frame, one number appearing twice constituting a formal error.
The perfect linearity of a trajectory, the signature of interpolation over too wide an interval.
And attribute stability, a value constant across a whole long track being suspect for an attribute meant to vary.
One practical consequence follows for a video annotation project. Those six checks are programmed once and apply to every batch, which makes them the best ratio between setup effort and defects detected in the whole arrangement.
What the automatic checks do not see
Four video annotation defects escape any programmed verification.
Omission, an object never annotated producing no detectable anomaly.
Systematic class error, a consistent confusion between two classes passing every formal check.
Imprecise delineation, a box too wide or too narrow staying plausible.
And misapplication of a convention, an annotator faithfully applying a rule they misunderstood.
One important observation follows. Those four defects require human control, which makes automatic and human two complementary arrangements rather than substitutable ones.
How to sample
Four principles govern sampling in video annotation.
They differ from a random draw.
Sample sequences rather than frames, consistency being observable only over a duration.
Target the moments of risk, crossings, occlusions and transitions rather than arbitrary instants.
Include the task boundaries, division points where inconsistencies concentrate.
And keep a constant sample between batches, the only way to measure drift over time.
One practical consequence follows for a video annotation project. The second principle multiplies the control’s effectiveness, one minute taken around an occlusion revealing more than ten minutes of ordinary sequence.
Control by accelerated viewing
This simple method detects several defects.
It is underused in video annotation.
The principle is to play an annotated sequence at speed with the annotations overlaid.
Three video annotation defects become immediately visible.
Flickering, revealing a detection instability no frame-by-frame examination shows.
The box jump, a position that leaps catching the eye far more than a comparison of coordinates.
And the change of identifier colour, which makes a switch visible instantly.
One important observation follows. This method exploits human vision’s sensitivity to abnormal movement, it handles a minute in a few seconds, and it is the best-return human control in video annotation.
Detecting omissions
This defect is the gravest and the only one no automatic check reveals.
An object never annotated produces no anomaly, verification bearing on what exists.
Three methods detect it, and their cost differs strongly.
Exhaustive re-annotation of a short sequence, a reliable and expensive method whose result bounds the corpus’s omission rate.
Comparison with an existing model, whose unannotated detections flag candidates to verify.
And the inverse continuity check, a track appearing in the middle of the scene without coming from an edge signalling that it existed before without being annotated.
One important practical consequence follows for a video annotation project. The third method is automatable and specific to video, an appearance with no origin constituting a physical anomaly continuity makes detectable, which recovers a share of the omissions with no re-annotation.
Measuring disagreement
This operation bounds what can be promised.
It is conducted on a short sequence.
Two trained annotators handle the same sequence and the results are compared.
Four usable pieces of information follow.
Agreement on the existence of objects, which measures reciprocal omissions.
Agreement on identities, two annotators potentially producing different track divisions.
The localisation error, distinct from the two preceding.
And the list of disagreements, the raw material of the case library.
One practical consequence follows. The second point is specific to video and rarely measured, two annotators applying the same occlusion policy nonetheless producing different tracks on the edge cases.
The indicators to track per batch
Five values are read automatically.
Their evolution reveals drift.
The mean number of objects per frame, whose fall signals omissions.
The mean track duration, whose fall signals growing fragmentation.
The number of keyframes per track, whose fall signals a slackening on placement.
The rate of objects marked uncertain, whose fall often reflects a slackening rather than progress.
And the rejection rate at validation, which measures upstream quality.
One observation follows for a video annotation project. The third indicator is specific to this task and highly revealing, a hurried annotator spacing out keyframes and letting interpolation produce a plausible and wrong result.
Checking task boundaries
One point of fragility deserves separate treatment since it belongs to how the work is organised.
A long sequence is divided into tasks entrusted to different annotators.
Three inconsistencies concentrate there.
Identity breakage, a track crossing the cut receiving two identifiers.
Convention divergence, two annotators having interpreted a rule differently on neighbouring batches.
And double annotation or omission on the overlap frames, where the division provides a margin.
One observation follows for a video annotation project. Those three inconsistencies concentrate on a minute fraction of the corpus, which makes their systematic verification very economical, and they produce structural defects a random sample would almost never encounter.
Controlling an interpolated corpus
Four verifications are added in video annotation where some positions are computed.
Examining the frames at the middle of the intervals, positions furthest from an observation.
Detecting perfectly straight trajectories over a long duration, the signature of too wide an interval.
Checking attributes on the intermediate frames, whose mechanical carry-forward is the commonest defect.
And verifying that the delivered format distinguishes observed from computed position.
One practical consequence follows for a video annotation project. The last verification concerns not quality but usability, a corpus without that distinction not permitting a valid evaluation set to be built, which makes it a conformity check rather than a quality one.
The correction cycle
Four stages organise the improvement, and they repeat between batches.
Running the automatic checks on the delivered batch.
Accelerated viewing of a sample targeted on the moments of risk.
Grouping the defects by type rather than correcting them individually.
And writing the missing rule before the next batch.
One important practical consequence follows. The third stage distinguishes a correction from an improvement, correcting a hundred occurrences of one defect without writing the rule guaranteeing it will reappear in the next batch.
What control reveals about the conventions
One loop organises the improvement and it transforms the nature of the control.
Every defect detected belongs to one of two categories only.
The individual error, where an annotator misapplied a clear rule, corrected by a redo.
The convention defect, where the rule did not cover the case, corrected by a piece of writing.
Three consequences follow.
Classing each defect into one of the two categories is more useful than correcting it.
A high proportion of the second category indicates that the reference is incomplete rather than the team insufficient.
And that proportion falls from batch to batch where the loop works, which makes it an indicator of project maturity.
One practical consequence follows for a video annotation project. This classification costs a few minutes per batch and it directs effort towards writing rather than towards correcting, a shift that distinguishes an arrangement that progresses from one that catches up.
Framing a verifiable requirement
A three-step method produces a tenable requirement in video annotation.
Distinguish the error families by what they cost the use, an identity switch and a box imprecision not carrying the same consequence.
Frame the requirement by family rather than as a global rate, a single indicator permitting the contract to be met while failing on what matters.
And verify that each requirement stays below the measured disagreement between annotators on the same dimension.
One important observation follows. The third step protects both parties, a requirement above the measured bound being unachievable, and demonstrating it with a figure turns a negotiation into a finding.
What the control must report
Five elements compose a useful control report in video annotation.
The sampling method, with the targeted part and the random part.
The results of the automatic checks, alerts raised and alerts confirmed.
The disagreement measured between annotators, on existence, identity and localisation.
The evolution of the indicators between batches.
And the list of configurations for which no stable convention could be established.
One observation follows. The last element is the rarest and the most useful, since it turns an invisible limit into a declared one, a principle every preceding series established.
The record of decisions
One document accompanies the reference and it is almost always absent.
Every rule written during a project results from an observed defect, and the case that motivated it is lost faster than the rule.
Three reasons justify recording it.
A rule whose origin is known is revised knowingly when the context changes.
An old batch is interpreted correctly when one knows which rules applied when it was produced.
And an extension by another team presupposes understanding why a convention was settled that way.
One observation follows. This document costs one sentence per decision at the moment it is taken, it cannot be reconstituted afterwards, and in video it is the only way to explain why two batches of one corpus differ on a point of convention.
What control costs
Four lines compose this video annotation control.
Their distribution is surprising.
Setting up the automatic checks, a one-off cost of a few hours.
Running them, near nil and repeatable across every batch.
Human verification of the alerts, proportional to the number of real defects.
And accelerated viewing, very fast against the duration handled.
One practical consequence follows. This line costs markedly less in video than in images at equal object volume, temporal continuity supplying verifiable constraints independent images do not, which contradicts the intuition of a heavier control.
The minimal arrangement
Five measures address most of the difficulties set out and their total cost stays modest.
The six automatic checks, programmed before the first batch and run on every subsequent one.
Accelerated viewing of a sample targeted on the moments of risk, at every batch.
Systematic verification of the task boundaries, a short and high-return operation.
Measuring disagreement between two annotators, conducted once at project start.
And tracking the five indicators per batch, a simple table revealing drift.
One observation follows for a video annotation project. Those five measures take a small share of production time, they require no proprietary tooling, and they turn a quality claim into a verifiable fact.
Controlling the first batch
Four verifications apply to the first batch and to it alone.
The feasibility of the conventions, a frequent uncovered case requiring a piece of writing before continuing.
The throughput observed, which replaces the estimate and serves as the basis for costing subsequent batches.
Disagreement between annotators, which bounds what can be promised contractually.
And the exported format, whose early verification avoids discovering an incompatibility after several batches.
One observation follows. Those four verifications measure not the batch’s quality but the arrangement’s viability, a distinction that justifies treating the first batch differently rather than applying the routine control to it.
Demonstrating quality to a client
Four demonstrations convince more reliably than a rate and they are prepared in advance.
Accelerated viewing of an annotated sequence, which shows stability in seconds where a figure stays abstract.
The list of automatic alerts raised and handled, which demonstrates an arrangement rather than a result.
Comparison against the measured disagreement between annotators, which situates the performance obtained relative to what is achievable.
And the case library itself, whose richness demonstrates a method indicators do not establish.
One observation follows for a video annotation provider. The first demonstration is the most effective and the fastest, a client understanding immediately what a stable track is and what flickering is, with no explanation of a metric needed.
Approaching the control of a video corpus
Five questions scope a control arrangement in video annotation.
Which video-specific defects does the project risk. That answer directs which checks to program.
Are the automatic checks planned from the first batch. Adding them late lets the first batches through.
Is the sample targeted on the moments of risk. A random draw wastes the effort.
Is a constant sample kept between batches. Its absence makes drift undetectable.
And has disagreement on identities been measured. Its value bounds what can be promised.
Those five answers determine the arrangement and its cost. Asking them before production avoids observing after the fact a structural defect the first batches already contained.
Three checks to set up first
Three arrangements address the majority of defects and they are set up before the first batch.
The position jump check, which detects identity switches, the gravest and commonest defect.
Verification of the task boundaries, which addresses a known point of fragility at minimal cost.
And accelerated viewing of a sample, which reveals the instabilities no figure shows.
Those three are built in a day, they apply to every subsequent batch, and they cover what distinguishes a usable video corpus from one that is correct frame by frame.
What this chapter teaches
One cross-cutting observation deserves closing this examination.
Video makes some defects invisible and others automatically detectable.
Three findings compose it.
The characteristic defects are visible in no isolated frame, which condemns any arrangement imported from image annotation.
Temporal continuity supplies verifiable physical constraints, which makes automatic control more powerful than in images.
And accelerated viewing exploits a human capacity frame-by-frame examination does not, which makes it the natural complement to programmed checks.
That finding matches the one the preceding series established about controls exploiting a structural constraint: where the data carries a regularity independent of the annotated content, it serves verification as much as production.
The advantage this gives a provider
One consequence of this chapter deserves stating separately, since it concerns competitive position rather than method.
Video quality control is unusually demonstrable, and most providers do not demonstrate it.
Three properties of video annotation control make that so.
The checks produce a visible artefact, a list of alerts raised and resolved being a document rather than an assertion.
The accelerated viewing is watchable by a client with no training, which turns quality from a claim into an observation.
And the disagreement measurement produces a figure the client can compare against what any provider would achieve.
One practical consequence follows for a video annotation provider. Those three artefacts cost little to produce and they are absent from most competing proposals, which makes including them a differentiator disproportionate to their cost.
One qualification keeps this honest. They only differentiate where they are real, and a provider showing an alert list on a corpus with no checks behind it is exposed by the first serious question.
Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.
- Checking isolated frames rather than sequences.
- Sampling randomly rather than at the moments of risk.
- Adding the automatic checks after the first batches.
- Neglecting the task boundaries, where inconsistencies concentrate.
- Not measuring disagreement on identities, specific to video.
- Reading a fall in the uncertain rate as progress.
- Correcting defects one by one without writing the missing rule.
- Omitting the tracking of keyframes per track.
- Expecting the automatic checks to detect omissions.
- Delivering a report without the list of unstabilised configurations.
The check that pays for itself first
A closing note names the single measure worth building before any other.
The position jump check.
Three reasons put it first.
It targets the identity switch, which is both the gravest defect and the most frequent.
It needs no reference, since it compares a track against itself rather than against a ground truth.
And it runs on a delivered export without access to the annotation tool, which makes it available even where the production happens elsewhere.
That last property is worth noting by anyone receiving annotated video from a supplier: this check can be run on what arrives, by the recipient, with no cooperation required.
What to take away
The characteristic defects of a video corpus leave every frame individually correct, which makes them undetectable by an arrangement designed for image annotation.
Three readings emerge. Six automatic checks exploit temporal continuity and are programmed once to apply to every batch, which makes them the best ratio between effort and defects detected. Accelerated viewing with annotations overlaid exploits human sensitivity to abnormal movement, it handles a minute in seconds and it reveals instabilities no frame-by-frame examination shows. And control costs less in video than in images at equal object volume, continuity supplying verifiable constraints independent images do not.
For the difficulties this arrangement reveals without resolving, the article on occlusions and re-identification sets them out. For the tooling that makes these checks possible, the article on video annotation tools details the options.
To explore delivery arrangements, supported formats and applicable control mechanisms, see our dedicated page on 2D and 3D annotation. And if you need quality control set up on a video corpus, let us discuss your project.
