One figure captures the gap between annotating images and annotating video. A reference autonomous driving dataset describes its tracking component this way: 2,000 videos totalling around 400,000 frames, annotated at five frames per second, producing 130,600 track identities and 3.3 million bounding boxes.
Three million boxes for two thousand forty-second videos. And that at only five frames per second, a fraction of the real acquisition rate. That is the order of magnitude defining the field: video differs from images not through conceptual difficulty but through a multiplier on everything, on volume, on cost, and on the space temporal consistency occupies.
This guide covers the field end to end: tasks, formats, time-specific conventions, metrics, work organisation and quality control. It serves as the entry point to a set of articles each covering one aspect in detail.
What separates video annotation from image annotation
A video is a sequence of images, and that is exactly what intuition takes too literally. Four differences make it a distinct problem.Persistent identity
On a single image, it is enough to say what is visible and where. On video, you must say that the object in frame one thousand is the same as the one in frame one. That identity continuity is the information specific to video, the one no frame-by-frame annotation produces, and it is the most expensive thing to establish and to control.Redundancy
Two consecutive frames are almost identical. Annotating each independently means paying several dozen times for the same information, which is economically absurd and methodologically pointless. All productivity in the field rests on exploiting that redundancy, through interpolation, propagation or assisted tracking.Events
This is the family with no image equivalent at all. Some information does not exist on a single frame: a gesture, an action, an interaction, a change of state. It is defined by a temporal interval rather than a spatial region, which introduces a family of tasks with no equivalent in image annotation.Occlusion as the normal case
On an image, a partly hidden object is an edge case. On video, an object disappears behind another then reappears, and whether it is the same object is a central question. Occlusion is not an occasional difficulty, it is the mechanism that tests identity consistency.Task families in video annotation
Five families cover almost every video annotation project in practice, with very different costs and uses.Frame-by-frame detection
The simplest form: locating objects in each frame with no link between them. It produces a corpus usable for training a detector, but it does not exploit the temporal dimension and supports neither tracking nor trajectory analysis. It remains relevant where the intended use is instantaneous detection.Multi-object tracking
This is the field’s emblematic task. Each object receives an identifier maintained throughout its presence, allowing its trajectory to be reconstructed. The difficulty lies not in localisation but in managing identities through occlusions, crossings and exits from frame.Video instance segmentation
It combines pixel precision with identity persistence, making it the most expensive task in the field. It is justified where the object’s precise shape matters, in motion analysis, visual effects or manipulation robotics.Temporal annotation
It concerns intervals rather than regions: the start and end of an action, a phase of a process, the occurrence of an event. Its own conventions, notably defining the boundaries, make it a task in its own right, often underestimated because it looks simple.Keypoints and pose
Tracking articular points or landmarks on a deformable object is a fifth family. It combines the difficulties of tracking and of precision, and it is particularly sensitive to partial occlusion, a hidden limb raising the question of annotating a point that is not visible.Temporal conventions specific to video annotation
These are what make a video annotation corpus coherent, and their absence produces variability that spatial metrics do not detect.The birth and death of an identity
At what point does an object entering the frame start being annotated? As soon as one pixel is visible, above a given proportion, or once it is identifiable? The same question arises on exit. Without a written rule, each annotator decides differently and trajectory durations become incomparable.Reappearance after occlusion
This is the most decisive convention. Does an object hidden for several seconds then reappearing keep its original identifier or receive a new one? The answer depends on use: a counting system requires continuity, an instantaneous detection system does not. The protocol must set a maximum occlusion duration beyond which identity is considered lost.Annotating during occlusion
Should a fully hidden object continue to be annotated, with its position estimated? Both options exist and serve different purposes. Annotating the invisible teaches a model object permanence; not annotating it reflects what is observable. Mixing both within one corpus produces a contradictory signal.Event boundaries
Does a gesture begin when the hand moves, when it reaches the object, or when contact is made? That imprecision is the main source of disagreement in temporal annotation, and it is settled by a written definition with examples, never by implicit consensus.Interpolation, propagation and pre-annotation
Video annotation productivity rests on exploiting redundancy, and three mechanisms serve that. Keyframe interpolation means annotating only one frame in every n, with the tool computing intermediate positions. It works well on regular motion and becomes wrong as soon as the trajectory changes abruptly. The practical rule is to place a keyframe at every change of direction or speed rather than at fixed intervals. Propagation by automatic tracking advances the annotation frame by frame following the object’s appearance, with the annotator correcting when tracking drifts. It performs markedly better than interpolation on complex motion, and its limit lies in occlusions and visually similar objects. Model pre-annotation produces a complete proposal the annotator corrects. It is now very effective on common objects and raises the anchoring problem familiar from elsewhere: faced with a plausible proposal, an annotator accepts more readily than corrects. The countermeasure is reserving a fraction of the corpus for unassisted work and comparing. These three mechanisms reduce cost substantially but displace the work rather than removing it. Effort concentrates on the difficult frames, crossings, occlusions, rapid changes, which are precisely those carrying the highest learning value.Video annotation metrics
Spatial indicators are not sufficient in video annotation, and the tracking field has developed its own metrics that apply as much to controlling a corpus as to evaluating a model. The multi-object tracking literature uses a settled set: trajectory identifier metrics include identity precision, identity recall, the IDF1 score and identity switches between tracklets, while overall tracking accuracy measures the joint performance of tracking and detection. That distinction is decisive for quality control. A detection metric measures whether objects are found and well positioned. An identity metric measures whether continuity is respected. A corpus can be excellent on the first and mediocre on the second, and that is the commonest case since temporal consistency is what annotators master least. The identity switch count is the most directly interpretable indicator for quality control purposes. It counts the times a trajectory changes identifier without reason, which is a pure annotation error. Reporting it per sequence, broken down by probable cause, occlusion, crossing, exit from frame, directs correction immediately.Formats
A video annotation project handles three things that must stay consistent: the frames, the annotations, and the relationship between them. The first point of vigilance is the temporal reference. An annotation can be tied to a frame number, a timestamp or a timecode, and those three do not necessarily coincide, notably where the video has been resampled or where the frame rate is variable. Fixing and documenting the convention is the precondition to any conversion. The second is the relationship between the video file and the extracted frames. Most tools work on extracted frames, and the correspondence between frame number n and the position in the source file must be established reproducibly. Silent resampling during extraction produces an offset that renders the corpus unusable for any fine temporal analysis. The third concerns the structure of the annotations themselves. The format from multi-object tracking challenges is widely adopted and rests on one line per object per frame, carrying the sequence identifier, frame number, object identifier and geometry. Its simplicity makes it a good common denominator, and it suits rich annotations poorly, masks or multiple attributes calling for structured formats.Cost and what governs it
The video annotation multiplier is such that intuitions carried over from image annotation systematically underestimate. Four variables govern the cost of a video annotation project. The chosen frame rate, which directly multiplies volume: annotating at twenty-five frames per second costs five times more than at five. The number of simultaneous objects, which multiplies work per frame and complicates identity management. The density of occlusions and crossings, which determines the share of work that cannot be automated. And the primitive, a box costing a fraction of a mask. Frame rate deserves particular attention because it is often chosen by default. Annotating at the acquisition rate is rarely necessary: an object moves little between two frames at twenty-five frames per second, and redundancy is then near total. The dataset cited in the introduction annotates at five frames per second, which remains sufficient to learn tracking while dividing volume by five. Choosing that rate from object speed and intended use is the first cost lever in a video project.Application domains and their constraints
Video annotation uses fall into a few families whose requirements differ markedly. Mobility and autonomous driving is the most documented domain and the most demanding in volume. Scenes are dense, occlusions constant and frame rates high, making it the most expensive case. The reappearance convention is critical there, since a vehicle masked by another and then reappearing must be tracked continuously for trajectory prediction to work. Surveillance and flow analysis pose a different problem: few objects but long durations, and a requirement for continuity over several minutes. Counting often being the purpose, an identity switch directly produces a counting error. Sport and performance analysis combine visually very similar objects, frequent occlusions and a precision requirement on position. It is the domain where identity switches are hardest to avoid, players on the same team being nearly indistinguishable. Industrial and gesture video, finally, is more interested in events than trajectories. Annotation there concerns temporal intervals primarily, which shifts the difficulty from identity continuity towards boundary definition.Quality control in video annotation
The arrangement follows the general logic of annotation, with one major inflection: control must address time as much as space. Sampling isolated frames, as practised in image annotation, detects no continuity errors. An identity swapped between two objects produces two perfectly correct frames taken separately. Detecting that error requires examining the sequence, or computing identity metrics across the full trajectory. Three control mechanisms combine effectively. Automatic consistency checking, flagging trajectory anomalies, position jumps, brief disappearances, size changes incompatible with the dynamics. Sequence review, where a reviewer watches the annotated video at speed, which makes temporal inconsistencies immediately visible where they are invisible frame by frame. And computing identity metrics on a double-annotated sample. Accelerated playback review deserves flagging because it is cheap and remarkably effective. The human eye instantly catches an identifier that jumps or a box that drifts, whereas the same error is undetectable in a still capture.Selecting what to annotate
Volume makes annotating everything collected impossible in video annotation, so selection becomes design work in its own right. Three distinct levels of selection combine here. Choosing sequences, which must span the diversity of conditions, lighting, density, weather, rather than favouring the longest or easiest. Choosing segments within a sequence, since a forty-second video is not uniformly interesting, static passages contributing little and crossings a great deal. And choosing the frame rate, already discussed. Recent literature addresses this problem and highlights a video-specific point: existing active learning methods for multi-object tracking operate primarily at frame level, which is structurally misaligned with modern end-to-end trackers whose inference and training rely on multi-frame clips. The practical consequence is that selecting isolated frames, as one would on an image corpus, destroys the temporal context learning needs. Selection in video annotation is done by continuous segment, never by frame, which is an important and frequently ignored methodological difference.Organising the work
Two principles separate the organisation of a video annotation project from an image project. The first is the unit of work. A sequence must be annotated by one person from start to finish, because identity consistency does not transfer between annotators. Splitting a sequence across several people produces breaks at the junctions, and it is a frequent organisational error on long videos. The second is cognitive load. Tracking multiple identities across a long sequence is mentally demanding, different from repeating a gesture on independent images. Fatigue there shows up as identity errors rather than imprecise drawing, and it calls for shorter sessions than image tasks. Those two principles bear on sizing: a video team’s capacity does not add up as simply as on images, since parallelisation happens at sequence level rather than frame level.The annotator profile
The competence a video annotation task requires differs appreciably from an image annotator’s, and underestimating that explains part of the recruitment difficulty on these projects. Three aptitudes stand out. Visual working memory, since following a dozen simultaneous identities means holding who is who through occlusions. Attentional endurance, the task being long and continuous rather than divided into short units. And convention discipline, since an annotator freely interpreting the reappearance rule produces an incoherent corpus despite impeccable drawing. These aptitudes can be measured and trained. A qualification test on a short sequence containing occlusions and crossings immediately reveals whether a candidate manages identities correctly, and that measure is more predictive than their drawing quality on still images. Training benefits from covering the temporal conventions first, since they are what image annotation practice does not teach.What the client must supply
A video annotation project depends on a few inputs that only the client holds, and their absence explains most scoping failures. Four items matter. The source video in its original form rather than a re-encoded copy, since compression artefacts and resampling both degrade what can be annotated and break the temporal reference. The intended use stated concretely, since tracking for counting, for trajectory prediction and for behaviour analysis impose different continuity conventions. A representative sample including the hard passages, dense crowds, poor lighting, rapid motion, rather than the clean footage a demonstration would use. And the acceptable frame rate, or the willingness to determine it jointly. That last point is worth pushing on. Clients frequently specify the acquisition rate by default, and a short conversation about object speed and the decision the model must support usually reduces it substantially. Since frame rate multiplies every other cost, that conversation is the single most valuable half hour in scoping a video project.The most common mistakes
These failures recur often enough across video projects that naming them is usually enough to avoid them.- Annotating at the acquisition frame rate without checking whether a lower rate suffices.
- Not writing the reappearance-after-occlusion convention.
- Mixing objects annotated during occlusion with others not annotated, in one corpus.
- Controlling by sampling isolated frames, which detects no identity errors.
- Splitting a sequence across several annotators.
- Confusing temporal reference with frame number after resampling.
- Placing keyframes at fixed intervals rather than at trajectory changes.
- Reporting only detection metrics with no identity metric.
- Letting pre-annotation drift into passive validation on the difficult cases.
- Transposing a cost estimate from an image project.
