In image detection, the deliverable is a box. In object tracking, the deliverable is an identity: the same vehicle must carry the same number from the first frame it appears in to the last. That difference changes the nature of the work far more than moving from a still image to a sequence does.
This article sets out what tracking requires, where it fails and how to scope it. It extends the guide to video annotation.
What tracking adds to detection
Four requirements distinguish a tracked sequence from a succession of annotated frames. The identifier must persist, an object keeping the same number throughout its presence on screen. The identifier must be unique, two distinct objects never sharing a number even at different moments in the sequence. The trajectory must be continuous, a visible object not being able to disappear and reappear under another number. And the transitions must be decided, entry into frame, exit and passing behind an obstacle each calling for a rule. One important practical consequence follows. Those four requirements concern the relations between frames rather than the content of each, which makes video annotation that is correct frame by frame potentially unusable for tracking.The characteristic error of tracking
This failure dominates tracked video annotation. It has no equivalent in image detection. An identity switch occurs where two objects exchange numbers or where an object resumes under a different number after an interruption. Three properties make it particularly damaging. It is invisible in an isolated frame, each frame staying correct taken separately. It propagates, a wrong number continuing to the end of the sequence or until the next switch. And it destroys the information tracking was meant to produce, a count or a trajectory analysis becoming wrong from the switch point onward. One important observation follows. This property requires quality control bearing on the sequence rather than on the frame, which radically distinguishes the arrangement from image annotation control.The three configurations that produce switches
Three situations concentrate most identity errors. The crossing of two similar objects, two people dressed in the same colour crossing producing an ambiguity appearance alone does not resolve. Prolonged occlusion, an object hidden for several seconds reappearing in a place and a pose that do not permit attaching it with certainty. And density, a scene containing many similar objects multiplying the possible pairings at every frame. One practical consequence follows. Those three configurations compound in the scenes that most interest clients, crowds, junctions and sporting fixtures, which means difficulty grows precisely where demand concentrates.The occlusion policy
This convention is the most decisive in a tracked video annotation project. It is nonetheless regularly omitted. A hidden object must receive a defined treatment, and three coherent policies exist. Interruption with reassignment, the track stopping at disappearance and a new number being assigned on reappearance, a simple policy producing fragmented tracks. Suspension with resumption, the track being marked occluded and then resuming the same number, a policy that preserves identity and presupposes the annotator can link the two segments. And interpolation through the occlusion, position being estimated during the masking, a policy producing boxes with no visual support and suiting certain uses only. One important observation follows for a video annotation project. Those three policies produce mutually incompatible corpora, and a model trained under one and evaluated under another shows a performance that measures nothing.The reappearance threshold
This quantitative decision accompanies the occlusion policy. Resuming a number after an interruption presupposes fixing the duration beyond which linking is no longer attempted. Three considerations guide that threshold. The nature of the scene, an object leaving frame on a busy street having little chance of returning where a player leaving a sports shot will certainly return. The annotator’s ability to decide, a link beyond a few seconds often resting on inference rather than on observation. And the downstream use, a count of passages tolerating fragmentation where an individual path analysis forbids it. One practical consequence follows. That threshold must feature in the convention with an explicit value, failing which each annotator applies their own and the corpus’s consistency degrades with no individual error being observable.Entry and exit conventions
Four transitions each require a written rule. Partial appearance at the frame edge, a half-visible object having to receive a track or not according to a visible-area criterion. Progressive exit, an object leaving frame having to be annotated up to a defined threshold. Appearance within the scene, an object emerging from behind an obstacle without ever having been visible before. And definitive disappearance, distinct from an occlusion and calling for a track closure rather than a suspension. One observation follows. The fourth transition is the hardest to settle at annotation time, since an annotator does not know whether a hidden object will return, which argues for a convention based on duration rather than on intention.What measurement must report
Two families of indicator coexist in video annotation. Confusing them produces misleading evaluations. Detection metrics measure whether objects are found and well located, independently of their identity. Identity metrics measure whether the same object keeps the same number, independently of localisation quality. Three consequences follow for a video annotation project. A system can excel at detection and fail at identity, a frequent situation a single indicator masks. The number of switches must be reported separately, a raw value being more telling than a composite indicator. And mean track duration usefully completes those measures, high fragmentation signalling a problem the rates do not show. One practical consequence follows. A contractual requirement framed without distinguishing those two families permits delivering a compliant corpus unusable for the intended purpose.Two ways of organising the work
Two video annotation methods coexist. Their choice affects throughput as much as quality. Frame-by-frame annotation handles every object in a frame before moving to the next, an intuitive method that favours switches since the annotator must re-pair the whole set at every step. Track-by-track annotation follows one object from the start to the end of its presence before moving to the next, a method that preserves identity and requires traversing the sequence as many times as there are objects. One important observation follows. The second method strongly reduces switches and costs more in traversal time, a trade-off leaning towards it as soon as identity constitutes the deliverable, which is to say in most tracking projects.What tracking costs
Five factors determine the load and they do not reduce to the video’s duration. The number of simultaneous objects, the dominant factor determining how many tracks must be maintained. Each object’s duration of presence, a long track costing more than a brief one even at equal throughput. The frequency of occlusions, each interruption requiring a linking decision. The similarity of objects to one another, a scene of identical objects demanding attention distinct objects do not. And the annotation rate adopted, distinct from the video’s rate and settled separately. One practical consequence follows. These factors multiply, which explains why a minute of dense video costs more than several minutes of sparsely populated video, and makes costing by duration particularly misleading in video annotation.The annotation rate
This decision precedes the work and it determines the volume. Annotating every frame is almost never necessary, a reduced rate sufficing for most uses. Three considerations guide that choice. Object speed, rapid movement requiring a higher rate for pairing to stay possible. The downstream use, a real-time system working at a given rate and having no need of a finer corpus. And the interpolation strategy, a reduced rate combined with interpolation producing a result close to exhaustive annotation at a fraction of the cost. One observation follows. This decision is the most powerful cost lever in a tracking project, and it is taken by examining object speed on a sample rather than by adopting a default value.What the annotator must see
Five interface functions condition the quality of tracking work and their absence produces switches whatever the competence. Display of the past trajectory, which permits checking a movement’s consistency without going back. Frame-by-frame navigation in both directions, indispensable for settling a crossing. Filtering by identifier, which isolates one track and permits traversing it alone. Visual distinction of identifiers, colour or persistent label, without which two neighbouring tracks merge. And flagging of interrupted tracks, which draws attention to pending links. One practical consequence follows for a video annotation project. Those five functions belong to tooling rather than to method, and their absence caps the achievable quality, which makes them a point to verify before committing to a production rather than after observing a high switch rate.Quality control on a tracked corpus
Four automatic checks detect most defects and they run without intervention. Detecting position jumps, a displacement incompatible with the object’s physical speed signalling a switch. Checking track durations, a distribution containing many very brief tracks signalling excessive fragmentation. Verifying identifier uniqueness, one number appearing twice in a frame constituting a formal error. And tracking box size, an abrupt variation signalling a change of object rather than a movement. One practical consequence follows for a video annotation project. Those four checks are programmed once and applied to a whole batch, which makes them the best ratio between setup effort and defects detected, and they exploit the temporal continuity image annotation does not supply.The attributes accompanying a track
Four pieces of information frequently join the box and the identifier, and each multiplies the work. Occlusion state, indicating whether the object is partly hidden and in what proportion. Truncation state, distinct from occlusion and signalling an object cut by the frame edge. The class, liable to change mid-track where an object becomes identifiable as it approaches. And domain attributes, a vehicle’s orientation or a person’s posture, which vary frame by frame. One important observation follows. The third case deserves an explicit rule, a distant object classified generically and then recognised precisely raising the question of whether the track changes class or keeps the initial one, a decision affecting training as much as evaluation.Model-assisted tracking
One method appreciably reduces the load and it calls for a precise caveat. A tracking model produces tracks the annotator corrects rather than creates. Three benefits follow. Easy segments are handled automatically, which represents most of an ordinary track’s duration. Effort concentrates on crossings and occlusions, which are precisely the decisive moments. And disagreements between the model and the annotator locate the difficult configurations, information usable for the next batch. Two caveats accompany it. Validation bias, an annotator accepting a plausible track more readily than constructing one, which lets discreet switches through. And error propagation, a switch produced by the model continuing until a check stops it. One observation follows for a video annotation project. Those two caveats argue for systematic verification at crossing and occlusion points rather than for control spread uniformly, which concentrates effort where the assistance fails.The delivery format
Four format decisions commit a tracked corpus’s exploitation and they are taken before production. File structure, an organisation by track and an organisation by frame carrying the same information and not reading the same way. The coding of occlusion, which must distinguish an observed box from a box estimated during masking. The treatment of unannotated frames, a reduced rate leaving intervals where the absence of a box does not mean the absence of an object. And retention of the link to the source, timestamp and sequence reference, without which an annotation cannot be replaced in its context. One practical consequence follows for a video annotation project. The third decision produces the commonest misunderstanding, a user reading the absence of annotation as an absence of object where it reflects an untreated frame, which distorts any metric computed without accounting for the rate.The scenes that pose the most difficulty
Four types of sequence concentrate identity errors and they are better identified at scoping. Crowds, where density and similarity compound and where the number of simultaneous tracks exceeds what an annotator can comfortably follow. Sports scenes, where players of one team wear identical kit and cross constantly. Moving cameras, where the viewpoint’s displacement adds to the objects’ and complicates pairing. And low-rate or heavily compressed scenes, where the displacement between two annotated frames becomes too large for reliable pairing. One observation follows for a video annotation project. Those four types require a higher rate and a track-by-track method, which produces a cost per minute unrelated to that of an ordinary scene, a gap that should appear in a costing rather than be discovered in production.What tracking does not solve
Four limits deserve stating. A total and prolonged occlusion stays undecidable, no observer being able to guarantee that a reappeared object is the same one. Two strictly identical objects that cross cannot be separated, the information not existing in the images. Re-identification between distinct cameras is a different problem, one that single-camera tracking does not prepare for. And the annotated trajectory says nothing about behaviour, an action analysis requiring a distinct temporal annotation. That first limit deserves emphasis. It means a project must accept a degree of undecidability and provide a class to record it, rather than demand a continuity the data does not carry.The first batch to handle
Four criteria compose a representative pilot batch for a tracking project. One low-density sequence and one high-density sequence, whose throughput gap reveals the principal cost factor. At least one crossing of similar objects, a configuration that tests the convention more reliably than dozens of ordinary frames. At least one prolonged occlusion, which tests the reappearance threshold adopted. And one entry and one exit from frame, which verify that the transition conventions are applicable. One practical consequence follows. That batch represents a few minutes of video, it supplies the real throughput per scene type and it reveals the cases the convention does not cover, which makes it the only serious basis for a costing in video annotation.Approaching a tracking project
Five questions scope this type of project and they are asked in one meeting. Is identity the deliverable or a convenience. A negative answer permits far cheaper image annotation. Which occlusion policy is adopted. That answer determines the corpus structure and its incompatibility with others. What reappearance threshold applies. That answer must be a figure and it must be written. What annotation rate. That answer is the principal cost lever. And how many simultaneous objects do the scenes contain. That answer determines the load far more reliably than total duration. Those five answers determine the volume, the method and the control arrangement. Asking them before starting avoids this field’s commonest situation, a corpus annotated frame by frame whose identities are not usable.What this chapter teaches
One cross-cutting observation deserves closing this examination. Tracking moves the deliverable from the object to the relation between objects. Three findings compose it. Quality is no longer verified on a frame but on a sequence, which renders inoperative any control arrangement designed for image annotation. The characteristic error is locally invisible and globally destructive, a property requiring automatic checks that exploit temporal continuity. And the transition conventions, occlusion, entry and exit, determine compatibility between corpora more than the quality of the boxes themselves. That finding matches the one the preceding series established in very different domains: what determines a corpus’s value is the documented definition of what it represents, and that definition here concerns relations rather than objects.The question that halves the budget
One question deserves asking before any other, since a negative answer removes most of this article’s difficulties. Does the downstream use actually need identity, or does it need counting. Three uses are regularly framed as tracking and do not require it. Counting passages across a line, which needs detection plus a crossing rule and no persistent identity beyond the moment of crossing. Measuring occupancy of a zone, which needs the number of objects present per frame and nothing about which object is which. And detecting an event, which needs the event’s frames identified and not the trajectory that led to it. One practical consequence follows for a video annotation project. Each of those three is served by a considerably cheaper corpus, and clients frequently request tracking because it is the familiar term rather than because the use requires identity. One observation completes that for a provider. Asking this question can reduce the engagement, and it establishes something a larger contract would not, which is that the scope was fitted to the need rather than to the request.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Annotating frame by frame where identity constitutes the deliverable.
- Not writing the occlusion policy.
- Leaving the reappearance threshold to each annotator’s judgement.
- Omitting the entry and exit conventions.
- Reporting a global performance without isolating identity switches.
- Checking quality on isolated frames rather than on sequences.
- Costing by video duration without accounting for simultaneous objects.
- Annotating at native rate without examining the objects’ real speed.
- Requiring an identity continuity occlusions make undecidable.
- Combining corpora produced under different occlusion policies.
