Annotating a video means annotating images that resemble each other. The difficulty comes not from the number of boxes to draw but from a piece of information no image holds: knowing that the object seen now is the one already seen before.This article sets out that work. It extends the article on file formats.
What video adds
Five requirements separate a sequence from a set of images.Each object carries a constant identity from one frame to the next.That identity must survive a temporary disappearance.The box must stay stable where the object is not moving.An object’s appearance as well as its exit are marked explicitly.And consistency is judged on the sequence rather than on the frame.One important practical consequence follows. The first requirement carries on its own most of the work in bounding box terms, an identity being a purely relational piece of information no single frame permits verifying.What breaks an identity
Five situations lose track of an object followed in bounding box work.A complete passage behind an obstacle in the scene.An exit from the frame followed by a later return.A crossing with an object of similar appearance.An abrupt change of appearance or else of lighting.And a cut between shots introduced in the edit.One important observation follows for a bounding box project. The third case produces the most expensive error, two objects of similar appearance crossing being able to swap identities with nothing to signal it, which contaminates the whole rest of the sequence.What the convention must fix
Six points make a tracking instruction genuinely applicable.The reattribution rule applying after a disappearance.The exact duration beyond which an identity is abandoned.The treatment given to an object whose identity becomes doubtful.The fate given to identities at a cut between shots.The conduct to adopt on an object both immobile and masked.And the explicit marking of entry as well as of exit.One important practical consequence follows for a bounding box project. The third point is almost always missing, an uncertain operator having to choose between keeping a false identity and creating a new one with no justification at all, which produces two different errors depending on the operator.What the measurement must judge here
Five indicators describe the quality of tracking in bounding box work.The rate of objects correctly detected, measured frame by frame.The number of identities swapped between two distinct objects.The number of trajectories cut and resumed under a wholly different number.The average duration for which an identity stays correct.And the stability of the box drawn on an object that stayed still.One important practical consequence follows. The first indicator can stay excellent while the tracking is unusable, every box taken separately being correct and the identities all mixed up, which makes a frame-by-frame measurement insufficient for judging tracking work.What interpolation permits
Four benefits follow from drawing bounding boxes on key frames.The number of boxes drawn by hand falls very sharply.The trajectory obtained becomes quite naturally regular.The box’s tremor disappears between two keys.And correction concentrates on the moments of change alone.One observation follows. Those four benefits presuppose regular motion, an abrupt trajectory or else an object that accelerates producing entirely false intermediate positions, which calls for closer key frames wherever the motion changes.What the sampling rate decides
Five consequences follow the choice of step between two annotated frames.The total volume of work to be done on the sequence.The fineness of motion that will effectively be captured.The real reliability of the interpolated positions.The effective detection of the briefest events.And comparability with any other sequence in the corpus.One practical consequence follows. The fourth point often settles the debate, an event shorter than the step finally adopted being able to pass between two annotated frames, which makes the step depend on the duration of the phenomenon observed rather than on the budget.What pre-annotation changes here
Four effects follow from using automatic tracking.Simple, well separated trajectories are then produced on their own.Crossings remain the most habitual point of failure.An identity error propagates across every following frame.And the operator checks a continuity rather than drawing boxes themselves.One important observation follows for a bounding box project. The third effect changes the nature of the control, a single identity error corrupting hundreds of frames where a badly drawn box affects only one, which justifies concentrating the rereading on the moments of crossing rather than spreading it uniformly.What the delivery format requires here
Four choices condition the use of a video corpus.An object identifier unique and constant throughout the sequence.An explicit frame number rather than a plain implicit order.A marking of interpolated boxes as against drawn boxes.And an explicit indication of shot cuts within the sequence.One important practical consequence follows for a bounding box project. The third choice is neglected and deprives the client of useful information, a merely interpolated box not having the same value as one verified by an operator, which permits setting calculated positions aside during a demanding evaluation.What throughput becomes here
Four factors determine the time spent per sequence.The number of objects present simultaneously in the frame.The frequency of crossings as well as of disappearances.The regularity of the motion, which decides how useful interpolation is.And the number of key frames that prove necessary per object.One important observation follows for a bounding box project. The second factor explains the gaps between sequences of comparable appearance, two videos of the same length and the same apparent density being able to demand very different times depending on whether the objects cross or circulate separately, which makes an estimate per minute of video misleading.What the corpus must cover here
Five situations deserve an explicit place in the corpus.Simple, well separated trajectories, serving here as a reference.Crossings between objects of very similar appearance.Disappearances followed by a return into the visible frame.Marked changes of lighting during one same trajectory.And shot cuts placed within one same sequence.One important practical consequence follows for a bounding box project. The second situation is often missing and deprives the corpus of its main interest, a sequence where the objects never cross in no way testing the tracking, which produces a reassuring corpus and a system that will fail at the first real scene.Three errors proper to video
Three bounding box defects belong to tracking rather than to drawing.An identity swapped at a crossing and then never recovered.An interpolation applied to motion that in no way lends itself to it.And quality judged frame by frame when the work is one of sequence.Those three defects nonetheless leave every box perfectly correct, they reveal themselves by following a trajectory from beginning to end, and their common point is to bear on continuity rather than on position.What the annotator must know here
Five kinds of knowledge condition correct work on a sequence.The reattribution rule applying after a temporary disappearance.The conduct to adopt when a crossing makes an identity doubtful.The precise moment to place a key frame rather than let interpolation run.The way to verify a whole trajectory rather than a single frame.And the reflex of going back as soon as a doubt appears.One important observation follows for a bounding box project. The last kind limits how far errors reach, a doubt lifted immediately costing only seconds whereas a false identity left in place propagates across the whole rest of the sequence, which makes going back a saving rather than a loss of time.What the client must specify here
Four pieces of information come from the client rather than from the provider.What the use will precisely calculate from the trajectories delivered.The duration of the briefest phenomenon they wish to be able to observe.The conduct expected when an object leaves the frame then returns to it.And the tolerance they possibly grant to an identity swap.One important practical consequence follows for a bounding box project. The second piece fixes the sampling step with no negotiation possible, a phenomenon lasting less than the interval finally adopted escaping observation purely and simply, which turns a question of budget into a technical constraint.What video changes about the project’s economics
Four effects distinguish the budget of a video project.The cost per frame falls sharply thanks to interpolation.The cost per object tracked far exceeds that of an isolated box.Control then bears on trajectories rather than on frames.And a rework costs a great deal since it affects a whole sequence.One important observation follows for a bounding box project. The first effect regularly misleads at the moment of the quotation, a per-frame charge giving a very attractive figure on a video whereas the real work bears on the trajectories alone, which makes a badly chosen billing unit more expensive than a high rate.What the working tool changes here
Five functions make this work practicable.Frame-by-frame navigation, forwards as well as backwards.A display of a given object’s complete trajectory.An automatic propagation of the box between two key frames.A flagging of identities newly appeared or disappeared.And the possibility of merging or else splitting two trajectories.One important practical consequence follows for a bounding box project. The last function avoids the most discouraging work, an identity error discovered late being corrected by a plain merge in seconds whereas it would otherwise demand reworking hundreds of boxes one by one.What the first batch must establish
Four results justify a trial sequence before production.The average number of objects present simultaneously in the frame.The frequency of crossings set against the total duration.The number of key frames that prove necessary per object.And the identity swap rate observed between two distinct operators.One important observation follows for a bounding box project. The second result determines the cost better than the total duration does, a sequence dense in crossings demanding several times the time an equivalent sequence with well separated trajectories requires, which makes the sample more informative than any description of the need.What this subject has that is particular
Four traits distinguish video from the course’s other chapters.The information that must be produced exists in no single frame.A single error propagates across hundreds of frames.Control demands covering each trajectory in its entirety.And the billing unit adopted decides the very meaning of the quotation.One important observation follows for a bounding box project. The second trait alters the whole balance of control, a partial rereading being able to suffice on still images whereas here it lets entirely false trajectories pass, which makes random sampling markedly less effective than a control targeted on crossings.What revising an instruction imposes here
Four situations force reworking a tracking rule.A density of crossings far above what had been foreseen.A type of disappearance the initial instruction quite simply did not envisage.A persistent disagreement between operators as to reattributions.And a change of video source coming to modify the frame rate.One important observation follows for a bounding box project. The last situation is noticed late and distorts everything, a different frame rate making the sampling step initially adopted unsuited to the phenomenon observed, which forces reopening a parameter nobody thought would need revisiting.What the provider brings here
Four contributions distinguish a serious engagement in video.A written rule covering disappearances, crossings and shot cuts.A record of the number of identity swaps rather than a plain global rate.A rereading deliberately concentrated on the moments of crossing.And a delivery clearly separating drawn boxes from calculated ones.One important practical consequence follows. The second contribution tells the truth the global rate masks, a corpus showing excellent detection and yet a dozen identity swaps being unusable for any trajectory analysis, which makes that figure more revealing than all the others together.Approaching a video project
Five questions scope a bounding box project in video.Are identities needed. They carry the cost.What sampling step to adopt. It follows the phenomenon.What to do after a disappearance. The rule must be written.How to handle crossings. They swap identities.And does the measurement bear on the sequence. Otherwise it misleads.Those five answers determine the very consistency of the tracking. Asking them before production avoids a corpus whose trajectories end up mixed together.The question that frames the project
One question determines the scale of the work in video bounding box terms.Does the use need to know that it is the same object.An instant count or an event detection is satisfied by independent boxes and ignores identities completely.A count of passages, a duration measurement or a trajectory analysis demands constant identities and at least doubles the cost.That question belongs to what the use will have to calculate, it is asked before the first sequence, and it separates two projects whose name alone is identical.Three decisions before producing
Three decisions commit a bounding box project in video.Deciding whether identities are truly needed, they carry most of the cost.Fixing the sampling step from the duration of the phenomenon observed.And writing down plainly the rule applying after a temporary disappearance.Those three decisions cost only one meeting, they precede the very first sequence, and their absence produces trajectories nobody will be able to untangle.Three checks on a video corpus
Three controls qualify bounding box tracking already produced.The written reattribution rule applying after a disappearance.The number of identity swaps effectively recorded at the crossings.And the distinction made between drawn boxes and interpolated ones.Those three controls are each asked in one single question, they require no particular technical competence, and their absence indicates a corpus whose trajectories have never been verified as such.Three questions before quoting
Three questions precede any estimate on a video.Are identities truly necessary to the use finally foreseen.What is the frequency of crossings observed on a typical sequence.And will the rate bear on the minute, the frame or the trajectory.Those three questions are asked well before the quotation, they call for a sample sequence rather than a description, and their absence produces a costing unrelated to the real work.What this chapter teaches
One cross-cutting observation deserves closing this examination.An identity is information no single frame holds.Three findings compose it.Two similar objects that cross can swap identities with no signal.An uncertain operator chooses between a false identity and an invented one.And an event shorter than the sampling step passes unnoticed.That finding extends the preceding chapters, the object to describe lying no longer in the frame but between the frames.What this chapter shares with the previous ones
Three findings recur from the start of this course.A global measurement can stay excellent while the deliverable is unusable.A corpus that does not test the difficulty reassures wrongly.And automatic help reinforces what already worked.One important practical consequence follows for a bounding box project. The first finding takes its most striking form here, tracking showing an excellent detection rate and yet entirely mixed identities being of no use at all, which shows that a correct figure never guarantees that the right thing has been measured.What this chapter leaves to the next
Everything above translates at some point into a line on a quotation.Three questions stay open in bounding box work.What actually composes the price of one annotated object.Why two comparable quotations cover different scopes.And how to estimate a volume before having started.Those three questions belong to the cost of annotation, which constitutes the subject of the following chapter.Why the billing unit deserves the argument
One point about how these projects get priced belongs at the close.Clients almost always ask for a rate per minute of video.Three reasons follow in bounding box work.Minutes are what they have and what they can count.A rate per minute makes two quotations look comparable.And a provider proposing another unit appears to be complicating things.One important practical consequence follows for a bounding box provider. Arguing for a rate per trajectory anyway is worth the friction, since a minute of video says nothing about how many objects cross within it, and a provider who accepts the per-minute unit has agreed to absorb a variation they cannot see and the client cannot describe.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Annotating a video as a series of independent images.
- Omitting the reattribution rule after a disappearance.
- Neglecting the handling of crossings between similar objects.
- Interpolating over abrupt motion without closer key frames.
- Choosing the sampling step from the budget alone.
- Measuring quality frame by frame without judging the sequence.
- Letting the operator settle a doubtful identity alone.
- Ignoring shot cuts in the handling of identities.
- Confusing a new identity with an object found again.
- Neglecting the box’s stability on an object that is not moving.
