Interpolation is the most powerful productivity lever in video annotation, and the most regularly misused. Its principle is simple: annotate some frames and let the intermediate positions be computed. Its conditions of validity are less so.
This article sets out when it works, when it fails and how to control it. It extends the article on object tracking.
The principle and its condition
This reasoning underpins the most widely used method in video annotation. It carries a strong assumption. Between two annotated frames, an object’s position is computed by assuming regular movement. Three consequences follow directly. The gain grows with the interval, one annotated frame in ten producing ten times less manual work. The error grows with departure from the assumed movement, an object accelerating or changing direction diverging from the computed trajectory. And validity depends on content rather than on duration, an interval acceptable in a slow scene not being so in a fast one. One important practical consequence follows. That third property forbids fixing a single interval for a video annotation project, the same value producing an exact result on one sequence and an unusable one on another.What breaks interpolation
Five events invalidate the computation. Each requires an annotated frame. A change of direction, a trajectory computed in a straight line missing a turn entirely. A change of speed, an acceleration or a stop displacing every intermediate position. A change of scale, an object approaching rapidly seeing its size vary non-linearly. The start and end of an occlusion, moments where the observable position ceases or resumes. And a shot change, a cut rendering any computation between the two frames meaningless. One important observation follows. Those five events define the frames that must be annotated, which shifts the question from the interval to the break points, a markedly more workable formulation.Placing the keyframes
Two strategies coexist in video annotation. They do not produce the same result. Fixed spacing annotates one frame in a constant number, a simple method to organise that wastes effort on regular segments and misses the breaks. Adaptive placement annotates at the break points and spaces out elsewhere, a method that produces a better result for a smaller volume and requires the annotator to identify those points. One practical consequence follows for a video annotation project. The second strategy presupposes a prior pass through the sequence to identify the breaks, a quick operation that conditions the whole gain and that fixed spacing avoids at the price of an inferior result.Identifying breaks in practice
Four methods identify the frames to annotate and their cost differs. Accelerated visual review, which spots changes of direction and cuts in a fraction of the real duration. Automatic shot change detection, a reliable and inexpensive operation whose result is quickly verified. Global motion analysis, which flags the segments where the scene changes rapidly without identifying the objects. And identification by interpolation error, which consists of computing first and then annotating where the result visibly diverges. One observation follows for a video annotation project. The fourth method inverts the usual order and is the most economical, a first coarse computation locating the problem zones better than a review conducted in advance.The computation modes
Three methods produce the intermediate positions. Their suitability depends on the movement observed. Linear interpolation assumes constant-speed movement in a straight line, an assumption sufficient over short intervals and wrong on curved trajectories. Curve interpolation smooths changes of direction, which suits continuous movement and introduces oscillation on angular trajectories. And interpolation guided by a tracking model uses the content of the intermediate frames rather than a geometric computation, a method that adapts to real movement and introduces errors of its own. One observation follows. The first method remains the most used and the most predictable, its failure being comprehensible and locatable, which counts for more than a higher precision whose errors would be harder to diagnose.What interpolation does not compute
Four elements escape the computation in video annotation. They require an explicit decision. The appearance and disappearance of an object, discrete events no interpolation produces. A change of class, a categorical attribute not interpolating. Boolean attributes, occlusion or truncation, whose intermediate value is meaningless. And identity, which propagates rather than being computed and whose propagation carries forward any error. One important practical consequence follows. The third point produces a frequent defect, an occlusion attribute filled at the keyframes being carried mechanically onto the intermediates while it changes in between, which makes that attribute unreliable in an interpolated corpus without specific checking.Measuring what interpolation costs in precision
A simple method quantifies that loss. It is conducted on a sample. The principle is to annotate a short sequence exhaustively and then compare with the interpolated result at various intervals. Three usable pieces of information follow. The mean position error per interval, which shows from what value precision becomes insufficient. The distribution of that error, an acceptable average potentially masking large errors on fast segments. And identification of the segments where the error explodes, which reveals the movement types requiring adaptive placement. One important observation follows for a video annotation project. This measurement takes a day and it replaces an arbitrary interval choice with a justified value, which makes it disproportionately profitable against its cost.The real gain
Four factors moderate the saving in video annotation. It falls below what the interval ratio suggests. Identifying the breaks, which requires a pass through the sequence not counted in the number of annotated frames. Verifying the interpolated result, without which the gain is paid in undetected defects. Correcting badly computed segments, which returns a share of the frames to manual handling. And handling the non-interpolable attributes, which stays with the annotator across the whole sequence where they are required. One practical consequence follows. The observed gain remains substantial and it sits markedly below the raw interval ratio, a gap that should feature in a costing rather than produce an untenable promise.The gains outside interpolation
Four levers reduce the load without touching precision and they apply before any computation is contemplated. Reducing the delivered rate, a decision belonging to the downstream use that divides the volume with no approximation introduced. Segmenting the sequences, excluding stretches with no object removing work rather than accelerating it. Simplifying the geometry, an enclosing box sufficing where a mask was requested out of habit. And reducing the number of attributes, each variable attribute multiplying the work across a track’s whole duration. One important observation follows for a video annotation project. Those four levers act on the scope rather than on the method, they degrade nothing, and they are systematically examined after interpolation when they should be examined before.Controlling an interpolated corpus
Four checks detect the defects specific to this video annotation method. Examining the frames at the middle of the intervals, positions furthest from an observation and therefore likeliest to be wrong. Detecting abnormally straight trajectories, the signature of interpolation over too long an interval. Checking size variations, a perfectly linear evolution signalling a computation rather than an observation. And verifying attributes on the intermediates, whose mechanical carry-forward is the commonest defect. One observation follows for a video annotation project. The first check suffices to detect most badly sized intervals, it automates by systematically sampling the middle of each interval, and it costs a fraction of a uniformly spread control.Control by systematic sampling
One control method exploits the very structure of an interpolated corpus and it is more economical than a random survey. The principle is to verify only the frames at the middle of the intervals. Three reasons justify it. Those positions are furthest from an observation and therefore likeliest to be wrong. Their error bounds that of the whole interval, a correct middle position implying the others are more so. And their selection is deterministic, which permits automating the extraction rather than drawing at random. One practical consequence follows. This control bears on a fraction of the corpus, it detects the badly sized intervals the rest of the corpus does not reveal, and it produces an error measurement usable for adjusting the interval on subsequent batches.When not to interpolate
Four situations make the method counterproductive. Erratic movement, where the number of breaks approaches the number of frames and where identifying them costs more than direct annotation. Deformable objects, whose outline changes independently of displacement and which a position computation does not follow. Mask segmentation, where interpolating an outline produces intermediate shapes that are often unacceptable. And already very low rates, where the interval between two annotated frames is such that the regularity assumption no longer holds. That third situation deserves emphasis. It means interpolation’s gain falls sharply as the annotation geometry grows more complex, and that a video segmentation project does not obtain the saving a bounding box project does.Interpolation in three dimensions
One extension deserves flagging because its conditions differ. Point cloud sequences annotated with three-dimensional boxes lend themselves to the same method. Three differences characterise it. A vehicle’s movement is more regular than a pedestrian’s, which makes the regularity assumption more solid and permits larger intervals. Orientation interpolates as well, an additional parameter whose computation requires particular attention in turns. And the sensor’s position moves, which requires distinguishing the object’s movement from the scene’s before any computation. One practical consequence follows. That third difference makes interpolation in an absolute frame markedly more reliable than in a sensor frame, a stationary object appearing to move in the second while staying fixed in the first.The relation to model-assisted tracking
These two methods resemble one another. It is better to distinguish them. Interpolation computes positions without looking at the intermediate frames, a geometric operation whose behaviour is entirely predictable. Assisted tracking analyses those frames to estimate the position, an operation that adapts to real movement and whose errors are less predictable. Three practical differences follow. Interpolation requires no model and works on any sequence. Assisted tracking produces a better result on complex movement and an unpredictable one on configurations it handles poorly. And their combination is possible, tracking producing a proposal the annotator corrects at the identified break points. One practical consequence follows. That combination is the most efficient working mode in video annotation, and it presupposes tooling that exposes both mechanisms rather than one.What the annotator must decide
Five decisions stay human whatever the degree of assistance and they define the competence expected. Placing the keyframes, an operation presupposing anticipation of where the computation will fail. Accepting or rejecting a computed segment, a judgement requiring a trajectory to be compared with what the frames show. Handling occlusion boundaries, moments where the object ceases to be observable and where computation becomes extrapolation. Targeted rather than global correction, a badly computed segment being fixed by adding a keyframe rather than by full manual annotation. And flagging the sequences where the method does not suit, information that goes to the project lead rather than translating into slower work. One important observation follows for a video annotation project. The fourth decision distinguishes a trained annotator from one who endures the tool, adding a keyframe in the right place correcting a whole segment where a manual redo would cost ten times more.What interpolation changes in the format
Three documentary requirements accompany an interpolated corpus. The distinction between observed and computed position, information most formats do not carry natively and which conditions any evaluation. The interval used, which permits a user to know what observation density they are dealing with. And the computation method adopted, without which a result is not reproducible. One important observation follows. The first requirement is the most often omitted and the most consequential, an evaluation set containing computed positions measuring conformity to an interpolation rather than to reality, which silently invalidates any comparison of models.The interval and the annotation rate
Two notions are regularly conflated and distinguishing them clarifies a trade-off. The annotation rate denotes the density of frames for which a position is delivered. The interpolation interval denotes the density of frames actually observed by an annotator. Three combinations occur. Reduced rate without interpolation, which delivers sparse positions and leaves the intervals empty. Full rate with interpolation, which delivers one position per frame of which some are computed. And reduced rate with interpolation, a combination that only makes sense where the delivered rate stays above the observed interval. One practical consequence follows for a video annotation project. The second combination is the commonest and the least well documented, a corpus appearing exhaustive while the real observation density is far lower, which misleads a user who does not hold the interval information.Negotiating the interval with a client
Four arguments make this discussion productive and they avoid a negotiation over an abstract figure. Show the error measurement on their own sequences, which replaces a debate of opinion with a finding. Distinguish scene types, a differentiated interval producing a better result for a smaller volume than a uniformly tightened one. Set out what the interval costs in precision rather than what it saves in volume, a framing that shifts the decision towards the use. And recall that the evaluation set must stay free of computed positions whatever value is adopted. One observation follows. The second argument is the most effective and the least used, a client readily accepting a differentiation that improves the result where they would refuse a uniform rise in annotated volume.A simple decision rule
One formulation condenses this chapter into a question asked sequence by sequence. How many frames separate two changes in an object’s behaviour. If the answer is several dozen, interpolation delivers most of its gain and the interval can be wide. If the answer is a few frames, identifying the breaks costs more than direct annotation and the method is not justified. And if the answer varies strongly between sequences, a differentiated interval is required rather than a single value. That question is answered by watching a few minutes of video, it precedes any tooling choice, and it avoids this field’s commonest error, applying a uniform interval to a heterogeneous corpus.Approaching interpolation in a project
Five questions scope this decision and they are asked before production. What annotation geometry. Mask segmentation benefits little from this method. What is the dynamics of the scenes. That answer determines whether fixed spacing suffices or adaptive placement is required. Has the tolerated error been measured. An interval value not justified by a measurement is arbitrary. Are variable attributes required. Their presence strongly reduces the expected gain. And will the evaluation corpus contain computed positions. A positive answer distorts every measurement. Those five answers determine the real gain and the validity of the result. Asking them before starting avoids the commonest situation, a corpus whose apparent saving is paid in unmeasured imprecision.What this chapter teaches
One cross-cutting observation deserves closing this examination. Interpolation trades precision for volume, and that trade should be measured rather than assumed. Three findings compose it. The gain is real and systematically overestimated, identification, verification and correction consuming part of the announced saving. The precision loss is measurable in a day and is rarely measured, which means most projects choose an interval without knowing what it costs. And a computed position present in an evaluation set invalidates the measurement, an invisible defect that distorts a model comparison with no signal indicating it. That finding matches the one the preceding series established about assistance tools: they accelerate the easy work and leave the hard work intact, which makes their contribution real and their effect on total cost more modest than announced.What the client is actually buying
One framing question deserves settling explicitly, since interpolation changes what a delivered corpus represents. A client commissioning annotated video usually believes they are buying observations. With interpolation, they are buying observations plus a model of what happened between them. Three consequences follow from stating that plainly. The interval becomes a specification rather than an internal production choice, since it determines how much of the delivery is observed. The precision measurement becomes part of the deliverable, since it is what turns an assumption into a quantified one. And the evaluation set becomes a separate line, since it must be exhaustively annotated where the training corpus need not be. One practical consequence follows for a video annotation provider. Presenting interpolation as a specification rather than as a shortcut removes an ambiguity that otherwise surfaces at delivery, and it makes the interval negotiable on its merits instead of appearing as a corner cut to protect a margin.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Fixing a single interval for every sequence in a project.
- Choosing that interval without having measured the error it produces.
- Not requiring annotation at trajectory break points.
- Mechanically carrying boolean attributes onto intermediate frames.
- Interpolating across a shot change.
- Applying interpolation to mask segmentation without checking the result.
- Promising a gain equal to the interval ratio.
- Checking uniformly rather than at the middle of the intervals.
- Not distinguishing observed from computed position in the delivered format.
- Building an evaluation set containing interpolated positions.
