The choice of a video tool weighs more on a project than the choice of an image tool. Working throughput, the feasibility of tracking and the detectability of errors depend on functions not every tool supplies, and the gap is measured in factors rather than in percentages.
This article sets out what a tool must permit and how to arbitrate. It extends the article on the cost of video annotation.
What a video tool must supply
Six functions distinguish a video annotation tool from an adapted image tool. Fine temporal navigation, frame by frame in both directions, indispensable for settling a crossing or a boundary. Identity persistence, an object keeping its identifier from frame to frame with no manual reassignment. Interpolation between keyframes, with those frames placed and removed at will. Track visualisation, showing each object’s existence on a timeline. Looped playback of a segment, which permits judging a transition by comparison. And frame preloading, without which navigation becomes unusable on a long sequence. One important practical consequence follows. The absence of any of those six functions does not slow video annotation work down, it makes certain tasks impracticable, which makes this list disqualifying rather than indicative.What fluidity changes
This non-functional property dominates the real experience. It nonetheless appears in no comparison chart. An annotator navigates between frames thousands of times a day. Three concrete consequences follow. A one-second delay per navigation represents hours lost on a project, a cost invisible in a demonstration on a short sequence. Latency discourages verification, an annotator ceasing to go back when the operation costs too much. And fatigue sets in faster, which degrades quality towards the end of a session. One important observation follows. This property is tested only on a long and representative sequence, which makes a demonstration on a short clip uninformative in video annotation.The families of tool
Four tool categories occur in video annotation. Their economics differ markedly. Self-hosted open platforms, of which CVAT is the most widespread, which offer full control and presuppose an operations competence. Commercial online platforms, which remove the operations burden and place the data with a third party. Tools integrated into a service offering, where the tool accompanies an annotation engagement rather than being sold alone. And bespoke development, justified by a task nothing covers and rarely worthwhile otherwise. One practical consequence follows. The first and second categories oppose one another principally on the question of processing location, a legal and organisational trade-off rather than a functional one.What self-hosting brings
Four benefits justify this configuration. They are decisive in certain contexts. Control of the processing location, a disqualifying requirement where sequences contain identifiable people or sensitive industrial data. No transfer to a third party, which removes a risk surface and a clause to negotiate. Control of access, each consultation being logged on a controlled infrastructure. And independence from a supplier, a change of pricing or product direction not affecting a project under way. Three burdens accompany it. Operating the infrastructure, scaling and availability included. Updates and their validation, which are not delegated. And sizing, a video platform demanding fast storage and internal bandwidth.What CVAT brings and what it demands
One platform dominates open usage and its real profile is worth setting out. Four strengths characterise it in video annotation. Video functional coverage, interpolation, tracks and temporal navigation being native rather than bolted on. The diversity of geometries, boxes, polygons, points and cuboids coexisting in one project. Collective functions, assignment, validation and logs being built in. And export to numerous formats, which avoids a conversion development. Four requirements accompany it. Infrastructure sizing, a video platform consuming fast storage and memory. Operations competence, version upgrades and backups included. Directory integration for access management across a substantial team. And initial project configuration, whose structure conditions the consistency of the exports. One observation follows for a video annotation project. Those four requirements constitute a real entry cost and a low marginal cost, which makes this configuration economical above a certain volume and unsuited to an isolated short project.What the export format determines
Four decisions commit the exploitation of a video annotation corpus. They are verified before choosing a tool. The representation of identities, without which a tracking corpus is not usable. The distinction between observed and interpolated position, information few formats carry natively. The representation of variable attributes, an attribute changing during a track not storing like a constant one. And retention of the link to the source, timestamp and sequence reference. One important observation follows for a video annotation project. The second decision is the one whose absence produces the gravest defect, a corpus delivered without it not permitting a valid evaluation set to be built, which silently invalidates any comparison of models.The assistance functions
Three levels occur. Their interest depends on the video annotation project. Geometric interpolation, available everywhere and sufficient for most needs. Model-assisted tracking, which proposes tracks the annotator corrects and whose quality varies strongly with the content. And pre-annotation by the client’s own model, a rarer function presupposing an integration and producing the greatest gain as soon as a first model exists. One practical consequence follows. The third level is the most decisive on a durable project and the least present in comparison charts, which makes it a criterion to verify explicitly rather than assume.What matters for a team
Five functions concern organising a video annotation team rather than the work itself. Task assignment, which permits allocating sequences to annotators and tracking progress. The validation circuit, which distinguishes an annotated batch from a verified one. Version management, without which a correction overwrites the earlier state with no trace. Activity logs, which supply the real throughput and traceability. And rights management, which restricts access to sensitive sequences. One observation follows for a video annotation project. The fourth function is the most underestimated, the logs supplying the throughput data that feeds every later costing and that cannot be reconstituted afterwards.Sizing a video platform
Four resources condition a video annotation platform’s fluidity. Underestimating them produces most of the disappointments. Fast storage, fluid navigation presupposing access to frames far quicker than archive storage allows. Internal bandwidth, several annotators simultaneously loading sequences saturating an ordinary network. Memory, preloading a neighbourhood of frames consuming it in proportion to resolution. And compute, required only where assisted tracking functions are enabled. One important observation follows for a video annotation project. The first two resources determine the daily experience and they are frequently sized as for an image project, which produces a platform that works in a demonstration and is painful in production.Configuring a project
Five configuration decisions precede the first annotation and they correct badly afterwards. The structure of classes and attributes, a modification mid-project invalidating part of the batches already produced. The division into tasks, a sequence too long preventing parallel work and one too short fragmenting the tracks. The default interpolation regime, enabled or not according to the geometry adopted. Access rights, differentiated between annotation, validation and administration. And the target export format, whose early verification avoids discovering an incompatibility at delivery. One practical consequence follows for a video annotation project. The second point deserves particular attention in video, a division cutting a track in the middle producing two identities where the corpus expected one, a structural defect the tool does not flag.Evaluating a tool before committing
Five trials suffice. They take a day. Load a long and representative sequence, rather than a clip supplied by the vendor. Measure the navigation time between two distant frames. Annotate a complete track with an occlusion and verify that the identity holds. Export and read back the file produced, checking for the four pieces of information set out above. And have an annotator use the tool for a full session, rather than judging it on a demonstration. One important practical consequence follows. The fifth trial reveals difficulties no function list shows, and it costs a day where a bad choice costs a project.Three checks before signing
Three trials suffice to rule out most bad choices and they are conducted in a day. Load a sequence of several minutes from the real project and navigate end to end while measuring the delay. Annotate a track crossing an occlusion and verify that the identity holds with no manual intervention. Export and open the file, looking explicitly for the distinction between observed and computed position. Those three trials concern what makes a tool usable rather than what appears on a product sheet, and a tool that passes them suits the great majority of projects.What the video format imposes
Four technical constraints precede the choice of tool. The codec, some tools decoding certain formats badly and producing frame offsets. Variable frame rate, present in many files and a source of desynchronisation between the annotation and the source. Compression, whose artefacts degrade the legibility of small objects. And prior extraction into frames, which removes those three difficulties at the price of higher storage volume. One important observation follows. The last option is the most reliable and the most used in production, working on extracted frames guaranteeing an exact correspondence between the annotated number and the content, a correspondence on-the-fly decoding does not guarantee.Combining several tools
One arrangement occurs frequently and it deserves setting out. A complete chain combines several components rather than one tool. Four stages are clearly distinguishable. Preparation, frame extraction and anonymisation, done by scripts upstream of any platform. Annotation, on the chosen platform. Automatic control, run on the exports by scripts rather than in the tool. And conversion to the format the client expects, often a bespoke operation. One important practical consequence follows. The third component is the most profitable and the least often present natively, the temporal consistency checks set out in the preceding articles being programmed outside the tool and applying to every batch.What the tool must let you measure
Four pieces of data condition the steering of a video annotation project. They must be extractable. Time spent per task, the basis of any throughput measurement and any later costing. The number of objects and keyframes produced, which against the time gives the real productivity. The modification history, which distinguishes a correction from an initial production. And the rejection rate at validation, which measures upstream quality rather than final quality. One observation follows. The fourth is the most useful for steering and the least exploited, a rising rejection rate signalling a drift before the delivered corpus reveals it, which permits correcting during rather than after.When bespoke development is justified
Three situations make a development worthwhile and they are rarer than supposed. A geometry no tool offers, a particular domain representation reducing to no existing primitive. A strong integration constraint, annotation having to fit into an existing application flow rather than constitute a separate step. And a considerable recurring volume, where a throughput gain of a few percent amortises a development. Three signals indicate on the contrary that it is not justified. The stated need reduces to an existing function under another name. The motivation is dissatisfaction with a tool’s ergonomics rather than a functional impossibility. And the project is a one-off, a development amortising only over repetition. One observation follows for a video annotation project. The second signal is the most frequent, an ergonomic irritation appearing to justify a development where it is treated by a configuration change or by training.Training on the tool
Five competences distinguish a trained annotator and they are not acquired by use alone. Placing keyframes at the break points rather than at regular intervals. Correcting by adding a keyframe rather than by manually redoing a segment. Using the navigation shortcuts, which multiplies throughput on a task repeated thousands of times. Using looped playback to settle a boundary rather than judging on a single view. And flagging an uncovered case rather than taking an arbitrary decision. One practical consequence follows for a video annotation project. Those five competences are taught in half a day and they produce a throughput gap larger than the one separating two tools, which makes training more profitable than changing platform.Migrating from one tool to another
Four difficulties accompany a change mid-project and they justify choosing well at the outset. Converting the existing annotations, some of whose information has no equivalent in the target format. Loss of the histories, logs and versions generally not migrating. Retraining, shortcuts and habits acquired being tool-specific. And temporary coexistence, two tools producing exports to be harmonised during the transition. One observation follows. The first difficulty is the most constraining, an attribute or a track convention with no equivalent forcing either its loss or a bespoke conversion, which makes a migration more expensive than a better-informed initial choice.What a tool does not solve
Four difficulties persist whatever the video annotation tooling. The conventions, which no function replaces and whose absence produces an inconsistency the tool does not detect. Undecidable cases, a prolonged occlusion staying undecidable with the best tool. Annotator training, a powerful tool used without method producing fast and inconsistent work. And the choice of scope, rate and attributes belonging to scoping and not to tooling. That third difficulty deserves emphasis. It means a change of tool does not correct a quality problem whose origin is methodological, a frequent and expensive substitution.Approaching the choice of a tool
Five questions scope the choice of a video annotation tool. Can the data leave the client’s infrastructure. That answer settles between self-hosted and hosted. Is identity tracking part of the scope. A positive answer makes six functions disqualifying. Does a pre-annotation model exist or will one. That answer determines the integration need. What volume and what duration per sequence. That answer determines the performance requirements. And which team will work on it. That answer determines the need for collective functions. Those five answers reduce the choice to a few candidates. Asking them before comparing avoids a selection founded on functions no project will use.The question that settles the choice
One question narrows the field of candidates faster than any comparison chart. Can the sequences leave the client’s infrastructure. A negative answer rules out every hosted platform and requires a self-hosted configuration with the corresponding operations competence. A positive answer opens the field and moves the trade-off towards functions, cost and fluidity. That question belongs to law and to the client’s internal policy rather than to technique, and it is asked before any demonstration since it makes evaluating some candidates pointless.What this chapter teaches
One cross-cutting observation deserves closing this examination. The tool determines what is feasible and not what is good. Three findings compose it. The absence of certain functions makes tasks impracticable rather than slow, which makes verifying them a precondition and not a comfort criterion. Fluidity weighs more on real throughput than functional richness, and it is measured only on a long sequence. And no function replaces a convention, a perfect tool producing an inconsistent corpus if the rules were not written. That finding matches the one the preceding series established about assistance tools: they move the starting point without moving the limiting factor, which remains the documented definition of what is being annotated.Who should own the platform
One question comes up when a client engages a provider and it is worth settling explicitly. The annotation platform can belong to the client or to the provider, and the choice has consequences beyond convenience. Three arguments favour the client owning that platform. The data never leaves their infrastructure, which resolves the legal question at its source. The corpus and its history stay with them when the engagement ends. And a change of provider does not require a migration. Three arguments favour the provider owning it. The configuration is already tuned to the way their team works, which shows in throughput. The operations burden does not fall on a client with no reason to carry it. And the automatic checks are already built and applied as a matter of course. One practical consequence follows for a video annotation provider. A hybrid arrangement resolves most cases, the platform running on the client’s infrastructure and being configured and operated by the provider, which keeps the data in place and the competence where it sits.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Judging a tool on a demonstration with a short clip.
- Choosing without having checked the contents of the exported file.
- Neglecting the distinction between observed and interpolated position at export.
- Ignoring the processing location question on sequences containing people.
- Working on a variable frame rate file with no prior extraction.
- Selecting a tool on functions the project will not use.
- Omitting the activity logs, the only source of real throughput.
- Expecting a change of tool to correct a convention problem.
- Not having an annotator test the tool before committing.
- Undersizing the storage and bandwidth of a self-hosted platform.
