A video corpus is not built by accumulating hours of footage. What determines its value is the number of distinct situations it covers, and that quantity bears almost no relation to total duration.This article sets out the method. It extends the article on GDPR and video annotation.
Why duration misleads
Four properties distinguish a video annotation corpus from an image corpus.Redundancy is extreme, two consecutive frames carrying almost the same information.Diversity does not grow with duration, an hour on one scene contributing less than a minute across sixty different scenes.Storage volume, by contrast, grows in proportion to duration.And annotation cost follows the number of objects rather than the duration.One important practical consequence follows. Those four properties mean a corpus of several hundred hours can cover fewer situations than a corpus of a few well-chosen hours, which shifts a video annotation project’s issue from collection to selection.The dimensions to cover
Six axes define the coverage of a video annotation corpus.They multiply against one another.Lighting conditions, from full day to night by way of the transitions.Weather, whose effects differ by sensor.Scene configurations, density, layout and type of place.Object classes, including the rare ones.Behaviours and dynamics, a stationary object and a fast one not being the same problem.And technical conditions, equipment, rate and compression.One important observation follows. Those six axes multiply rather than add, which explains why a corpus that appears voluminous actually covers a fraction of the combinations, and makes counting contexts more informative than counting hours.Selecting rather than collecting
Four selection strategies apply in video annotation.Their effectiveness differs markedly.Random drawing, which reflects the real distribution and over-represents the mundane.Selection by event, which retains the sequences containing what matters.Selection by uncertainty, which retains what an existing model does not master.And selection by coverage, which completes the plan’s missing combinations.One practical consequence follows for a video annotation project. Those four strategies combine rather than replace one another, the first supplying a representative base and the other three enriching it on what counts.Dividing into sequences
This decision structures the corpus.It is rarely examined in video annotation.Continuous footage must be divided into workable units.Four criteria guide that division in video annotation.Duration, a sequence too long preventing parallel work and one too short fragmenting the tracks.Natural boundaries, shot or action changes supplying cut points with no consequence.Homogeneity of conditions, a sequence crossing a day-night transition mixing two contexts.And self-contained comprehension, a sequence having to hold enough context to be annotatable alone.One important observation follows. The second criterion is the most effective and the least applied, cutting at a shot change removing by construction the problem of tracks crossing a task boundary.Frame sampling
A second granularity decision accompanies the division into sequences.Not every frame of a retained sequence is necessarily kept.Three sampling regimes occur.Full retention, which preserves continuity and suits tracking.Regular sampling, which reduces the volume and breaks continuity beyond a certain step.And targeted extraction, which keeps the frames around an event and discards the rest.One important practical consequence follows for a video annotation project. The second regime is incompatible with identity tracking beyond a modest step, which requires choosing between volume reduction and preserved continuity rather than hoping for both.Separating training from evaluation
One rule governs that separation.Breaching it produces flattering and false figures.Two frames of the same sequence share too much context to sit on either side.Four levels of separation exist and their demands increase.By frame, unavailable in video since neighbouring frames are almost identical.By sequence, the acceptable minimum and still insufficient where several sequences come from one scene.By scene or by site, correct for most uses.And by campaign or by period, the most demanding, which measures what the system will do on new data.One important practical consequence follows. The first level is still practised and it produces a performance with no meaning, two neighbouring frames split between training and evaluation amounting to evaluating on training data.The metadata to record
Six pieces of information condition the corpus’s exploitation.They cost a few fields at capture.Date and time, which permit reconstructing the conditions.The site or the camera, indispensable to a correct separation.Acquisition parameters, equipment, rate, resolution and compression.The conditions observed, weather and lighting.The reference to the source sequence and the temporal position within it.And the treatment applied, anonymisation and extraction notably.One observation follows for a video annotation project. The second is the one whose absence is most damaging, with no site identifier it becomes impossible to guarantee a correct separation, which makes any announced performance unverifiable.Sizing
Four factors determine the volume needed, and they do not reduce to a duration.The number of contexts to cover, the product of the six axes adopted.The number of objects per class and per context, the real basis of sizing.The internal variability of each class, which determines how many examples are needed.And the difficulty of the task, a fine distinction requiring more examples than an obvious one.One practical consequence follows. The volume is reasoned in objects per class and per context rather than in hours, an hour of empty scene and a minute of dense scene not comparing, which makes counting on a sample indispensable to any serious sizing.The rare classes
This difficulty dominates building a video annotation corpus.It is handled upstream.A rare event is not met by random sampling.Three methods permit recovering them.Searching the archives from incident reports, which presupposes access to the registers.Querying an existing model, whose low-score detections locate candidates.And deliberate staging, practicable in some industrial contexts and impossible elsewhere.One important observation follows. The availability of those three methods conditions the project’s feasibility and it is verified before any commitment, the absence of all three meaning the positive examples will not be assembled.Progressive construction
An organisation in batches is better than a mass acquisition and it deserves defending.A corpus built in one block fixes conventions before they are tested.Four advantages accompany the progressive approach in video annotation.Conventions are corrected between two batches rather than after the whole.Throughput and cost are measured early and projected.A first model permits pre-annotating the following batches.And the gaps that model reveals direct the next batch’s content.One practical consequence follows for a video annotation project. The fourth advantage is the most important, it turns construction into a loop rather than a straight line and concentrates effort on what is actually missing.Exploiting an existing collection
One frequent situation deserves treating, a client often holding footage accumulated before any project.Those hours have real value and often less than is attributed to them.Four verifications establish what that footage permits.Context coverage, an exercise that almost always reveals a concentration on a few scenes.The presence of metadata, whose absence strongly limits exploitation.Acquisition homogeneity, a collection built over several years mixing incompatible equipment and settings.And representativeness, a collection built during demonstrations systematically presenting good conditions.One practical consequence follows for a video annotation project. The first verification produces the commonest finding, hundreds of hours from three cameras covering fewer contexts than one planned acquisition day across ten sites.The evaluation corpus, apart
One distinction deserves stating because these two corpora do not follow the same rules.The training corpus must cover what the system will meet, the evaluation corpus must measure what it is worth.Four differences follow in video.Evaluation requires a far smaller volume, a few sequences sufficing to measure.It requires exhaustive annotation, a reduced rate or an interpolation distorting the measurement.It must cover ordinary and difficult conditions separately, a single set measuring neither.And it must stay stable over time to permit comparing two versions.One important observation follows for a video annotation project. The second difference is specific to this field, an evaluation set containing interpolated positions measuring conformity to a computation rather than to reality, which silently invalidates any comparison.Organising an acquisition campaign
Five logistical decisions condition success and they are prepared before the outing.Scouting the sites, ensuring they cover the target contexts and that access is authorised.The calendar, set on the conditions to cover rather than on availability, with several visits to the same places.The acquisition protocol, height, angle, settings and rate, recorded so as to be reproducible.Metadata capture, organised to be filled in during acquisition and not reconstituted afterwards.And checking the sequences the same day, an unsuitable exposure or a recording defect being correctable while on site.One important practical consequence follows for a video annotation project. The last point is the most profitable, an outing lost for want of an immediate check imposing a return whose cost far exceeds that of an on-site verification.What not to do
Four practices degrade a video annotation corpus.They nonetheless appear reasonable.Discarding poor quality sequences, which will be present in production.Over-sampling an available scene rather than seeking new ones, which inflates volume without adding diversity.Cutting at fixed intervals with no regard for natural boundaries.And keeping only the annotated frames while deleting the source sequence, which precludes any re-exploitation.One observation follows. The second practice is the most frequent, an accessible scene easily producing hours of content whose marginal contribution falls rapidly.What storage imposes
Four specific constraints weigh on organising a video annotation corpus.The volume of the sources, several orders of magnitude above an image corpus of equivalent coverage.The choice between sequences and extracted frames, the second form multiplying the volume and guaranteeing exact correspondence of the numbering.Transfer, whose duration becomes a scheduling constraint rather than a technical detail.And retention of the sources after delivery, without which no re-exploitation is possible.One observation follows for a video annotation project. The last constraint conflicts with the retention periods applicable to personal data, which requires an explicit trade-off between the value of a re-exploitable corpus and a deletion obligation.The pilot batch as a test of the plan
Four verifications apply to the first batch and they concern the plan rather than the corpus.Object density per scene type, which replaces the volume estimate with a measurement.The feasibility of the division adopted, a sequence too long or too short being correctable at this stage.The real coverage of one outing, often lower than planned.And the completeness of the metadata, a field forgotten at acquisition not being recoverable.One observation follows for a video annotation project. The third verification produces the most useful finding, one acquisition day covering in practice fewer contexts than the plan assumed, which permits correcting the calendar before consuming it.The coverage table
A simple document steers this dimension and it fits on one page.It crosses the target contexts with what was actually acquired.Four columns compose that table.The context, expressed in observable categories rather than continuous values.The number of sequences acquired for that context.The number of annotated objects arising from them, more informative than the duration.And the performance measured on that context, filled in after the first evaluation.One practical consequence follows for a video annotation project. This table is kept from the first outing and it reveals the gaps while it is still possible to fill them, where a review at the end of the campaign reveals them once the budget is spent.What the documentation must contain
Six elements accompany a video annotation delivery. Their absence limits reuse.The sampling plan actually carried out.The table of counts per class and per context.The annotation conventions, with the rate and the occlusion policy.The separation criterion adopted between training and evaluation.The quality report.And the list of contexts not covered, which explicitly bounds the domain of use.One important practical consequence follows. The last element is the rarest and the most useful, declaring what the corpus does not cover turning an invisible limit into a known one.What the corpus will be worth afterwards
Four properties determine whether a video corpus behaves as an asset.Retention of the source sequences, an annotation separated from its video losing all value.Documentation of the conventions, rate and occlusion policy notably, without which it is not combinable with another.Completeness of the metadata, which permits selecting a subset by condition rather than reworking everything.And respect for the applicable retention periods, which conditions the right to keep it.One observation follows. The first three properties cost little at the time of building and cannot be added afterwards, the fourth belonging to a trade-off that must be settled at the same moment.Approaching the construction of a video corpus
Five questions scope this phase of a video annotation project. They are asked before any acquisition.How many distinct contexts does the scope contain. That product determines the coverage needed.Are the rare classes accessible. Their absence makes the project infeasible as framed.What separation criterion will be adopted. That answer conditions the validity of any measurement.Will the site identifier be recorded. Its absence precludes a correct separation.And will construction proceed in batches. A mass acquisition fixes untested conventions.Those five answers determine the volume and the validity of the result. Asking them before collecting avoids a corpus that is voluminous and poor in diversity.The question that frames the collection
One question replaces the debate about volume and it is asked in one sentence.How many distinct situations will the system meet in production.An answer in dozens of contexts leads to a campaign planned across several sites and several moments, the total duration mattering little.An answer limited to a few stable configurations leads to a modest corpus and to effort going into covering conditions rather than scenes.That question concerns the deployment and not the collection, it is asked of the end user rather than of the technical buyer, and it produces a plan where a discussion about hours produces a figure with no foundation.Three decisions before the first acquisition
Three decisions determine a corpus’s validity and they cannot be recovered.Establishing the list of contexts to cover, which replaces an objective in hours with an objective in diversity.Fixing the separation criterion between training and evaluation, scene or site rather than frame.And settling the metadata to record, site identifier first among them.Those three decisions cost one meeting, they precede the first outing, and their absence produces a corpus whose announced performance cannot be verified.What this chapter teaches
One cross-cutting observation deserves closing this examination.A video corpus’s value lies in its diversity and not in its quantity.Three findings compose it.Temporal redundancy makes duration unrelated to the information contained, a property with no equivalent in image annotation.Selection weighs more on value than production does, which shifts the issue towards design.And separating training from evaluation requires a stricter criterion than in images, two sequences from one scene sharing too much context.That finding matches the one the preceding series established in other domains: what determines a corpus’s value is what it deliberately covers, and that coverage is designed before acquisition rather than observed after.The conversation about hours
One exchange recurs at the start of video projects and handling it well changes the whole engagement.A client arrives with a quantity of footage and asks what it is worth.Three responses are possible in video annotation, and only one is useful.Quoting on the hours, which is the easiest answer and commits the provider to annotating material whose value nobody has established.Declaring the footage inadequate, which is often accurate and reads as a refusal.And offering to measure the coverage, which turns an unanswerable question into a short piece of work with a concrete output.One practical consequence follows for a video annotation provider. The third response costs a day, it produces the coverage table the client did not know they needed, and it reframes the engagement from processing footage to building a corpus.One observation completes that. A client who sees their three hundred hours resolve into eleven contexts understands the rest of this chapter without further explanation.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Sizing a corpus in hours rather than in contexts and objects.
- Over-sampling an accessible scene instead of seeking new ones.
- Separating training from evaluation by frame rather than by scene.
- Omitting the site identifier from the metadata.
- Cutting at fixed intervals with no regard for shot changes.
- Discarding degraded sequences that will be present in production.
- Deleting the source sequence after extracting the annotated frames.
- Awaiting rare classes instead of deliberately seeking them.
- Acquiring in bulk before testing the conventions.
- Delivering without the list of contexts not covered.
