Urban surveillance combines a volume constraint no budget covers with a legal framework that bounds what is permitted. It is the vertical where the question of law precedes the question of method.
This article sets out what this context imposes. It extends the article on video annotation for sports analysis.
What distinguishes this context
Five properties characterise urban video annotation. The camera is fixed, which supplies a stable background and a constant geometry. The stream is continuous, one camera producing twenty-four hours of images a day indefinitely. The people filmed have not individually consented, which places this vertical under a strict legal regime. The events sought are rare, most of the stream containing nothing of interest. And conditions vary on a known cycle, day and night alternating predictably over one view. One important practical consequence follows. The first and the last property make this vertical more predictable than the others, one camera producing a finite number of configurations that can be covered methodically in video annotation.What the fixed camera supplies
This property simplifies several difficulties from the preceding articles. The background stays identical from frame to frame. Four direct exploitations follow. Background subtraction isolates moving objects with no learning at all, an operation supplying a free pre-annotation. Zones of interest are defined once and for all, entrance, passage and forbidden area being fixed in the image. Calibration happens once, the relation between the image and the ground not changing. And recurring configurations repeat, one person taking the same route producing comparable images. One important observation follows. The first exploitation is this vertical’s principal productivity lever, background subtraction drastically reducing the volume of frames containing anything to annotate.The volume problem
This constraint structures every urban video annotation project. A camera estate produces a volume unrelated to any annotation capacity. Four reduction strategies apply successively. Motion detection, which discards the periods where nothing happens. Zone filtering, which retains only the events occurring in the regions of interest. Selection by uncertainty, which retains the sequences where an existing model hesitates. And selection by coverage, which completes the corpus on the missing conditions. One practical consequence follows. Those four filters apply in cascade and reduce the volume by several orders of magnitude, which makes the problem tractable and shifts the issue towards designing the filtering rather than towards annotation capacity.Designing the upstream filtering
This operation weighs more on the result than the annotation method does and it deserves detailed treatment. Five decisions compose it in practice. The motion detection threshold, a value too low letting foliage and rain through and a value too high missing slow objects. The definition of the zones handled, which excludes neighbouring roadway or residential windows for legal as much as practical reasons. The minimum duration of a retained event, which eliminates fleeting triggers of no interest. The temporal margin kept before and after, without which an event arrives with no context. And the sampling rate of quiet periods, some of which must be kept so the corpus contains negative examples. One important observation follows. The last decision is the most often omitted, a corpus composed solely of filtered sequences containing no example of an ordinary situation, which produces a system unable to distinguish an event from a background whose variety it has never seen.The legal framework as a precondition
This constraint precedes any technical consideration. An image of an identifiable person constitutes personal data, whatever the intended use. Four legal requirements follow. The purpose must be defined before the processing and it bounds what may be annotated. Minimisation requires keeping only what the purpose demands. The retention period must be bounded, which affects building a durable corpus. And the place of processing carries obligations, a transfer outside the European area calling for a specific framework. One important observation follows for a video annotation project. Those four requirements are handled at scoping and not afterwards, an image processed with no legal basis not being regularised retrospectively.Anonymisation and its effects
This operation is frequently requested in urban video annotation. Its consequences deserve setting out. Blurring faces reduces identifiability and it modifies the data. Three consequences follow. Re-identification after occlusion becomes harder, the face being a continuity cue. Some tasks become impossible, gaze direction and expression disappearing. And partial anonymisation does not always suffice, silhouette, gait and context sometimes permitting identification. One important practical consequence follows. Those three effects require anonymising before annotation rather than after, failing which annotators work on identifying data and the delivered corpus does not match the one it was produced on.Objects and people
One nomenclature distinction deserves stating since it has legal as much as technical consequences. Four categories compose a typical urban nomenclature. People, the most sensitive category and often the most frequent. Vehicles, whose plate constitutes personal data even where the driver is not visible. Inanimate moving objects, luggage and parcels, whose annotation raises no identification question. And scene elements, furniture and infrastructure, which serve as a geometric reference rather than as a target. One practical consequence follows for a video annotation project. The second category is regularly treated as neutral when it is not, a legible plate making a vehicle identifying, which requires anonymising it on the same footing as a face where the purpose does not demand it.The tasks requested
Five families occur. Their legal sensitivity differs strongly. Counting people or vehicles, the least sensitive task since it requires no identity. Measuring flow and occupancy, which requires trajectory tracking with no named identification. Event detection, a fall, a gathering or an abandoned object, which concerns a situation rather than a person. Behaviour detection, which presupposes qualifying an action and raises questions beyond the law. And person recognition, strictly framed and forbidden in many contexts. One observation follows for a video annotation project. The first three families cover most of the real needs of urban management, and careful scoping frequently permits bringing a request back towards them rather than towards the last two.What the rarity of events imposes
This property distinguishes the vertical from all the others. A sought event occurs a few times a day in a continuous stream. Three important consequences follow. A corpus built by random sampling contains almost no positive examples. Examples must be sought in the archives from incident reports rather than collected as they come. And the class imbalance reaches a level that makes the usual metrics uninformative. One practical consequence follows. The second consequence presupposes access to archives and incident registers, a resource whose availability conditions feasibility and which should be verified before any commitment.The conditions to cover
Five dimensions characterise a fixed camera’s variability. Their coverage is more accessible than elsewhere in video annotation. The day and night cycle, whose effect on one view is considerable. Artificial lighting, which produces fixed cast shadows and saturated areas. Weather, rain and fog degrading the image recognisably. Footfall density, one view at midday and at midnight not presenting the same difficulties. And scene modifications, works or moved furniture invalidating the reference background. One important observation follows. The last dimension is the most neglected and the most disruptive, a background change invalidating both the subtraction and the defined zones of interest, which requires an arrangement to detect those changes.What resolution imposes
A physical constraint bounds the achievable tasks and it is often discovered late. A camera covering an entire square grants only a few dozen pixels to a distant person. Four consequences follow. Counting stays possible at low resolution, a silhouette sufficing to establish a presence. Tracking becomes difficult, appearance no longer carrying enough information to re-pair after an occlusion. Action recognition becomes impossible beyond a certain distance, the gesture not being resolved. And performance varies strongly with position in the image, one camera being usable in the foreground and unusable at the back. One important observation follows for a video annotation project. The last consequence justifies defining a usable zone within the image and declaring it, rather than annotating uniformly a field part of which does not carry the necessary information.Adapted quality control
Four video annotation checks exploit the fixed camera’s properties. Zone consistency, an object annotated outside the handled regions signalling a scope error. Trajectory plausibility, the fixed geometry permitting verification that a movement follows a possible path. Comparison between periods, one camera at the same hour on two days producing comparable statistics whose divergence signals an anomaly. And verification of day and night transitions, moments where performance drops and where control should concentrate. One practical consequence follows for a video annotation project. The third check is specific to this vertical and very economical, the stability of the view supplying a comparison reference no other context offers.Organising production
Four requirements distinguish an urban video annotation project operationally. Restricted access to the data, a limited number of authorised people working in a controlled environment. Access traceability, each consultation having to be attributable to a person and a moment. No local copy, images staying on a controlled infrastructure rather than being downloaded. And deletion at the end, proof of which constitutes an obligation and not a formality. One important practical consequence follows for a video annotation provider. Those four requirements belong to a certified organisation rather than to a contractual undertaking, and they constitute a selection criterion that in practice excludes providers without a suitable infrastructure.Tracking between cameras
One extension is frequently requested and it constitutes a distinct problem. Attaching a person observed by one camera to the same person seen by another belongs to re-identification. Three technical difficulties characterise it. Appearance changes between views, angle and lighting strongly modifying the perceived silhouette. The temporal gap between two observations makes pairing uncertain, several candidates being compatible. And the number of possible matches grows with the number of cameras, which multiplies pairing errors. A legal difficulty is added and it takes precedence. This processing constructs an individual path, which changes its nature in law and calls for a legal basis distinct from that of counting. One observation follows for a video annotation project. That legal difficulty must be examined before the technical one, an arrangement technically successful and legally inadmissible not being deployable.What this vertical costs
Four factors determine the load. The effectiveness of the upstream filtering, which determines the volume actually annotated and dominates every other factor. The density of people in the retained scenes. The tracking requirement, a count costing far less than a flow measurement. And handling anonymisation, a distinct line preceding annotation. One observation follows. The first factor overwhelms the others, well-designed filtering dividing the volume by several orders of magnitude, which makes its design the principal value-added line of an urban video annotation project.Uses outside security
Four applications belong to this vertical without concerning security and they carry lower legal sensitivity. Traffic management, counting vehicles and measuring queues at intersections. Urban planning, measuring footfall in a public space to assess its real use. Transport management, platform occupancy and interchange flows. And infrastructure maintenance, detecting damage and obstructing objects on the highway. One practical consequence follows for a video annotation provider. Those four applications are satisfied by a count or a flow measurement with no identification, which places them in the least constrained regime, and they represent a substantial share of local authorities’ real needs.What this context does not permit
Four limits deserve stating in urban video annotation. Person recognition stays strictly framed and often forbidden depending on the purpose. Qualifying intent rests on no established ground truth, which makes annotating suspicious behaviour methodologically fragile. Tracking between cameras is a distinct processing whose admissibility must be examined separately. And long retention of a corpus runs into legal durations, which limits building a durable asset. That second limit deserves emphasis. It means a project asking for abnormal behaviour to be annotated must first define normality observably, an exercise that frequently fails and is better conducted before committing.The first batch to handle
Four criteria compose a representative pilot batch on an urban camera. One day sequence and one night sequence on the same view, whose throughput gap reveals the principal difficulty factor. One period of heavy footfall, which tests the convention on groupings and mutual occlusions. One quiet period, which verifies that the upstream filtering does not remove the negative examples needed. And one sequence containing the sought event, extracted from the archives from a report rather than hoped for. One observation follows. The last criterion conditions the whole project, an inability to compose that fourth element indicating that positive examples are not accessible and that the project is not feasible as framed.Approaching an urban project
Five questions scope this type of video annotation project. The first precedes the others. What purpose is declared. That answer legally bounds what may be annotated. Does anonymisation happen before annotation. A negative answer exposes annotators to identifying data. What upstream filtering is planned. That answer determines the volume and therefore the cost. Are the incident archives accessible. Their absence makes composing positive examples impossible. And may processing take place outside the European area. That answer conditions how production is organised. Those five answers determine legal feasibility and volume. Asking them before starting avoids a project technically complete and legally unusable.The question that precedes all the others
One question determines whether an urban project is feasible and it does not concern the technique. What will the operator do with the information, person by person or globally. A global answer, a flow or an occupancy, places the project in the least constrained regime and makes full anonymisation possible upstream. An individual answer, a path or an identification, calls for a distinct legal basis to be established before any annotation. That question is asked in one sentence, it belongs to law before technique, and it avoids committing to work whose result cannot be used.What this vertical brings to the others
Three practices born here hold beyond the urban context. Upstream filtering designed as a line in its own right, applicable anywhere raw volume exceeds annotation capacity. Deliberate retention of negative examples, a precaution every filtered corpus requires and few projects apply. And comparison between periods as quality control, available as soon as an acquisition repeats under stable conditions. Those three practices presuppose no particular legal constraint, and their adoption improves a project whatever its domain.What this chapter teaches
One cross-cutting observation deserves closing this examination. This vertical inverts the usual order of questions. Three findings compose it. The legal purpose bounds the scope before technical feasibility comes into play, an unusual situation in video annotation. Upstream filtering weighs more on cost than the annotation method, which shifts the value towards designing the pipeline. And the stability of the view supplies a control arrangement by temporal comparison no other vertical offers. That finding matches the one the series on the agricultural regulatory framework established: an external constraint formalises practices that were already rational, and a provider having adopted them by method finds themselves compliant with no transition.The conversation to have about scope
One discussion recurs in this vertical and handling it well distinguishes a provider. A client frequently arrives asking for more than their purpose requires, not out of intent but because the broader request feels safer. Three reframings help. Ask what decision the output triggers, since a decision taken at the level of a place rarely needs data at the level of a person. Show that the narrower task is cheaper, faster and less exposed, which aligns the commercial and the legal argument instead of setting them against each other. And point out that a narrower scope can be widened later with a fresh basis, where an over-broad one cannot be narrowed after the data has been processed. One practical consequence follows for a video annotation provider. This conversation reduces the engagement and it removes the risk that makes urban projects stall, which is why raising it early tends to produce a signed contract faster than accepting the original scope would.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Starting annotation before the declared purpose is established.
- Anonymising after annotation rather than before.
- Building a corpus of rare events by random sampling.
- Neglecting the design of the upstream filtering, which dominates the cost.
- Ignoring background modifications that invalidate the defined zones.
- Annotating abnormal behaviour without having defined observable normality.
- Treating cross-camera tracking as an extension of single-camera tracking.
- Omitting day and night transitions from the control arrangement.
- Assuming blurring suffices to make a person non-identifiable.
- Building a durable corpus without checking the applicable retention periods.
