Not all annotation errors are equal. Some dissolve into the volume and disappear, others lastingly shape the system’s behaviour. Knowing which is which decides how an always limited correction budget is spent.
This article sets out those effects. It extends the article on quality measurement.
What noise produces
Five defects are distributed at random across a bounding box corpus. A tightness now too wide, now far too narrow. An occasional hesitation in favour of a neighbouring class. An object forgotten in the middle of a very busy image. A box drawn very slightly askew. And a limiting case settled differently depending on the time of day. One important practical consequence follows. Those five defects partly offset each other at scale, their effects cancelling out when they follow no direction, which makes them markedly less damaging than their frequency suggests.What bias produces
Five bounding box defects follow a constant direction instead. A tightness systematically too wide across a whole class. A rare class always confused with the same neighbouring class. Masked objects omitted in a perfectly regular way. Small objects systematically ignored below a certain threshold. And a convention applied backwards by one entire operator. One important observation follows for a bounding box project. Those five defects never offset each other, the system learning the false rule as well as it would have learnt the correct one, which makes them far more expensive than noise ten times as frequent.What omission teaches
Four effects follow an object plainly present and yet not annotated. The system then learns that this object must not be detected. The learning bears on the exact appearance of the object omitted. Objects resembling it become less well detected. And the usual quality measurement signals nothing at all. One practical consequence follows. The second effect explains this defect’s gravity, an omission concentrated on difficult objects teaching precisely how to ignore the cases the use made important, which inverts the corpus’s purpose on the population that mattered most.What a false class produces
Four effects follow a classification error in bounding box work. Two neighbouring classes then become partly interchangeable. The boundary between the two blurs across the whole corpus. The system ends up hesitating where the reference already hesitated. And the least frequent class suffers far more from it. One observation follows. The last effect deserves attention, a confusion spread equally between two classes weighing much more heavily on the rare class than on the frequent one, which makes one same error the graver for touching a reduced population.What a box in excess produces
Four effects follow a box drawn with no corresponding object. The system learns to detect a plain patch of ordinary background. False detections then concentrate on that type of background. The defect is spotted on rereading, unlike omission. And it is corrected without having to re-examine whole images. One important practical consequence follows for a bounding box project. Those four effects make this defect markedly less grave than its counterpart, a box in excess staying visible and correctable at low cost, which justifies accepting a little over-annotation rather than risking an omission.What ranking corrections presupposes
Five criteria order the rework when the budget is finite. The systematic character of the defect rather than its frequency alone. The population touched, whether rare or common. The proximity of the defect to the final use foreseen. The possibility of correcting by a rule rather than case by case. And the unit cost of the correction currently envisaged. One important practical consequence follows for a bounding box project. The first criterion reverses the spontaneous hierarchy, a rare but systematic defect deserving correction before a frequent but random one, which leads to handling first what the quality report signals least.What the distribution of defects reveals
Four observations separate ordinary noise from a deeper problem. A defect spread equally across all operators signals the instruction. A defect concentrated on one single operator signals the training. A defect concentrated on one single class signals the reference. And a defect concentrated on one single batch signals the image source. One important observation follows for a bounding box project. Those four readings are made without any particular competence, the distribution of a defect alone designating its probable cause, which avoids searching at length for what a simple sort by operator or by class brings to light.What correction by rule permits
Four situations are corrected without reopening a single image. A constant coordinate offset tied to an earlier conversion. A whole class renamed or else merged with another. A size threshold applied afterwards to the whole corpus. And an attribute added by deduction from the existing data. One important practical consequence follows for a bounding box project. Those four situations cost a few hours where manual rework would cost weeks, which makes the question of correctability by rule more determining than a defect’s apparent scale.Three errors proper to correction
Three bounding box defects belong to the rework itself. A correction started on the most visible defect rather than the gravest. An entirely manual rework where a simple rule would have sufficed. And data corrected without the original instruction being corrected too. Those three defects consume a budget without improving the corpus as much as expected, they are observed on the next batch, and their common point is to treat the symptom without tracing back to what produced it.What the corpus really tolerates
Four factors determine the quantity of defects that can be borne. The total volume of the corpus, which comes to dilute the noise. The frequency of the class directly concerned by the defect. The proximity between the defect observed and the final decision. And the existence of other correct examples bearing on the same case. One important observation follows for a bounding box project. The last factor explains why a rare class forgives so little, an object seen a hundred times bearing a few false annotations whereas one seen ten times bears none, which makes the quality requirement inversely proportional to frequency.What the client must settle here
Four decisions belong to the client rather than to the provider. The classes whose correctness weighs most heavily on their use. The type of error they prefer to see remain in the corpus. The budget they devote to rework rather than to volume produced. And the precise moment at which they judge the corpus sufficient. One important practical consequence follows for a bounding box project. The second decision is rarely raised and directs everything, a client accepting a few boxes in excess obtaining a more complete corpus than one demanding zero false detections, which makes that choice more determining than the quality level announced.What pre-annotation adds as a defect
Four defects of its own accompany an automatic proposal. A tightness bias inherited from the tool rather than from the reference. An omission concentrated on the objects the tool does not detect. A far too quick validation of the plausible proposals. And a homogeneity that gives only the appearance of consistency. One important practical consequence follows for a bounding box project. Those four defects are all systematic by nature, a tool going wrong always in the same way where an operator varies, which places them at once in the most expensive category described by this chapter.What this subject has that is particular
Four traits distinguish this question from the course’s others. It arises once the work is done and the budget largely consumed. It sets a defect’s real gravity against its mere visibility. It leads to correcting what the report signals least. And it makes a hierarchy far more useful than additional effort. One important observation follows for a bounding box project. The third trait makes this chapter counter-intuitive, the natural approach consisting in handling first what appears at the top of the report, which regularly leads to spending on noise while a discreet bias goes on being learnt.What rework really costs
Four items compose the expenditure of a correction. Identifying the images genuinely concerned by the defect. The rework itself, whether manual or by rule. The control verifying that the correction has taken effect. And rewriting the reference on the precise point at issue. One important practical consequence follows for a bounding box project. The first item often surprises, finding the objects touched by a defect sometimes costing more than correcting them, which makes a well documented corpus markedly cheaper to rework than one delivered without traceability.What imbalance between classes produces
Four effects accompany a corpus where the classes are unequally represented. The most abundant class dominates what the system learns. The rare class comes to be detected with far less reliability. A confusion between the two is settled in favour of the more frequent. And the global rate stays excellent nonetheless despite that failure. One important observation follows for a bounding box project. The third effect worsens the initial imbalance, a system hesitating between two classes choosing the one it has seen most, which penalises the rare class twice over and makes enriching it more useful than any correction.What the order of batches changes
Four effects follow a defect’s position in the calendar. A defect present from the first batch touches the whole corpus. A defect appearing late concerns only a fraction of the data. A defect corrected early does not recur in the following batches. And a defect discovered at delivery affects everything produced. One important observation follows for a bounding box project. Those four effects make the control calendar a more powerful cost lever than its coverage, a first batch seriously examined sparing a rework proportional to all the rest of the corpus.What rework does not recover
Four situations resist any later correction. A convention unsuited to the use from the very outset. A size threshold having excluded objects that mattered. A class absent from the initial reference. And images the corpus has never covered. One important practical consequence follows for a bounding box project. Those four situations call for fresh production rather than rework, which makes them incomparably more expensive and confirms that scoping decides what no correction will be able to save.What the annotator must know here
Four kinds of knowledge help the operator limit costly defects. The difference between an isolated hesitation and a habit taken up. The particular weight of rare classes within the corpus. The gravity of an omission compared with that of a box in excess. And the value of flagging a doubt rather than always settling it the same way. One important observation follows for a bounding box project. The last kind avoids precisely the most expensive defect, an operator who settles systematically in the same direction turning an uncertainty into a bias, which shows that a flagged hesitation is worth more than a constant and false decision.What the provider brings here
Four contributions distinguish a well conducted bounding box rework. A sorting of defects by distribution before engaging any correction. An explicit distinction between noise and bias in the report. A correction by rule each time that is possible. And a rework of the reference conducted alongside that of the data. One important practical consequence follows. The last contribution avoids the most absurd expenditure, a rework conducted without correcting the original instruction producing exactly the same defect on the following batches, which means paying twice for a correction nobody made lasting.What this chapter presupposes of the provider
Four practices distinguish a useful rework from a cosmetic one. Flagging a bias even when it touches few objects. Proposing a correction by rule rather than invoicing volume. Rewriting the instruction at the same time as the data. And saying when a rework would serve no purpose. One important practical consequence follows. The last practice runs against the provider’s immediate interest, a rework invoiced on noise paying as well as a useful one, whereas only the second justifies coming back to them for the next corpus.Approaching a correction campaign
Five questions scope a rework on a bounding box project. Is the defect systematic. That changes everything. Does it touch a rare class. The relative weight rises. Is omission involved. It is the gravest defect. Would a rule correct the whole. Otherwise the cost explodes. And does the defect concern the use. Otherwise it can wait. Those five answers determine the use of the rework budget. Asking them before correcting avoids spending on what was offsetting itself.The question that frames the rework
One question determines the use of the correction budget in bounding box work. Does the defect follow a direction or is it distributed at random. A random distribution, a variable tightness or an occasional hesitation, dissolves into the volume and justifies rework only if it becomes massive. A constant direction, a class always confused or a population always omitted, is learnt in full and justifies correction even on a small number of objects. That question belongs to the defect’s structure and not to its volume, it is asked before engaging the rework, and it separates a useful expenditure from one of comfort.Three decisions before correcting
Three decisions commit a rework campaign in bounding box work. Separating systematic defects from purely random ones. Checking whether a rule would correct the whole rather than one by one. And handling omission before any other defect. Those three decisions cost half a day of analysis, they precede the rework, and their absence leads to spending on what was offsetting itself.Three checks before engaging a rework
Three controls direct a bounding box correction. The distribution of the defect by operator, by class as well as by batch. The existence of a rule capable of correcting the whole at once. And the clarity of the original instruction on the precise point at issue. Those three controls are conducted in a few hours, they require only a sort of the existing data, and their absence leads to reworking a job the same instruction will then reproduce identically.Three questions before an observed defect
Three questions precede any rework decision. Is the defect distributed at random or does it follow a direction. Does it touch an abundant class or a rare one. And was the original instruction clear on this point. Those three questions are asked in a few minutes, they require only a sort of the existing annotations, and their absence leads to correcting without knowing whether the correction will hold.What this chapter teaches
One cross-cutting observation deserves closing this examination. The direction of an error matters more than its frequency. Three findings compose it. Defects with no direction partly offset each other at scale. A systematic defect is learnt as well as a correct rule. And an omission concentrated on difficult cases inverts the corpus’s purpose. That finding extends the preceding chapter, measurement there becoming a decision tool rather than a plain observation.What this chapter shares with the previous ones
Three findings recur from the start of this course. The gravest defect is the one that cannot be seen. A reduced population bears far fewer errors than an abundant one. And correcting the data without correcting the instruction settles nothing. One important observation follows for a bounding box project. The first finding is confirmed one last time here, omission staying the most expensive defect and the only one no examination of the deliverable reveals, which completes the justification of the measurement arrangement described in the preceding chapter.What this chapter leaves to the next
Most of these defects trace back to a badly written instruction. Three questions stay open in bounding box work. What an instruction must hold to be applied everywhere in the same way. Why a visual example transmits better than a paragraph. And how to keep a reference alive throughout the project. Those three questions belong to writing instructions, which constitute the subject of the following chapter.Why the rework conversation is worth having early
One point about when this discussion happens belongs at the close. Rework is usually discussed once a client is already unhappy. Three reasons follow in bounding box work. Nobody raises correction budgets while the work still looks fine. The first defect noticed is the one that prompts the conversation. And by then the tone is defensive rather than analytical. One important practical consequence follows for a bounding box provider. Raising the hierarchy of defects before any is found changes what happens later, since a client who agreed in advance that systematic beats frequent will accept a rework plan that ignores the defect they happened to spot, and one who did not will read that same plan as an excuse.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Correcting the most frequent defects first.
- Treating a systematic bias as ordinary noise.
- Neglecting omission because no indicator signals it.
- Ignoring the relative weight of an error on a rare class.
- Correcting case by case what a rule would settle.
- Reworking gaps the threshold made acceptable.
- Confusing a defect’s gravity with its visibility.
- Correcting without checking that the original instruction was clear.
- Spending a rework budget with no hierarchy established.
- Assuming a corrected defect will not reappear.
