Machine learning teams spend weeks refining model architectures, testing new neural network topologies, and tuning hyperparameters. Yet the most recent research in data-centric AI consistently points to a sobering conclusion: most computer vision failures originate not in the model architecture but in the training data itself, and more precisely in the mistakes that accumulate during image annotation.
Image annotation is not a mechanical task that can be safely handed off to interchangeable operators. It demands well-structured guidelines, disciplined quality control, and genuine domain expertise. When these conditions are not in place, errors settle silently into the dataset and their effects often only surface during model evaluation or, worse, after deployment in production.
A 2026 survey published in Artificial Intelligence Review on annotation errors in object detection datasets demonstrates that even the most widely used computer vision benchmarks contain meaningful proportions of incorrect labels. These errors, when left undetected, degrade performance metrics and produce unpredictable behavior in real-world conditions.
This article examines the ten mistakes most commonly observed in professional image annotation projects. For each one, it describes how the error degrades model performance and what can be done concretely to prevent it. The goal is to give ML engineers, project managers, and data leads the tools to catch these issues before they translate into costly retraining cycles and production delays.
Mistake 1 – Imprecise bounding boxes: the most widespread image annotation mistake
The bounding box is the most commonly used annotation geometry for object detection. Its apparent simplicity makes it paradoxically the leading source of error across image annotation projects. The ease of drawing a rectangle around an object creates a false sense of control, until you try to get multiple annotators to agree on the same rules.
An imprecise bounding box can take two opposite forms. Too tight, it excludes pixels that belong to the object: the model learns to detect a truncated representation of the object and becomes systematically blind to the parts that are consistently cut off. Too loose, it incorporates background pixels into the region of interest, blurring the boundary between the object and its environment. In both cases, the IoU (Intersection over Union) score measured during evaluation is degraded, and the model reproduces these inaccuracies at inference time.
Bounding box errors are particularly insidious because they remain visually plausible. A box that is five pixels too wide on a high-resolution image is almost imperceptible during a quick review pass, but can turn a correct detection into an incorrect one on an IoU threshold of 0.5. Across a dataset of tens of thousands of images, these errors accumulate and create a performance ceiling that resists any adjustment to the architecture or training configuration.
The right approach is to formalize explicit tightness rules in the image annotation guidelines, illustrated with comparative examples of boxes that are too loose, too tight, and correct for each main class. Collective annotation sessions at the start of a project, where annotators draw boxes on a shared set of images and compare their work with the project lead, are the most effective way to align practices before scaling up production.
Mistake 2 – An ambiguous class taxonomy in image annotation
Before any image annotation work begins, the class ontology must be defined with absolute precision. This is consistently the step that time-pressured teams shortcut, on the assumption that class names are self-explanatory.
When a class is defined only by a generic label (“defect,” “plant,” “anomaly”), each annotator interprets it according to their own frame of reference. One annotator will categorize a superficial scratch as a “minor defect,” another as a “major defect,” a third as “acceptable.” The Cohen’s kappa coefficient measured between these annotators will be low, and the resulting dataset will contain contradictory signals that the model cannot reconcile during training.
The effect on the model is twofold. Classification accuracy drops, especially on examples near the boundary between two classes. And the variance of predictions increases: the model’s behavior becomes unpredictable on images that differ slightly from training examples. In production, this instability manifests as unexplained fluctuations in results across different image batches.
For each class, the image annotation guidelines must include a positive definition (what falls inside the class), a negative definition (what is excluded), and a gallery of contrastive examples illustrating the most common edge cases. This taxonomy must be validated by the business stakeholder before production begins and updated under version control throughout the project.
Mistake 3 – Overlooking occluded or truncated objects
In any real-world scene, objects are partially hidden by others, cut off by image edges, or concealed by environmental elements. Occlusion and truncation situations represent a significant portion of the cases the model will encounter in production, and their handling must be explicitly defined in the image annotation guidelines.
If guidelines do not prescribe how to handle these situations, annotators adopt divergent behaviors. Some label every partially visible object, others only annotate those with more than half their area visible, still others ignore them entirely. The result is an inconsistent dataset where visually similar situations are sometimes labeled, sometimes ignored, depending on which annotator processed the image.
A model trained on this data develops pathological uncertainty around partial objects, producing false negatives in production at exactly the most critical moments: a pedestrian partially hidden by another vehicle, a surface defect partially out of frame, an organ partially visible at the edge of a medical image, a component incompletely captured during an inspection pass.
The right practice is to define a minimum visibility threshold, for example at least 20% or 30% of the object must be visible to warrant annotation, and to illustrate typical occlusion cases for each main class in the guidelines. This simple rule prevents hundreds of inconsistent annotations in a dataset of any significant size.
Beyond the visibility threshold, guidelines should also specify how to handle the bounding box or mask geometry for truncated objects. Should the annotation extend to the image boundary, or stop at the last visible pixel? This seemingly minor detail, left undefined, generates yet another layer of inconsistency across annotators and accumulates into a measurable degradation of detection performance on edge objects.
Mistake 4 – Unversioned image annotation guidelines
An image annotation project rarely wraps up in a few days. Over weeks of production, guidelines evolve naturally: the client clarifies an ambiguous rule, new classes emerge from observing the data, an edge case triggers a collective arbitration decision. If these changes are not formalized, versioned, and communicated to the entire team, the dataset becomes heterogeneous without anyone noticing.
The symptom appears during dataset analysis: images annotated in the first week of production show systematically different characteristics from those annotated in week five. The model trained on this dataset learns implicit rules tied to production timing rather than the actual properties of the objects being detected. Its performance then varies inexplicably depending on which subset of the data is used for training.
Managing image annotation guidelines must be treated like a software artifact: version numbering, a change log, formal notification for each update, and verification that batches already produced under an older version are recalibrated if the change applies retroactively. Simple shared documentation tools are sufficient to maintain this discipline on most projects.
Undocumented guideline drift is one of the most frequent causes of failed retraining cycles, precisely because its trace in the dataset is hard to isolate after the fact. Identifying that a quality problem stems from an untracked guideline change can take days of investigation, time that most project schedules cannot afford.
Mistake 5 – Ignoring class imbalance in image annotation
In most real-world use cases, the classes to be detected are not naturally balanced in the collected data. Surface defects are rare on a well-calibrated production line. Crop diseases affect only a fraction of the photographed plants. Emergency braking events are a small minority in a driving dataset. This statistical reality must be factored into project planning before image annotation begins.
If imbalance is not addressed, the model converges toward the naive strategy: systematically predicting the majority class, which maximizes accuracy on the dataset while delivering no operational value whatsoever. A model that predicts “no defect” for every image on a production line with a 2% defect rate will achieve 98% accuracy while being completely useless for the intended industrial application.
Class imbalance must be addressed at two distinct levels. At the image annotation level first, by ensuring the final dataset includes enough examples of each minority class, through targeted collection protocols if necessary: controlled capture campaigns under specific conditions, data augmentation, or controlled synthetic data generation. At the training level second, through loss weighting or oversampling techniques, but these algorithmic mechanisms can only partially compensate for a structural imbalance that is too pronounced in the original annotated data.
Mistake 6 – A homogeneous dataset: the silent mistake in image annotation
An image annotation dataset can be technically impeccable, with well-fitted geometries, clearly defined classes, and rigorous quality control, and still lead to a model that fails in production. The reason: the dataset faithfully represents the conditions under which the images were collected, without reflecting the variability of real-world operating conditions.
A personal protective equipment detection model trained exclusively on well-lit indoor images will fail on outdoor construction sites in winter conditions or low light. A weld defect detector trained on parts photographed from a single orientation will miss the same defects photographed from a different angle. An agricultural surveillance model trained on images captured at a single growth stage will not recognize the same pathologies at different stages.
This mistake is particularly treacherous because it does not appear in the metrics calculated on a test set drawn from the same homogeneous dataset. The model shows excellent evaluation performance and disappointing results as soon as it is deployed on real production data.
Image annotation must be accompanied by a systematic diversity analysis of the dataset before large-scale production begins. This analysis covers the distribution of lighting conditions, camera angles, resolutions, backgrounds, seasons, or time of day depending on the application. Identified gaps should trigger targeted supplementary collection, because no data augmentation technique can create from scratch the variability that does not already exist in the original annotated dataset.
Mistake 7 – Inaccurate segmentation masks
Semantic or instance segmentation requires significantly higher precision than bounding boxes. A polygon mask must follow the actual contour of the object, with a sufficient density of control points to faithfully reproduce curves, angles, and fine boundary details.
The most common segmentation errors in image annotation projects fall into three categories. Polygons drawn with too few control points, generating straight segments between vertices that incorrectly include or exclude pixels. Smoothed boundaries where the object presents sharp angles, and conversely, angular boundaries where the reality is curved. And variable precision depending on visual conditions: objects with strong contrast against their background are well-delineated, while objects on complex backgrounds or in low contrast are handled approximately.
The effect on the model depends directly on the application. For a medical model delineating a tumor or segmenting an organ to support surgical planning, an inaccuracy of a few pixels at the boundaries can have direct clinical consequences. For an industrial surface inspection application, the imprecision of masks degrades Dice and IoU metrics in proportion to the density of boundaries in the images, and the model learns to tolerate inaccuracies that it will reproduce at inference time.
Verifying segmentation masks cannot be limited to visual sampling. Geometric consistency metrics, such as contour regularity or detection of polygons with too few control points, can be calculated programmatically to identify suspicious annotations before they are integrated into the final dataset. Combining this automated check with targeted human review of flagged cases is the most efficient approach for large-scale datasets.
Mistake 8 – No independent quality control in image annotation
One of the fundamental principles of quality control in image annotation is the organizational separation between the production function and the verification function. When an annotator reviews their own work, cognitive biases lead to near-systematic validation of initial choices, even incorrect ones.
Independent quality control, carried out by a verifier who is different from the production annotator and belongs to a separate team, makes it possible to detect drift before it contaminates an entire batch. The intensity of this control must be proportional to the stakes of the project: a 25% sampling rate of annotations may be sufficient for standard applications, while high-stakes projects, particularly in medical imaging or autonomous driving, require exhaustive review of every annotation produced.
Integrating test images whose correct annotation is known in advance, inserted anonymously into production batches (commonly called “honeypots” or quality control images), allows for continuous and objective per-annotator quality monitoring. This simple mechanism provides a reliable indicator for identifying annotators who are drifting from the guidelines and triggering targeted corrective action before errors spread across the production dataset.
The absence of independent quality control is the most costly organizational mistake in image annotation projects run without formalized processes. Problems only emerge during model evaluation, at a stage where correcting the dataset means revisiting potentially thousands of images, recalibrating the entire annotation team, and losing weeks of production time.
Mistake 9 – Undefined edge cases in image annotation guidelines
Edge cases are by definition rare in a dataset, but they represent a disproportionate source of errors in image annotation projects. An edge case is a situation where the object to be labeled does not clearly match a single defined class, or where the most appropriate annotation geometry is not obvious at first glance.
When guidelines do not address these situations explicitly, each annotator resolves them in their own way, based on their experience and personal interpretation of the general rules. Inter-annotator agreement breaks down precisely on the most informative examples for the model: ambiguous cases at the boundary between two classes are often the ones that, when correctly annotated, allow the model to learn the finest distinctions in the data distribution.
Edge cases must be actively collected during the pilot phase of the project, before scaling up to full production. An initial batch of a few hundred images is annotated in parallel by multiple annotators; disagreements are identified, discussed collectively with the project lead and if necessary with the business stakeholder, and the arbitration decisions are documented in the guidelines. This process, sometimes called “dispute resolution” or “inter-annotator calibration,” is the most effective way to enrich image annotation guidelines with real-world situations before industrializing production.
A catalog of resolved and referenced edge cases, updated throughout the project, is one of the most valuable deliverables of a mature image annotation program. It reduces variance between annotators on atypical situations, accelerates onboarding of new team members, and provides a valuable foundation for future projects in similar domains.
Mistake 10 – Combining heterogeneous image annotation sources without harmonization
To speed up dataset construction, it is tempting to combine annotations from different sources: an internally produced batch, a batch from an external vendor, annotations pulled from a public or academic dataset, and labels automatically generated by a pre-labeling model. Each source applies its own conventions, interprets ambiguous classes in its own way, and tolerates its own deviations in annotation geometry.
Merging these sources without prior harmonization is equivalent to training the model on contradictory signals. The same type of object in the same visual context may be annotated in three different ways depending on the batch of origin, with different occlusion rules, different boundary definitions, different minimum visibility thresholds. The model learns these divergences as if they represented real variability in the data, which degrades convergence during training and produces instability in predictions at inference time.
Before merging image annotation sources, a harmonization audit is essential. It consists of sampling each source representatively, comparing the conventions applied on a common set of images, and re-annotating batches whose practices deviate from the retained standard. Annotations produced by a pre-labeling model deserve particular attention: they may appear visually consistent while systematically reproducing the biases and errors of the model used to generate them. No automatically generated annotation should be integrated into a reference dataset without exhaustive human validation of suspicious samples.
Preventing image annotation mistakes: the measurable impact on your models
The ten mistakes described in this article share a common characteristic: they are difficult to detect and expensive to fix once a model has been trained, but relatively straightforward to prevent when the right processes are put in place before production begins. The gains are immediately measurable: a dataset annotated according to precise guidelines, verified by an independent team, and covering the full diversity of real-world conditions produces more robust models that require fewer retraining iterations and behave more predictably in production.
Three fundamental conditions make it possible to avoid the majority of these mistakes. The first is the quality and precision of the image annotation guidelines: a comprehensive, illustrated, versioned, and accessible guide is the investment that delivers the best quality-to-cost ratio on a dataset. The second is the effective independence of quality control: the separation between production and verification must be organizational and traceable, not merely declared. The third is dataset diversity: annotating more images under the same acquisition conditions does not improve model robustness.
These conditions apply whether image annotation is conducted in-house or outsourced to a specialized provider. For teams embarking on their first large-scale project or looking to industrialize their approach, professional image annotation services offered by providers with certified quality processes provide a proven framework for avoiding these ten mistakes from the very first batch.
To go deeper into the fundamentals of the field, our complete guide to image annotation covers annotation geometry types, the full project lifecycle, and the criteria for choosing between in-house and outsourced approaches. To understand how the discipline is evolving in the context of foundation models and why high-quality annotated data will remain decisive even under strong automation pressure, see the article Image Annotation in the Age of Foundation Models: What Future?
If you would like to have your current annotation process assessed or structure your next project on solid foundations, contact our team to discuss your specific use case.
