Measuring the quality of weed annotation poses a particular difficulty: the reference itself belongs to botanical judgement. Two competent annotators can diverge with neither being wrong, and that divergence bounds everything a model can reach.This article sets out how to measure, where to place the agronomist and how to frame a verifiable requirement. It extends the article on field conditions.
Why measurement differs here
Four properties distinguish weed detection from an ordinary detection task.Ground truth is not in the image, identifying a species belonging to outside knowledge the annotator applies.Disagreement between observers is structural rather than accidental, part of the plants being genuinely undecidable at the stage concerned.The classes are strongly imbalanced, target weeds representing a fraction of objects against the crop.And the cost of an error varies by class and by use, which the articles on the uses established.One important practical consequence follows. Those four properties render the most natural indicator inoperative, a global rate of correctly annotated objects, a high value of which is compatible with complete failure on what matters.The useful metrics
Five measures genuinely inform about a weed detection corpus.Agreement between annotators per class, which locates badly defined boundaries and supplies the achievable performance bound.Recall and precision per class, reported separately, a class well detected but over-assigned calling for the opposite correction to a class rarely detected.The confusion matrix, more informative than any derived indicator since it shows which classes a species drifts towards.The rate of undetermined objects, which measures the genuinely undecidable share and an abnormal value of which signals an insufficient reference.And localisation quality, distinct from classification quality, an object well classified and badly placed constituting a failure mode of its own.The agronomist’s exact role
A frequent confusion makes their involvement unrealistic.It deserves clearing.The agronomist does not annotate, throughput and volume making that involvement impracticable and its cost prohibitive.Four contributions justify their presence.Establishing the initial reference on a real batch, with observable criteria and illustrated confusions.Arbitrating the batches of undetermined cases annotators accumulate, a periodic and low-volume operation.Validating the evaluation corpus, whose reference quality conditions every later measurement.And settling nomenclature changes, mergers and abandonments notably.One important observation follows for a weed detection project. Those four contributions are counted in days spread across the project’s duration rather than in continuous presence, which makes them organisable where a demand for a full-time agronomist would block the project.Measuring disagreement between annotators
This operation supplies the bound of any weed detection performance.It is conducted in a day.The principle is to have one sample annotated by two trained people and to compare object by object.Four distinct pieces of information follow.The global agreement rate, which situates the general level and masks the disparities.Agreement per class, which identifies the problem classes and directs the reference effort.The pairs of classes confused, which indicate which precise boundary is involved.And the list of objects on which the disagreement falls, the raw material of the case library.One important practical consequence follows. That last piece is the least exploited and the most valuable: each disagreement points at a case the convention does not cover, which turns a quality measurement into an inventory of gaps.Agreement is not correctness
A fundamental distinction deserves stating because it is often conflated.Two annotators can agree perfectly and be wrong together.Three situations produce that configuration.A convention that is wrong but clear, which both apply faithfully and which does not match botanical reality.A shared training having transmitted the same identification error, which the agreement measurement structurally cannot reveal.And an automatic pre-annotation accepted by both, a confirmation bias the article on targeted spraying flagged.One important practical consequence follows for a weed detection project. Only an external reference establishes correctness, an agronomist, laboratory determination or field tracking, and an arrangement resting solely on agreement between annotators measures consistency without ever measuring exactness.The correction cycle
A loop organises improvement.It is better than a final check.Four stages compose it, and they repeat between batches.Double annotation of a sample of the current batch.Identification of disagreements and their grouping by type.Arbitration by the agronomist of the recurring types, with the rule written down.And circulation of the rule to annotators before the next batch.One observation follows. This cycle reduces disagreement from batch to batch down to a floor corresponding to real undecidability, and reaching that floor is the sign a reference is complete rather than an objective to be exceeded.Building the control sample
Four principles govern its composition and they differ from those of a random sample.Stratification by class is necessary, a random draw containing almost no objects of the minority classes that carry the issue.Over-representing difficult configurations is deliberate, the objective being to measure where the corpus is tested rather than to reflect its distribution.Coverage of the conditions must feature, a sample acquired under a single condition telling nothing about the variability the previous article set out.And part of the sample must stay constant between batches, the only way to measure drift over time.One practical consequence follows for a weed detection project. Those four principles lead to a composite rather than a single sample, a random part supplying the general estimate and a targeted part supplying the diagnosis, a distinction most arrangements neglect.What continuous control should track
Five indicators are read per batch in weed detection.Their tracking reveals drift.The proportion of each class, an unusual value signalling an atypical batch or a change of interpretation.The mean number of objects per image, a simple indicator that betrays incomplete annotation.The rate of undetermined objects, a sudden drop in which often signals that annotators have stopped using it rather than an improvement.The distribution of object sizes, a drift in which signals a change of convention on borderline plants.And agreement on a constant control sample, re-annotated periodically, which measures drift over time.One practical consequence follows for a weed detection project. The third indicator deserves particular attention, since its fall is spontaneously read as progress where it frequently reflects a slackening.The failure modes specific to this field
Six modes are distinguishable.They call for different corrections.Confusion between neighbouring species, a frequent error whose seriousness varies with whether the treatment differs.Confusion between crop and weed, a rarer and more serious error since it concerns the critical class.Omission of objects, a failure mode that rates computed on annotated objects mask since an absent object is not counted.False detection on residue or debris, whose origin is the acquisition condition more than competence.Delineation defects, a correct object whose box wrongly includes or excludes its surroundings.And inconsistency between batches, one configuration handled differently by period, which signals a convention defect rather than an individual error.One important observation follows. The third mode is the hardest to measure and the most damaging, since detecting it requires exhaustively re-annotating a sample, an effort most control arrangements do not include.Training as a quality arrangement
An upstream intervention produces more effect than any downstream control and it is regularly undersized.The quality of weed annotation depends more on what the annotator can recognise than on the rigour with which they are checked.Four elements compose an effective training.Recognising the crop first, the most frequent class whose mastery secures most of the volume.Progressive introduction of the target species, one at a time with their associated confusions.Training on edge cases, where the annotator mainly learns to recognise that they do not know.And a collective calibration session, where several annotators handle the same sample and compare results in the agronomist’s presence.One important observation follows. The fourth component produces the most effect per hour invested, since it reveals divergences of interpretation before they are written into thousands of objects, which makes it more profitable than the control it partly renders unnecessary.Framing a verifiable requirement
A three-step method produces a tenable requirement.It makes it verifiable.Establish what the error costs by class, a confusion between two weeds treated identically being inconsequential where an error on the crop is not.Translate that cost into a requirement per class and per condition rather than into a global rate.And verify that this requirement stays below the expert disagreement measured on those same classes.One important practical consequence follows. That third verification is what protects both parties: a requirement above the measured bound is unachievable, and demonstrating it with a figure rather than an opinion turns a negotiation into a finding.When to abandon a distinction
Three signals reveal it in weed detection.They indicate that a boundary should be abandoned rather than documented further.Disagreement stays high after two cycles of reference correction, which indicates the difficulty is in the data and not in the convention.The agronomist themselves hesitates on a significant share of the cases submitted, which places the bound below the target requirement.And the two classes concerned call for the same downstream treatment, which makes the distinction inconsequential for the use.One important observation follows. The third signal should be checked first, since it renders the other two moot, and it is checked with one question put to the end user rather than with two measurement cycles.What quality costs
Four lines grow with the requirement and their relationship is not linear.Annotation time, a species identification requiring several times that of a binary distinction.Double annotation, whose rate directly determines the surcharge and which can decrease as confidence establishes.The agronomist’s involvement, a distinct line whose hourly cost is higher and whose volume fortunately stays modest.And building the evaluation corpus, whose quality requirement exceeds that of the training corpus.One practical consequence follows. Halving a requirement divides the cost by more than half, which makes negotiating it particularly fruitful at scoping.One observation completes that for a weed detection provider. A requirement differentiated by each class’s importance produces a better result at lower cost than a uniform one, a conversation requiring the use to be understood rather than a standard applied.What measurement does not say
Four limits deserve stating.A measurement arrangement sometimes reassures wrongly.High agreement between two annotators trained by the same person measures the consistency of the training as much as the clarity of the reference.Agreement measured on an easy sample tells nothing about performance under difficult conditions.Annotation quality does not predict model performance, an impeccable corpus over too narrow a scope producing a disappointing system.And no internal measurement replaces field evaluation, which the article on targeted spraying established.That third limit is the most important commercially. It means a provider can deliver an irreproachable corpus and a client obtain a disappointing system, with neither party having failed, a situation prevented at scoping and settled badly afterwards.The quality report
Six elements compose a useful report and the absence of the last three makes a claim unverifiable.The control sampling method, draw plan and numbers.The complete confusion matrix between annotators.Recall and precision per class, reported separately.The disagreement measured between experts on difficult configurations.The rate of undetermined objects and its evolution between batches.And the list of configurations for which no stable convention could be established.One important observation follows. The last element is the rarest and the most useful, since it turns an invisible limit into a declared one, a principle every preceding series established and which finds its application here.The record of decisions
One document accompanies the reference and it is almost always absent.Every written rule results from a decision, and the reason for that decision is lost far faster than the rule itself.Three reasons justify recording it.A rule whose reason is known is revised knowingly, where an orphan rule is kept out of caution or dropped out of ignorance.Extending the corpus by another team presupposes understanding why a boundary was drawn there.And an audit requires demonstrating a method, which the list of rules alone does not do.One observation follows. This document costs one sentence per decision at the moment it is taken, and it cannot be reconstituted afterwards, a principle every preceding series established and which applies here to botanical conventions.Demonstrating quality to a client
Four demonstrations convince more reliably than a figure and they are prepared in advance.Visual comparison on a selection of difficult cases, with the decision adopted and its reason.The annotated confusion matrix, whose reading shows that residual confusions concern neighbouring classes rather than critical distinctions.Comparison against expert disagreement, which situates the performance obtained relative to what is achievable rather than to an ideal.And the case library itself, whose existence and richness demonstrate a method figures do not establish.One observation follows for a weed detection provider. The fourth demonstration is the most effective and the least used, since it shows the work rather than its result, and a client unfamiliar with metrics sees immediately what distinguishes it from a supplier with no method.The minimal arrangement
Five measures address most of the difficulties set out and their total cost stays modest.Measuring disagreement between two trained annotators, conducted once at project start and supplying the reference bound.Double annotation of a sample per batch, whose rate can decrease as confidence establishes.Tracking the five indicators per batch, a simple table revealing atypical batches and drift.Periodic arbitration of undetermined cases by the agronomist, with the resulting rule written down.And the quality report written during the work, with per-class matrices and the list of unstabilised configurations.One observation follows for a weed detection project. Those five measures take days on a project spanning months, they require no particular tooling, and they turn a quality claim into a verifiable fact.Approaching the quality of a weed corpus
Five questions scope this subject and they are asked before production.Which classes actually carry the issue. That answer concentrates control effort where it counts.Is agronomic competence available to arbitrate. Its absence caps achievable quality whatever the arrangement.Will disagreement between annotators be measured before commitment. Its value determines what can be promised.Is a constant control sample planned. Its absence makes drift undetectable.And is the requirement framed per class. A global requirement permits meeting the contract while delivering an unsuitable corpus.Those five answers determine the control arrangement and the contractual commitment. Asking them before production avoids the situation where an unrealistic requirement is accepted and then found impossible.What this chapter teaches
One cross-cutting observation deserves closing this examination because it matches a finding of the preceding series.The quality of a weed corpus is bounded by access to expertise rather than by the rigour of the arrangement.Three findings compose it.A perfect control arrangement with no agronomist measures the annotators’ consistency with one another and not their correctness.That correctness is verified only by a reference external to the image, an agronomist or field tracking.And that reference is expensive and scarce, which makes it the limiting factor rather than throughput or tooling.That finding matches exactly the one the medical imaging series established in a thematically unrelated domain: where ground truth belongs to expert judgement, corpus quality depends on access to that expertise, and organising it upfront is better than observing its absence in production.The uncomfortable number and what to do with it
One recurring moment deserves preparing, since it arrives in every project that measures properly.The disagreement figure comes back lower than anyone expected. Two trained annotators, given the same images and the same reference, agree less often than the contract assumes.Three reactions are possible and only one of them is useful.Suppressing it, which preserves the commitment and guarantees the problem resurfaces at delivery when it is no longer negotiable.Treating it as an annotator failing, which leads to more training on a difficulty that is in the data rather than in the people.And presenting it as a project finding, which reframes the requirement around what the data supports.The third reaction is uncomfortable for one conversation and protective for the rest of the engagement. It also carries an argument worth making explicitly: the figure describes what any provider would achieve on this data, not what this one achieves, and a competitor promising better is promising something the measurement rules out.One observation completes that for a weed detection provider. Measuring early is what makes this conversation possible at all, since the same finding produced after production reads as an excuse rather than as a specification.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Relying on a global rate of correctly annotated objects.
- Asking an agronomist to annotate rather than to establish and arbitrate.
- Not measuring disagreement between annotators before committing.
- Reading a fall in the undetermined rate as progress.
- Checking only at the end of production rather than between batches.
- Neglecting the measurement of omissions, which requires exhaustive re-annotation.
- Measuring agreement on an easy sample.
- Training every annotator through the same person and measuring their agreement.
- Omitting the constant control sample that reveals drift.
- Framing a requirement as a global rate rather than per class.
- Delivering a quality report without the list of unstabilised configurations.
