Measuring the quality of a point cloud classification poses a problem imagery does not encounter. There is no obvious reference object, the surface is not planar, and the errors that matter are not the ones the usual metrics penalise.
This article sets out how to evaluate an aerial LiDAR classification, which metrics to use and why global indicators mislead. It extends the article on annotation.
Why evaluation differs from imagery
Four differences structure this problem in aerial LiDAR. They invalidate a direct transfer of imagery methods. The unit of evaluation is a point rather than a pixel, which removes the regular grid the raster metrics rest on. Density is irregular, two areas of equal extent not containing the same number of points, which implicitly weights dense zones in any global indicator. The reference is expensive, manual verification requiring three-dimensional navigation rather than a simple glance. And errors do not carry equal weight, a misclassified ground point in the middle of a field having no consequence where the same point under a power line has considerable ones. One important consequence follows. A rate of correctly classified points, the most natural indicator, is also the least informative, and a high value is compatible with an unusable product.Why global indicators mislead
An arithmetic mechanism explains this inadequacy. It deserves setting out. Majority classes dominate the count. In a rural survey, ground and vegetation points make up most of the volume, which means a perfect ground classification and a complete failure on buildings produce an excellent global rate. Three effects follow directly. Minority classes vanish from the indicator, though they often carry what matters, conductors and structures notably. Difficult zones are underrepresented, most of the volume sitting in easy configurations. And improving a critical point registers as an imperceptible gain, which discourages effort where it counts. One observation follows for an aerial LiDAR project. This inadequacy reproduces exactly the one the boundary article set out for imagery, and it calls for the same response, per-class metrics targeted on difficult configurations.The useful aerial LiDAR metrics
Five measures genuinely inform about an aerial LiDAR classification. Their combination usefully replaces a single indicator. The per-class confusion matrix, distinguishing which classes are confused with which others, information no global rate supplies and which directs correction immediately. Recall and precision per class, separately, a class well detected but over-assigned and one rarely detected but reliable calling for opposite corrections. Rates measured on difficult configurations alone, deliberately sampled, informing about what the product is worth where it is tested. The discrepancy on derived products, the elevation difference between the terrain model produced and a reference, a metric expressing quality in the unit the user cares about. And the rate of unclassified points, a completeness indicator correctness metrics ignore.The question of the reference
This difficulty bounds everything above. It dominates evaluation in aerial LiDAR. Establishing a point’s true class requires human judgement, and that judgement carries a variability the boundary article established as the bound of any measurement. Three reference sources coexist and their qualities differ markedly. Re-annotation by an expert on a sample, the most reliable and most expensive method, whose representativeness depends on the sampling plan. Field measurements, elevations surveyed at identified points, an absolute reference in elevation and pointwise by nature. And external references, cadastre or building database, supplying a partial truth with the date and definition discrepancies the preceding series documented. One recommendation follows. Measuring disagreement between two experts on difficult configurations supplies the achievable performance bound, and that figure permits answering an unrealistic requirement with a fact rather than a negotiation.Sampling for evaluation
Four principles govern building an evaluation sample. Their omission produces a figure nothing supports. Stratification by class is necessary, a random sample of points containing practically no points of the minority classes. Stratification by configuration usefully adds to it, a sample deliberately containing difficult zones informing better than one representative of volume. The sampling unit must be chosen explicitly, sampling points and sampling zones not measuring the same thing and the zone unit generally being more informative. And geographic separation from training is necessary where a model is involved, two neighbouring zones sharing too much context. One important observation follows. A sample stratified on difficult configurations does not permit estimating global quality, which argues for a combination, a light random sample supplying the general estimate and a denser targeted one supplying the diagnosis.The failure modes to track separately
Six deserve distinguishing because they call for different corrections and an aggregate indicator reveals none of them. Confusion between neighbouring classes, ground taken for low vegetation or the reverse, a frequent error of moderate consequence. Confusion between distant classes, vegetation taken for building, a rarer and more serious error. Omission of whole objects, a building or a tree undetected, a failure mode per-point rates mask since a small object represents few points. False detection, an object created where none exists, generally through abusive grouping of noise points. Delineation defects, an object correctly identified whose limits wrongly include or exclude neighbouring points. And spatial inconsistency, one configuration classified differently according to the zone, a failure mode signalling an inadequate convention rather than an individual error. One practical consequence follows for an aerial LiDAR project. That last mode is the most informative to track, its presence indicating that no individual correction will resolve the problem and that the case library must be revisited.Checking through derived products
This method has the best return of all. It deserves systematising in any aerial LiDAR production. The principle is to generate a terrain model and examine it as a hillshade, a representation making classification defects immediately visible. Four characteristic signatures read in minutes. A bump signals an object wrongly classified as ground, a building or a vehicle most often. A hollow signals an absence of ground points, through occlusion or through an over-restrictive classification. Regular undulations signal a strip adjustment defect, a difficulty belonging to acquisition. And negative heights in the height model signal an inconsistency between classes, a physically impossible situation and therefore unambiguously diagnostic. One practical consequence follows for a project. This check costs minutes per batch, it requires no particular tooling, and its omission explains a substantial share of the defects that reach clients.Automatic consistency checks
Five verifications are programmed once. They then run across a whole batch without intervention. Ground points above points of another class at the same planimetric position, an impossible configuration. Isolated points floating far from any surface, generally unfiltered noise. Negative heights in the derived model, set out above. Objects of aberrant size, a building of a few points or a tree hundreds of metres across signalling a grouping error. And discontinuities at batch limits, an object classified differently either side of a boundary revealing an inconsistency between annotators. One observation follows. Those five verifications do not measure correctness but they detect the impossible, which makes them a preliminary filter whose marginal cost is nil after the initial development.Checking during production
Four arrangements distinguish continuous control from a final verification and they avoid discovering a problem on an already produced volume. Checking each annotator’s first batch, examined in full, which reveals convention misunderstandings before they repeat. Sample checking on each subsequent batch, whose rate can decrease as confidence establishes. Tracking simple indicators per batch, the proportion of each class and the unclassified rate, an unusual value signalling an atypical batch or a drift. And periodic calibration sessions, where several annotators handle the same sample and compare results. One observation follows for an aerial LiDAR project. The third arrangement is the cheapest and the most neglected, a simple table of proportions per batch revealing an anomaly nobody would have looked for.Expressing a quality requirement
An operational formulation is better than a rate. It is built in three steps. Establish what the error costs the use. A missing ground point in the middle of a field costs nothing; the same one under a power line can cost an outage. Translate that cost into a requirement on the classes and configurations that matter, rather than into a global rate across everything. And verify that this requirement exceeds the expert disagreement measured on those same configurations, failing which it is unachievable. One practical consequence follows. A requirement framed this way is verifiable, it protects both parties, and it avoids the situation where a provider meets the contractual rate while delivering a product unsuited to the use.The quality report
Six elements compose a useful report and their absence makes a quality claim unverifiable. The sampling method, draw plan and numbers by stratum, without which no figure carries meaning. The complete confusion matrix, more informative than any derived indicator. Recall and precision per class, reported separately. The results on difficult configurations, distinguished from global results. The measured expert disagreement, which situates the performance obtained relative to what is achievable. And the log of unclassifiable zones, distinguishing a known limit from a defect. One important observation follows. This report is a deliverable on the same footing as the classified cloud, it is written during the work at marginal cost, and it cannot be reconstituted afterwards, a principle every preceding series established and which finds here its direct application.Evaluating a classification received
One frequent situation deserves treating, a project often beginning by receiving a classification rather than producing one. Five verifications establish in half a day what the product is worth. Hillshading the terrain model over several contrasting sectors, which immediately reveals the state of the ground class. Counting the proportions per class, improbable values signalling a systematic problem, an absent building class in an urban area for instance. Examining a dozen deliberately chosen difficult configurations in section. Automatic search for the impossible configurations set out above. And comparing building footprints against an external reference where one exists. One commercial consequence follows for an aerial LiDAR provider. This evaluation is a short engagement, inexpensive for the client and often revealing, and it establishes on facts the scale of rework the client was underestimating.The case of successive surveys
One particular situation modifies the methods set out and it is becoming frequent with second national coverages. Comparing two surveys presupposes their classifications are comparable, a requirement distinct from individual quality. Four verifications are then added. Identity of conventions, a differing height threshold between campaigns producing an apparent change with no ground evolution having occurred. Consistency of vertical datums, whose discrepancy reads as generalised subsidence or uplift. Horizontal registration, a planimetric offset producing apparent differences on every slope. And comparability of densities and seasons, a denser or leaf-off survey classifying the ground better and producing a difference attributable to method. One practical consequence follows for an aerial LiDAR project. A difference map unaccompanied by a significance threshold leads to noise being read as change, and that threshold should derive from the measured quality of both classifications rather than from an arbitrary choice.What quality costs
Four lines grow with the requirement and their relationship is not linear. Annotation time, an exact object-by-object classification requiring several times that of a globally correct one. Checking, whose volume grows with the level of assurance demanded. Building the evaluation reference, a line distinct from production and whose unit cost is higher. And the number of classes, each additional distinction costing at annotation, at checking and in degraded per-class quality. One practical consequence follows. Halving a requirement divides the cost by more than half, which makes negotiating that requirement particularly fruitful at scoping. One observation completes that. A client frequently asks for a uniform requirement across all classes, out of caution, where a requirement differentiated by each class’s importance to their use produces a better result at lower cost, a conversation requiring the use to be understood rather than a standard applied.The minimal arrangement
Five measures address most of the difficulties set out and their total cost is modest. Checking by hillshading the terrain model, systematic before any delivery. Automatic consistency checks, developed once and applied to each batch. Tracking proportions per class, a simple table revealing atypical batches. Measuring disagreement between two experts on difficult configurations, conducted once at project start and supplying the reference bound. And the quality report written during the work, with per-class matrices and the log of unclassifiable zones. One observation follows for an aerial LiDAR project. Those five measures take days on a project spanning months, they require no particular tooling, and they turn a quality claim into a verifiable fact.Demonstrating quality to a client
Four demonstrations are better than a figure in aerial LiDAR. They are prepared in advance. Visual comparison before and after correction on a difficult sector, two hillshades side by side making the difference immediately perceptible. The annotated confusion matrix, whose reading shows that residual confusions concern neighbouring classes rather than critical distinctions. Comparison against expert disagreement, which situates the performance obtained relative to what is achievable rather than to an ideal. And the log of unclassifiable zones, whose presence demonstrates a methodological honesty few suppliers display. That fourth demonstration is counterintuitive and effective. Flagging what could not be done strengthens the credibility of what is claimed to have been done, where a deliverable with no declared limit invites doubt.Emerging regulatory requirements
One development transforms the place of these methods and it deserves following. The arrangements the preceding series described progressively require demonstrating a method rather than asserting a result. Three effects are emerging for an aerial LiDAR project. The quality report becomes required rather than recommended. Third-party auditability presupposes documentation whose after-the-fact reconstitution is impossible. And uncertainty characterisation ranks alongside the value announced. One commercial consequence follows. These requirements favour a provider already satisfying them and penalise one who must adapt, a redistribution the preceding series flagged as independent of technical quality.What this chapter teaches
One cross-cutting observation deserves closing this examination because it matches a finding the preceding clusters established. The difficulty of measuring quality in aerial LiDAR is not technical but definitional. Three findings compose it. The available metrics measure what is easy to count, the proportion of correctly attributed points, rather than what matters, the product’s fitness to support the decision it informs. The gap between the two widens with class imbalance, which makes it greatest precisely on the uses where the issue concentrates on a minority class. And the reference permitting measurement itself carries a variability, which bounds any measurement and makes quantifying it necessary before fixing a requirement. One consequence follows for a provider. Knowing how to frame and measure a quality requirement is a competence distinct from knowing how to produce, it is demonstrated through documents rather than assertions, and few actors display it, which makes it a verifiable differentiator.Agreeing acceptance criteria
One aerial LiDAR contractual question deserves setting out, since disputes at delivery almost always trace back to criteria that were never agreed. Four elements make acceptance criteria workable for both parties. The classes that matter, named explicitly, with the understanding that others are handled to a reasonable standard rather than to the same one. The configurations on which the criteria are measured, since a threshold met on easy terrain and one met on difficult terrain are different commitments. The verification method, including who samples, how and against what reference, so that neither party is surprised by the measurement. And the treatment of unclassifiable zones, agreed in advance as excluded from the criteria rather than counted as failures. One observation runs across those four. Each protects the provider as much as the client, since vague criteria expose a provider to an expectation that was never stated and a client to a delivery that meets the letter and misses the point. The practical rule follows. Agreeing these four before production costs one conversation; agreeing them after delivery costs a negotiation neither party wanted.Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.- Relying on a global rate of correctly classified points.
- Neglecting minority classes in the evaluation.
- Sampling randomly without stratifying by class.
- Evaluating on the easy configurations that dominate volume.
- Not measuring expert disagreement on difficult cases.
- Omitting the derived product check before delivery.
- Confusing correctness with completeness by ignoring unclassified points.
- Using an external reference as truth without measuring its discrepancy.
- Reporting recall without precision, or the reverse.
- Setting a requirement as a global rate rather than by class and configuration.
