Most vision projects that fail had a sufficient budget, a competent team and a performant model. What was missing sat elsewhere, and was visible in the first weeks to anyone who knew where to look.
This article sets out those causes. It extends the article on a project’s budget.
What missed scoping produces
Five initial defects condemn a computer vision project.
A need expressed with the expected decision never written down.
An acceptance threshold never fixed at the very start.
A task retained with no examination at all of the available data.
A cost of errors never costed by the business teams.
And deployment constraints ignored until the end.
One important practical consequence follows. The second defect makes the project unfinishable, a system being neither validatable nor abandonable for want of an agreed reference.
What a badly composed corpus brings
Five data weaknesses produce a late failure in computer vision.
Images carefully selected among the most successful.
A complete absence of every degraded condition.
Critical cases represented by a handful of examples.
A setting systematically tied to one single category.
And a test set drawn from exactly the same batch as the training.
One important observation follows for a computer vision project. The last weakness masks all the others, a flattering evaluation preventing the first four from being discovered before commissioning.
What the gap with the field reveals
Five differences separate the laboratory from real use.
Acquisition hardware different from the one that produced the corpus.
Variable lighting where the trials stayed perfectly stable.
A distribution of situations wholly other than the one expected.
A response time insufficient on the target hardware.
And outputs incompatible with the system meant to receive them.
One practical consequence follows. Those five differences are all discovered at the same moment, which turns commissioning into a general revelation rather than a formality.
What a successful demonstration masks
Five gaps separate a demonstration from a system in service.
The images shown were all chosen among the most favourable.
The hardware used differs markedly from that of the real site.
The volume handled stays without common measure with real use.
No real response time constraint applies at that point.
And nobody verifies what the system missed.
One important observation follows for a computer vision project. The last gap makes a demonstration almost always convincing, a system showing what it detects while never showing what it lets through.
What the absence of adoption signals
Five reasons explain why a delivered computer vision system never serves.
A rate of false alerts incompatible with the real pace of work.
An integration requiring a permanent detour in the habits.
An absence of visible benefit for those who use it.
A perception of the arrangement as a plain means of monitoring.
And an impossibility of correcting or else ignoring a proposal made.
One observation follows. The third reason decides more often than technical performance, a technically perfect system of no interest to its users being worked around within the very first weeks.
What silent drift produces
Five changes degrade a computer vision system with no alert at all.
A sensor replaced by a neighbouring but slightly different model.
Lighting modified by building work or else by the season.
Objects observed evolving little by little in their appearance.
An entirely new category appearing in the real flow.
And dirty optics nobody thinks of cleaning any more.
One important practical consequence follows for a computer vision project. Those five changes emit no signal, performance declining slowly with no indicator triggering as long as tracking has not been explicitly set up.
What the project’s organisation reveals
Five signals announce a difficulty before it appears.
No business contact genuinely available for the adjudications.
A project carried by the technology alone with no clear sponsor.
End users never consulted about the real need.
A technical team denied access to the site’s real conditions.
And a calendar fixed well before any examination of the data.
One important practical consequence follows for a computer vision project. The third signal explains a large share of adoption failures, an arrangement designed without those who will use it rarely answering the problem they actually meet.
What the moment of failure indicates
Five stages each reveal a different type of defect.
A failure at the very first batch signals a reference left ambiguous.
A failure at the moment of evaluation signals an insufficient corpus.
A failure during the port signals an ignored hardware constraint.
A failure at commissioning signals a gap with the field.
And a failure after a few months signals an untracked drift.
One important observation follows for a computer vision project. Those five moments permit tracing the cause with no long enquiry, the date of the failure designating fairly precisely the stage where the missing decision should have been taken.
Three errors that recur everywhere
Three computer vision defects appear in almost every failure.
A performance target set with no reference to the cost of errors.
A corpus built out of whatever happened to be available.
And a deployment approached as an end-of-project stage.
Those three defects are committed by experienced teams, they escape any technical review, and their common point is to bear on decisions nobody considers technical.
What the client can make fail alone
Five behaviours on the sponsor’s side suffice to condemn a project.
Sending a carefully chosen sample rather than the real data.
Staying durably unavailable to settle the ambiguous cases met.
Modifying the expected categories in mid production.
Refusing to cost the respective weight of the two error types.
And discovering the deployment constraints on delivery day.
One important practical consequence follows for a computer vision project. The second behaviour blocks more projects than all the others, one ambiguous case left unadjudicated for two weeks suspending the production of a whole batch.
What this chapter shares with the previous ones
Three findings recur from the opening of this course.
Scoping decides far more than the technical execution.
A system excellent in evaluation often fails in the field.
And the acceptable error threshold belongs to the business, not the technology.
One important observation follows for a computer vision project. This chapter merely gathers those findings in the form of their consequences, each failure described here matching one of the principles the preceding chapters stated without yet showing its price.
What this subject has that is particular
Four traits distinguish this question from the course’s others.
It bears on decisions nobody judges to be technical.
It is diagnosed after the fact with a misleading ease.
It produces explanations that protect everybody.
And it recurs from one project to the next for want of analysis.
One important practical consequence follows for a computer vision project. The third trait prevents collective learning, a convenient explanation avoiding conflict and leaving intact the practice that produced the failure.
What attributing the causes distorts
Five explanations recur after a failure and rarely name the cause.
The model finally used was not the right one.
The volume of data available stayed insufficient.
The technology itself was not yet mature.
The provider did not meet their commitments.
And the end users proved reluctant.
One important observation follows for a computer vision project. Those five explanations move responsibility towards the execution while the missing decision sat almost always at scoping, which guarantees the same failure will recur on the next project.
What the first weeks already reveal
Five early observations announce a project’s outcome.
The persistent difficulty in obtaining genuinely representative images.
The number of questions left with no answer at all from the business.
The gap observed between two people describing the same images.
The absence of a costed threshold despite repeated requests.
And the late discovery of a major deployment constraint.
One important observation follows for a computer vision project. Those five signals all appear before the end of the first month, which leaves ample time to rescope a project rather than pressing on to an announced failure.
What stopping a project permits
Five benefits follow from an owned decision to stop.
The means committed are freed for a better scoped project.
The corpus already produced stays entirely reusable elsewhere.
The lessons are documented while they stay fresh.
The business teams’ confidence is thereby preserved.
And the real need can then be reformulated calmly.
One important practical consequence follows for a computer vision project. The last benefit often justifies stopping, a project taken up again six months later on correct scoping arriving where stubbornly pressing on with the first would have led nowhere.
What a badly designed pilot produces
Five gaps make a pilot misleading rather than informative.
A site carefully chosen among the fleet’s most favourable.
A duration far too short to observe the real variations.
Close support the deployment will not benefit from.
Users who are volunteers as well as particularly motivated.
And a volume handled with no common measure at all with the intended use.
One important observation follows for a computer vision project. The third gap distorts the result most discreetly, a pilot followed daily by the technical team masking precisely the difficulties the deployment will make appear everywhere.
What this chapter presupposes of the provider
Five practices distinguish a partner from a plain executant.
Refusing to start without a clearly written acceptance threshold.
Flagging any deployment constraint the client has ignored.
Asking to meet all the arrangement’s end users.
Announcing a limit genuinely reached rather than prolonging the iterations.
And proposing to stop a project whose initial scoping condemns the outcome.
One important practical consequence follows. The last practice ends a running invoice and founds the next relationship, a client spared a pointless expense coming back with the projects where the expense is justified.
What a project’s success presupposes
Five conditions recur in the projects that succeed.
A business sponsor engaged as well as available from start to finish.
An acceptance threshold written and never moved afterwards.
A corpus built directly from the real conditions of use.
End users associated well before the first delivery.
And a tracking arrangement planned from the initial scoping phase.
One important observation follows for a computer vision project. Those five conditions all belong to organisation rather than technology, which explains why two equally competent teams obtain results with no common measure.
What the second project gains from the first
Five assets survive even an owned failure.
A reference written then tested on genuinely real cases.
A test set built directly from the field.
A precise knowledge of the site’s difficult conditions.
A business team trained in the questions to settle.
And a now reliable estimate of the real description time.
One important practical consequence follows for a computer vision project. Those five assets are lost only if nobody documents them at the moment of stopping, which makes that documentation more useful than the search for responsibilities.
What the commercial promise adds to the risk
Five gaps arise between what is sold and what will be delivered.
An announced rate measured on a public set with no connection at all.
A demonstration conducted on carefully chosen images.
A deadline proposed with no examination at all of the client’s data.
A scope described with no mention of quality control.
And a maintenance mentioned without ever being really costed.
One important observation follows for a computer vision project. Those five gaps create an expectation no correct execution can satisfy, which produces a perceived failure where the result obtained was in fact in line with what was possible.
What taking up an abandoned project requires
Five checks precede the relaunch of a stopped arrangement.
The written identification of the first failure’s real cause.
The verification that the already existing corpus stays usable.
The updating of the reference according to all the lessons drawn.
The rebuilding of a test set drawn from the current conditions.
And the commitment of a business sponsor genuinely available this time.
One important practical consequence follows for a computer vision project. The first check conditions all the others, a relaunch conducted without having named the initial cause reproducing the same failure with additional means.
What the data provider brings here
Four contributions reduce the risk before it materialises.
A careful examination of the available data before any commitment.
A flagging of every situation the corpus does not cover.
A test set built independently of the training corpus.
And a clear alert when the threshold aimed at seems out of reach.
One important practical consequence follows. The last contribution takes the most courage and avoids the most expenditure, a project stopped in time costing infinitely less than one carried through to a failure nobody dared announce.
Approaching the risk of failure
Five questions scope the risk of failure of a computer vision project.
Is the threshold aimed at written. Otherwise nothing concludes.
Does the corpus really come from the field. Often not.
Is the test set separated. It must be, first of all.
Do the users gain from it. Otherwise they will work around it.
And is the drift tracked. Nothing else signals it.
Those five answers determine a project’s solidity. Asking them at the start avoids a failure nobody will know how to explain.
The question that separates the failures
One question separates two categories of failure in computer vision.
Does the system work badly or does it work on the wrong problem.
Working badly, an insufficient corpus or an unsuited model, is corrected by additional data or extra technical work.
Bearing on the wrong problem, a badly defined decision or a misunderstood use, is corrected by no technical effort and requires redoing the scoping.
That question is settled by examining the errors observed, it is asked at the first signs of difficulty, and it avoids months devoted to improving what will not serve.
Three decisions to reduce the risk
Three decisions protect a computer vision project.
Writing the acceptance threshold before any development.
Building the test set from the real conditions of use.
And associating the future users from the scoping phase onwards.
Those three decisions cost a few days, they precede any production, and their absence explains most of the failures observed.
Three checks on a project in difficulty
Three controls direct a computer vision diagnosis.
The existence of an acceptance threshold written before the start.
The real provenance of the data in the test set used.
And the presence of the end users in the definition of the need.
Those three controls are each asked in one question, they require no technical competence, and their absence generally explains the difficulty met.
Three questions before a project that is stalling
Three questions direct a diagnosis in a few minutes.
Had the threshold aimed at been written before the start.
Do the errors bear on cases absent from the corpus.
And were the end users consulted at scoping.
Those three questions are asked with no technical competence, they name the cause in most situations, and their absence leads to a long search for a technical explanation to a problem that is not one.
What this chapter teaches
One cross-cutting observation deserves closing this examination.
Projects fail at scoping and discover it at deployment.
Three findings compose it.
A threshold never written makes the project impossible to conclude.
A badly separated test set ends up masking every other defect.
And a system with no benefit for its users will end up worked around.
That finding extends the preceding chapters, the difficulty rarely sitting where attention goes.
What this chapter leaves to the next
Avoiding these failures presupposes knowing what to work with.
Three questions stay open in computer vision.
Which tools exist and what they really permit.
What taking over an existing component commits to long term.
And where the boundary sits between free and expensive.
Those three questions belong to open source in vision, which constitutes the subject of the following chapter.
Why the technical explanation wins
One point about how failures get explained belongs at the close.
The post-mortem almost always names a technical cause.
Three reasons follow in computer vision.
A technical cause implicates a choice rather than a person.
Everyone in the room made the scoping decisions together.
And a model can be swapped where a way of working cannot.
One important practical consequence follows for a computer vision team. Naming the scoping decision that was missed is worth the discomfort it causes, since the technical explanation leaves the practice intact, and the next project will fail the same way with a different model and the same absent threshold.
What each side sees coming
Four early signals divide between the two parties.
The provider sees the reference failing to settle real cases.
The client sees the users’ indifference to the arrangement.
The provider sees the corpus missing whole conditions.
And the client sees the deployment constraints nobody asked about.
One important observation follows for a computer vision project. Each side sees signals the other cannot, which makes a short weekly exchange in the first month worth more than any formal review conducted once the production is well under way.
Common mistakes
These failures recur often enough that naming them is usually enough to avoid them.
- Launching a project with no acceptance threshold written at all.
- Building the corpus from the best available images alone.
- Drawing the test set from exactly the same batch as the training.
- Ignoring the material conditions of the planned deployment entirely.
- Neglecting the rate of false alerts genuinely acceptable in operation.
- Designing a system with no benefit at all for its users.
- Omitting any tracking arrangement once commissioning is done.
- Discovering the expected output format right at the project’s end.
- Treating commissioning as a plain formality.
- Attributing a failure to the model rather than to the initial scoping.
What to take away
Projects fail at scoping and discover it at deployment.
Three readings emerge. An unwritten acceptance threshold makes the project impossible to conclude, no result permitting it to be declared successful or abandoned. A badly separated test set masks every other defect, a flattering evaluation preventing timely discovery of what the corpus did not hold. And a system with no benefit for its users will be worked around whatever its performance, which places adoption on the same footing as the technology.
For the tools available, the article on open source in vision details it. For the recent models, the article on multimodal vision sets it out.
To explore delivery arrangements, supported formats and applicable control mechanisms, see our dedicated page on 2D and 3D annotation. And if you are preparing a vision project, let us discuss your need.