When AI boasts 99% accuracy, it often reveals a deep misunderstanding of what predictive maintenance models truly require.
If you graded someone’s ability to predict the outcome of field goal attempts and their prediction is “the attempt will be good but could also be a miss”, their accuracy would be 100% because all possible outcomes are covered. Similarly, predicting the result while the ball is already in flight through the end zone would also yield 100% accuracy. Rewatching the same game would confirm this perfect accuracy. But of course, these types of predictions are not useful. Numerous methods exist to manipulate accuracy metrics to appear favorable.
In predictive maintenance, companies claiming over 99% accuracy may base this on not missing faults. To achieve such high accuracy, the fault detection filter must be broad, which increases the number of false positives. Consequently, many predicted faults are either not actual faults or do not require action. This approach results in significant wasted effort for maintenance teams.
Any experienced vibration analyst knows that claims of 99.9% accuracy for a predictive maintenance tool is a misrepresentation of reality. This is because machine failure is complex when considering all the different machine types, applications, ages, configurations, operational conditions, environmental variables, maintenance histories, etc.
The comprehensive AI training needed to cover all these scenarios would likely take millions of monitoring points, a decade of data accumulation, and open access to operational data. In addition, the very definition of fault accuracy is difficult to pin down. Consider that most bearings have micro flaws even before they are leaving factory door. Similarly, at any given point in time, the lubrication of bearings is unlikely to be optimal for roughly half of the assets in a plant. But there is a threshold for what degree of a lubrication degradation warrants action. That threshold is highly subjective and asset and application dependent.
Predictive maintenance companies that claim extremely high accuracy often oversimplify complex scenarios. This practice erodes confidence and fosters distrust, as actual performance does not align with such claims. Therefore, very high-accuracy assertions should prompt skepticism regarding the credibility of the company claiming.
The Dunning-Kruger Effect may explain this behavior, as perceived confidence in a subject is roughly inversely proportional to actual mastery. The researchers demonstrated that “…people tend to hold overly optimistic and miscalibrated views about themselves.” They further proposed that “… those with limited knowledge in a domain suffer a dual burden: Not only do they reach mistaken conclusions and make regrettable errors, but their incompetence robs them of the ability to realize it.”