Annals of quantitative overconfidence, Boeing edition
On January 7, the auxiliary power unit (APU) of a Boeing 787 caught fire at Logan airport. The APU is a lithium-ion battery, roughly 1-foot cube, and the consequences of a fire can easily be catastrophic. There was no one on the plane at the time, which is lucky, because the fire was extremely difficult to extinguish, with firefighters encountering “no visibility” thanks to thick smoke. What’s more, the “quick-disconnect knob” was melted. In flight, these batteries control critical flight systems: they cannot fail.
And yet, twice in 58,000 hours of usage, the lithium batteries on the new 787 contrived to catch fire; this is obviously not something the FAA — or even Boeing, for that matter — can risk happening again. There’s really only one thing to be done: all lithium batteries on the 787 must be swapped out for nickel-cadmium or lead-acid batteries, which have the great advantage that they don’t catch fire.
The bigger story here, however, is about engineers’ hubris and regulatory capture. As the interim report from the National Transportation Safety Board says, the FAA was well aware, when Boeing said it wanted to use lithium batteries, that such batteries are inherently dangerous and have a tendency to catch fire whenever they are used elsewhere.
But Boeing persisted, and came up with some hilariously overprecise probability estimates. The batteries would only emit gas or smoke once every 10 million hours, the company calculated, and would only catch fire once every billion hours. The reasoning is bonkers: Boeing’s analysis “determined that overcharging was the only known failure mode that could result in cell venting with fire”. They then contrived to conclude that if they put in overcharge protections, the risk of overcharging would be brought down to one in a billion, and that therefore the risk of a fire would also be brought down to one in a billion.
As Steve LeVine notes, Nassim Taleb would take one look at that reasoning and simply laugh. For one thing, how on earth is it possible to determine that the risk of an overcharge is less than or equal to one in a billion? Probabilities that small simply can’t be measured. And more importantly, how did Boeing determine that the probability of a fire absent an overcharge was zero? There’s good evidence that neither of the battery fires were caused by an overcharge — but Boeing seems to have decided that fires caused for any non-overcharge reason were, literally, impossible. Once again, it’s incredibly hard to conceive of any coherent line of reasoning which could come to that conclusion.
But somehow the FAA accepted Boeing’s analysis at face value, and allowed Boeing to install lithium batteries on its planes, just as long as certain safeguards were put in place.
This is the same kind of literal quantitative thinking which helped cause the financial crisis. Put engineers in charge of something, and they’ll measure what they can measure, they won’t measure what they can’t measure, and they’ll protect against only the things they managed to foresee. And as all of us who spend our lives surrounded by electronic devices know, sometimes they fail. In a sense it doesn’t matter what the reason is: failure is just a fact of life, which is a real problem when failure could mean the fiery death of hundreds of people.
Statistically speaking, airplanes are safer today than they’ve ever been. And electronics are a key part of that trend: they might occasionally fail, but they are also increasingly good at preventing human error, or just at doing the things that fallible humans used to do, only much more reliably. That said, as airplane engineers stop being grease monkeys and start being coders, we’re losing a certain amount of holistic and heuristic understanding of how to ensure real-world safety.
If you basically outsource an entire airplane, as Boeing did, you lose your institutional ability to ensure that airplane is safe. And sadly, it seems that Boeing’s failures on that front will automatically cascade down to the FAA. The reports and post-mortems surrounding the lithium batteries’ safety will be very deep. Let’s hope the FAA is just as critical when it comes to its own decision to accept Boeing’s analysis at face value.