How Small Mistakes Build Into Disasters — and How Small Steps Can Prevent Them

"Over the past few decades, our understanding of complexity, organizational behavior, and cognitive psychology has opened a window onto how small mistakes build, step by step, into major disasters. We now understand not only how these accidents happen, but also how small steps can prevent them."

About the Authors:

Chris Clearfield studied physics and biology at Harvard University and spent years trading derivatives; he is also a licensed commercial pilot. He has written widely about complexity and failure for The Guardian, Forbes, and the Harvard Kennedy School Review.

András Tilcsik holds a chair in strategy, organizations, and society at the University of Toronto's Rotman School of Management. He has been named one of the world's top 40 business professors under 40. The United Nations rated his course on organizational failure the best course on disaster-risk management offered at any business school.

Book Excerpt:

Introduction: An Ordinary Day (Excerpt)

I

On a warm Monday in June, just before the evening rush, Ann Wherley and David Wherley boarded the first car of Washington Metro Train 112. They had just come from a hospital volunteer training session and were heading home. A young woman near the front of the car stood up and offered her seat, and the Wherleys sat down together. They had been inseparable like this since high school. David, 62, had recently retired, and the couple was looking forward to their 40th wedding anniversary, when they planned to travel to Europe.

David had been a decorated fighter pilot and Air Force officer. In fact, during the September 11 attacks, he was the general who ordered fighter jets scrambled to patrol the skies over Washington. Under his standing orders, the pilots had discretion to shoot down any airliner that threatened the city. Yet even as an Air Force general, he turned down the chauffeured car. He preferred the subway.

At 4:58 p.m., the train operator pulled the emergency brake, and a sickening screech of metal cut through the rhythmic clatter of the wheels. Then Train 112 hit something: another train, stopped on the tracks for reasons no one yet understood. Shattered glass and twisted metal flew in every direction, mixed with the screams of passengers. The impact crushed seats, ceiling panels, and metal bars into a wall of debris 13 feet thick, which tore into Train 112 and killed Ann, David, and seven others.

The collision could have been avoided. The Washington Metro system runs on more than 100 miles of track, all of it networked, with trains monitored throughout their runs. When two trains get too close, they slow down automatically. But that day, as Train 112 rounded a curve, another train stood on the track ahead of it — a real event in the physical world that the track sensors simply failed to register. Train 112 accelerated on its own, because as far as the sensors could tell, the track ahead was clear. By the time the operator saw the stopped train and hit the emergency brake, the collision was already unavoidable.

Even as rescue workers pulled the injured from the wreckage, Metro engineers went to work. They had to make sure no other trains were in danger, and to do that they had to solve a baffling mystery: how had a train two football fields long simply vanished?

II

Sensor failures like the one that doomed Train 112 happen more often than you would think. Consider the following list — real news headlines from a single week:

• A mine disaster in Brazil

• Another day, another hack: credit-card-stealing malware hits a hotel chain

• South Korea's Hyundai recalls vehicles over a faulty brake switch

• The Flint water crisis deepens, a "government failure" in Washington

• "Massive intelligence failures" behind the Paris terror attacks

• Vancouver man wrongfully imprisoned for nearly 30 years finally settles his lawsuit

• Fighting Ebola: scientists blast an "extremely fragile global system"

• Investigation into the murder of a 7-year-old finds systemic failures to protect the child

• Land-clearing fires in Indonesia spiral into massive wildfires, causing ecological disaster

• The FDA investigates an E. coli outbreak at Mexican restaurants in Washington and Oregon

That week may look unusually bad, but there was nothing special about it. Few weeks go by without several catastrophic breakdowns. One week it is a factory accident, the next a major corporate bankruptcy, and the week after that a horrifying medical error. Even small glitches can do enormous damage. In recent years, several airlines have grounded their entire fleets because of technical system failures, stranding passengers for days. Such failures naturally make us angry, but we have come to see them as routine. Living in the 21st century, we depend on countless complex systems that shape our lives profoundly — from power grids to water treatment plants, from transportation and communication networks to health care and the courts. And sometimes these systems land us in trouble.

Some disasters are enormous — BP's oil spill in the Gulf of Mexico, Japan's Fukushima nuclear disaster, the global financial crisis — and each appears to stem from its own distinct problem. But at bottom, these disasters have strikingly similar roots. They share a common DNA, one that researchers have only recently begun to recognize. A common DNA means that an accident in one industry can teach lessons to people in another: dentists can learn from pilots, marketing teams can learn from SWAT units. Understanding the deep causes of disaster in high-risk, novel fields like deep-sea exploration and high-altitude mountaineering also teaches us about breakdowns in our everyday systems. It turns out that everyday failures — a botched project, a bad hire, even a dinner party gone wrong — have a great deal in common with oil spills and deaths on the mountain. Fortunately, researchers around the world have spent the past few decades working out solutions we can borrow, which tell us how to make decisions, build teams, design systems, and prevent the breakdowns that are becoming ever more common.

The book is divided into two parts. Part One asks why systems fail, uncovering the shared causes hidden behind events that seem to have nothing to do with one another — Starbucks's social-media disasters, the Three Mile Island nuclear accident, the Wall Street crash, and a bizarre scandal at a post office in a remote English town. Part One also explores a paradox of progress: our systems have grown more capable, but also more complex and more demanding, creating exactly the conditions in which a small mistake can turn into a catastrophe. Systems that once seemed harmless can now, on occasion, cost lives, bankrupt companies, or send innocent people to prison. Part One also shows that the same changes that leave our systems defenseless against accidental breakdowns create fertile ground for deliberate crimes like hacking and fraud.

Part Two, the heart of the book, looks at solutions that work. It shows how people start from small mistakes and trace them back to the source of major threats: how a receptionist saved a life by reporting what she saw to her boss; how a training program pilots once mocked as "charm school" made their flying safer than ever before. In this part we examine how diversity helps us avoid big errors, and what climbers on Mount Everest and engineers at Boeing can teach us about the power of simplicity. We look at how film crews and emergency-room staff handle crises, and how those same methods rescued Facebook's botched IPO and salvaged Target's failed expansion into Canada. We also return to the puzzle of the train the system could not see, and to what engineers concluded about the tragedy.

We came to this book by two different routes. Chris began his career trading securities and financial derivatives, and watched from his own trading desk as Lehman Brothers collapsed and global stock markets went into turmoil. Around that same time, he began training as a pilot and grew fascinated with how to avoid catastrophic mistakes. András has stayed in academia, studying why organizations struggle as they grow more complex. A few years ago, he created a course called Catastrophic Failure in Organizations, which brings managers from different backgrounds together to study major disasters and share their experience of everyday breakdowns.

The material for this book comes from accident reports, academic research, and wide-ranging interviews with people from CEOs to first-time homebuyers. The ideas we found there can explain all kinds of breakdowns and offer practical insight anyone can use. In an age when disasters come one after another, these ideas can help people make the right decisions at work and in their personal lives, in running organizations, and in confronting some of the biggest global challenges we face.

III

On a clear afternoon in May 1996, Berman was in New York City with his family when his pager went off. Berman was then serving on the National Transportation Safety Board's "Go Team," which stood ready around the clock to respond to major accidents. He soon learned how grim this one was: ValuJet Flight 592, carrying more than 100 passengers, had disappeared from radar ten minutes after taking off from Miami and had crashed into the Florida Everglades. The pilots' radio transmissions showed a fire in the cabin, but what had started it was a mystery.

Investigations into the Washington Metro Train 112 crash and ValuJet Flight 592 revealed that both accidents came from the same cause: our systems keep growing more complex. When Train 112 crashed, National Public Radio producer Jasmine Garsd happened to be riding in one of the rear cars. "The impact was like a movie on fast-forward that suddenly stopped," she recalled. "In a moment like that, you realize two things — how small and fragile we are inside this vast machine world we have built, and how little we understand about that fragility."

But there is reason for hope. Over the past few decades, our understanding of complexity, organizational behavior, and cognitive psychology has opened a window onto how small mistakes build, step by step, into major disasters. We now understand not only how these accidents happen, but also how small steps can prevent them. A few companies, a few researchers, and a few teams are leading this revolution, searching out ways to prevent breakdowns — ways that require neither advanced technology nor a million-dollar budget.

In the spring of 2016, we arranged for Ben Berman to give a talk to a group of people interested in aviation risk-management courses. Their backgrounds were remarkably varied: HR professionals and civil servants, entrepreneurs and doctors, nonprofit managers and lawyers, even someone from the fashion industry. Berman's talk cut across every one of those fields. "The cost of system breakdowns is staggering, but it is easy to underestimate — and you are likely to face something like this in your career or your personal life," he told them, pausing to look at the audience. "The good news, I think, is that you can change the outcome."

Cover image: the Washington Metro Train 112 crash, via Wikipedia