★★★★ · 1984 #risk#systems

Normal Accidents

by Charles Perrow

The book in brief. Charles Perrow argues that complex interactions and tight coupling can turn small failures into major accidents. Additional safeguards may themselves add complexity, while rapid interdependence leaves little room for recovery. The book offers a framework for assessing when system architecture and organisational choices make failures hard to contain.

Key ideas

  • Complex interactions. Failures can combine and cascade before anyone understands how they fit together.
  • Tight coupling. Interdependence leaves little time or room to isolate a failure.
  • Safeguards add complexity. Extra protections can make the system harder to understand.
  • Architecture matters. System design and organisational choices determine whether failures can be contained.

Why it matters to my work. In Can We Control AI Without Understanding It?, Perrow gives me a lens for my most immediate worries about AI: cybersecurity and the fragility of essential social and technical infrastructure. As we connect agents to more services, failures are likely to spread through systems on which many people depend, and connected systems can transmit them before people can intervene. Serious harm does not require an agent escaping all human control.