Nancy Leveson, 2011

AI-generated editorial portrait of Nancy Leveson in a systems safety research setting

“Safety is a system property, not a component property.”

Nancy Leveson used this statement in her systems-safety work, including a 2011 paper in Safety Science and the book Engineering a Safer World. It challenges the habit of treating safety as the sum of reliable parts. A valve can operate exactly as specified, a software service can return the expected output, and an operator can follow a local procedure while their interactions still create a hazardous result. To judge safety, engineers must examine the system’s goals, constraints, feedback, and environment rather than certify each component in isolation. The question shifts from “Which part broke?” to “Why was the system able to enter an unsafe state?”

The distinction between safety and reliability is central. Reliability asks whether a component performs consistently according to its specification. Safety asks whether unacceptable losses are prevented in the larger context. A highly reliable component can contribute to danger if the specification is wrong, the surrounding assumptions have changed, or separate decisions combine in an unexpected way. Leveson’s systems approach therefore treats accidents as control problems involving technical design, human behavior, organizational policy, and information flow. It looks for missing constraints and delayed feedback as carefully as it looks for failed hardware or defective code.

This does not make component testing unimportant or excuse poor workmanship. Reliable hardware and software remain necessary in many critical systems; they are simply not sufficient. System safety adds questions that a component test cannot answer. Are hazards identified early enough to shape the design? Do controllers receive the feedback needed to recognize a dangerous state? Can operators understand what automation is doing? Do management incentives or communication gaps weaken a constraint that looks sound on paper? The quote expands responsibility without dissolving it. It asks every contributor to connect local success criteria with the losses the whole system must prevent.

Today, software coordinates transportation, medical care, energy, finance, and industrial equipment through layers of cloud services and automation. Machine-learning components add behavior that may be probabilistic or difficult to interpret, but the basic lesson remains stable: risk emerges from relationships. Teams should test individual parts while also modeling how decisions, interfaces, users, and operating conditions interact. Incident reviews should look beyond the last failed component to the structure that allowed the failure to become harmful. Safety improves when the whole system is designed to maintain constraints, detect drift, and recover before local problems combine into loss. That requires technical evidence, operational practice, and organizational learning to reinforce one another.

Leveson used the line in her systems-safety work to challenge accident models that search for one failed part or one mistaken operator.

Her 2011 Safety Science paper describes safety as an emergent property maintained through constraints and feedback across technical and organizational levels.

A component can be reliable according to its specification while still participating in an unsafe interaction, especially when assumptions or operating conditions change.

The quote does not replace component testing; it adds analysis of goals, interfaces, controllers, users, incentives, and feedback that local tests cannot cover.

Cloud services, medical devices, transportation, industry, and AI increasingly combine many locally correct parts into behavior no single owner fully controls.

Leveson’s principle directs teams to model interactions, maintain safety constraints, detect drift, and learn beyond the last visible failure in an incident.

Explore more "Quotes of The Day"

Discover more notable quotes from influential voices across politics, science, business, technology, sports, and culture. Each quote offers insight into how ideas, beliefs, and decisions shape the world around us.