“Everything fails, all the time.”
Werner Vogels used this blunt sentence at The Next Web conference in April 2008 while discussing the realities of operating Amazon’s large distributed services. A report from the event placed the words beside a startling example: whole data centers can be lost. Vogels’ point was not that customers should accept constant interruption. It was almost the opposite. Providers must assume disks, machines, networks, software, and facilities will fail so that the service presented to a customer can remain dependable. Coming soon after widely noticed problems with Amazon S3, the remark turned an uncomfortable operational truth into a concise design rule rather than a promise that infrastructure could be made invulnerable.
At small scale, failure feels exceptional because any particular component may work for a long time. At large scale, rare events become routine somewhere in the fleet. Reliability therefore cannot depend on predicting which device or request will break next. It must emerge from redundancy, isolation, recovery, and clear limits on how far a fault can spread. Data is copied across independent locations; work is retried without being duplicated incorrectly; unhealthy components are removed from service; queues absorb temporary disruption. Monitoring tells operators whether those mechanisms are succeeding. The sentence shifts attention from preventing every fault to maintaining the useful behavior of the whole system while faults occur.
The quote can sound fatalistic if separated from that purpose. “Everything fails” does not excuse careless engineering, weak testing, or avoidable outages. Nor does it require maximum redundancy for every feature. Resilience has costs, and the appropriate design depends on how much interruption or data loss a user can tolerate. The practical discipline is to name those tolerances, remove single points of failure where consequences justify it, and test recovery before a crisis. Some components should fail closed for safety; others should degrade gracefully to preserve access. Treating failure as normal creates responsibility because it forces teams to decide which outcomes must survive and how they will know when safeguards no longer work.
Modern systems make Vogels’ principle even harder to ignore. A single product may depend on cloud regions, identity providers, content networks, open-source packages, payment processors, and machine-learning services operated by different organizations. The application can fail although every line owned by its team is correct. Resilient design now includes dependency budgets, circuit breakers, idempotent operations, backups that are actually restored in tests, and interfaces that preserve a limited mode when optional services disappear. It also includes honest communication when customers are affected. The lasting lesson is simple but demanding: reliability is not the absence of component failure. It is the engineered ability to continue, recover, and learn while failure remains an ordinary condition.
Vogels used the line at The Next Web conference in April 2008 while explaining the operational reality behind Amazon’s distributed services.
The surrounding point was that providers should absorb infrastructure failures so customers can keep using the service without managing those events themselves.
At scale, rare component faults occur routinely, so availability must come from redundancy, isolation, recovery, and controlled blast radius.
The observation is not an excuse for outages; it raises the standard by requiring teams to design and test how essential behavior survives them.
Applications now depend on many external services and packages, so they can break even when their own code is functioning as intended.
Retries, circuit breakers, tested restores, graceful degradation, and honest incident learning turn the principle into responsible engineering practice.
Explore more "Quotes of The Day"
Discover more notable quotes from influential voices across politics, science, business, technology, sports, and culture. Each quote offers insight into how ideas, beliefs, and decisions shape the world around us.
