“The fate of humanity must never be left to the ‘black box’ of an algorithm.”
Addressing the United Nations Security Council in 2024, António Guterres declared, “The fate of humanity must never be left to the ‘black box’ of an algorithm.” His remarks focused on peace and security, where AI may influence surveillance, intelligence analysis, target selection, cyber operations, and autonomous weapons. A black box is dangerous in that setting not only because its internal reasoning is difficult to explain. It can place distance between a life-and-death decision and the person or institution legally responsible for making it. Guterres’s line insists that automation cannot become an escape from human judgment or international law.
Human control must be meaningful. A person who receives an algorithmic recommendation seconds before action and lacks the information or authority to question it is not exercising real oversight. Effective control requires time, training, access to evidence, and the power to refuse. Systems should communicate uncertainty and be designed so intervention remains possible. In military and security settings, boundaries must be established before deployment, because the speed of automated interaction can create escalation faster than diplomacy or command structures can respond.
The warning also applies beyond weapons. Black-box systems can influence borders, policing, public benefits, and access to essential services. Opacity makes it difficult for affected people to understand an outcome, correct bad data, or prove discrimination. Some complex models may never provide a complete causal explanation for every output, but institutions can still be transparent about purpose, evidence, error rates, governance, and appeal. The responsibility to explain a decision belongs to the authority using the system, not to the individual harmed by it.
Guterres’s quote places AI inside an existing moral and legal order. International humanitarian law, human rights, and democratic accountability do not disappear when software enters the process. If an application cannot preserve those obligations, the appropriate response may be a strict limit or prohibition rather than a better interface. Human beings built the systems, choose where they operate, and can establish rules for their use. The black box should never become the final sovereign. Intelligence can inform consequential choices, but accountable people and institutions must remain able—and required—to answer for them. That requirement should be tested in exercises before a crisis, not assumed from an organizational chart. Institutions need to show that operators can understand alerts, challenge recommendations, and halt action under realistic pressure. Control is credible only when it survives the conditions in which it will be needed.
Opacity can hide uncertainty, bias, or failure in decisions involving force, surveillance, and national infrastructure.
It can also blur the chain of legal and moral responsibility for actions with irreversible consequences.
Oversight requires time, expertise, relevant information, and actual authority to reject or stop an automated recommendation.
A ceremonial person in the loop is not enough when the process is too fast or opaque for independent judgment.
Human rights and humanitarian obligations govern institutions whether a decision is made directly or with algorithmic assistance.
Uses that cannot preserve those obligations may need strong limits rather than a promise of later technical improvement.
Explore more "Quotes of The Day"
Continue exploring how writers, thinkers, artists, and advocates have described the choices that shape an everyday life.