“Algorithmic bias, like human bias, results in unfairness.”
Joy Buolamwini delivered this warning in a 2016 TEDx talk about what she calls the “coded gaze”: the priorities, assumptions, and omissions that can be embedded in technical systems. As a graduate researcher at the MIT Media Lab, she found that facial-analysis software did not reliably detect her face until she put on a white mask. The experience exposed a gap between a system’s impressive general claim and its uneven performance for particular people. “Algorithmic bias, like human bias, results in unfairness” distilled the larger lesson: automation can reproduce discrimination even when no computer has an intention to discriminate.
Bias can enter at many stages. A data set may underrepresent certain skin tones, genders, ages, accents, disabilities, or cultural settings. Labels may reflect subjective judgments. A team may optimize for an average score that conceals large errors for a smaller group. Decision makers may then deploy the model in a setting far more consequential than the one in which it was tested. Once placed inside hiring, policing, lending, health care, or identity verification, those technical gaps become lived experiences. The apparent neutrality of a numerical output can make the resulting unfairness harder to see and easier to defend.
Buolamwini’s Gender Shades research, conducted with Timnit Gebru, made the disparity measurable. Commercial gender-classification systems performed substantially better on lighter-skinned men than on darker-skinned women in the study’s benchmark. The finding did more than identify a faulty feature. It showed why aggregate accuracy is insufficient when errors are distributed unequally. The work helped push companies and policymakers to examine data composition, demographic performance, consent, and the limits of face-analysis technology. It also demonstrated how an audit conducted outside a product team can reveal harms that internal testing overlooks.
The quote remains useful because it connects engineering practice with civil rights. Fairness is not achieved by removing people from a decision and inserting software. It requires explicit choices about which outcomes matter, how groups are compared, what error rates are acceptable, and who can appeal. Some uses may be too intrusive or dangerous to justify even after accuracy improves. Others can become safer through representative data, independent evaluation, documentation, and ongoing monitoring. Publishing results by demographic group gives communities and regulators evidence they can examine instead of asking them to trust a single headline score. Buolamwini’s message is ultimately constructive: when society acknowledges that bias can be coded, it can demand that those codes be examined, challenged, and changed.
Buolamwini’s phrase grew from a concrete failure: facial-analysis software had difficulty detecting her face. She used that experience to investigate whose faces and experiences were missing from widely used systems.
The “coded gaze” describes how the worldview of those who build technology can shape what a system sees and how it behaves, often without an explicit discriminatory rule.
Gender Shades evaluated commercial classification systems across skin tone and gender presentation. The study found sharply uneven error rates, with darker-skinned women receiving the least reliable results.
Those group-level measurements exposed a weakness hidden by overall accuracy. A system can look strong on average while repeatedly failing the people least represented in its data.
More representative data and disaggregated testing are essential, but accuracy alone does not make every application legitimate. Consent, purpose, oversight, and the possibility of appeal matter too.
Buolamwini’s warning asks institutions to treat algorithmic outcomes as choices that can be audited. Automated decisions require accountability precisely because they can spread unfairness quickly and quietly.
Explore more "Quotes of The Day"
Continue exploring how writers, thinkers, artists, and advocates have described the choices that shape an everyday life.