“Data mining can discover surprisingly useful regularities that are really just preexisting patterns of exclusion and inequality.”
Solon Barocas and Andrew D. Selbst examined how data-driven systems can create disparate impact even when designers do not include an explicitly protected characteristic or intend to discriminate. Historical data record earlier decisions, unequal access, and institutional priorities. A model trained to reproduce successful past outcomes may therefore treat an unjust pattern as a useful prediction and carry it forward at greater scale. Read in its original setting, the sentence is less a slogan than a marker of what the work established and what still had to be tested.
The central problem is that correlation does not erase history. Zip codes can reflect residential segregation; employment histories can reflect unequal opportunities; medical spending can reflect unequal access to care. Removing one sensitive field may leave many proxies. Fairness work must therefore examine labels, sampling, features, objectives, and the decision process rather than looking only for a direct use of race, sex, disability, or another protected attribute. The underlying idea is therefore practical as well as intellectual. It gives teams a way to ask which evidence supports a claim and which conclusions still require human judgment.
Automated screening now affects hiring, credit, housing, insurance, education, healthcare, and public benefits. Organizations should conduct impact assessments before deployment and measure outcomes after real use. That includes comparing error rates and selection rates across relevant groups, examining intersections, and providing routes for correction. Legal, technical, and community expertise should shape the evaluation together because no single metric captures the entire problem. In practice, usefulness should be measured after deployment as well as before it. Teams need to watch for distribution shifts, workarounds, unequal effects, and costs that a laboratory metric cannot reveal.
A fairness metric can narrow attention to what is easy to count while leaving the policy itself unchallenged. A perfectly balanced model may still allocate too little opportunity or enforce a harmful rule. Institutions need to ask whether prediction is appropriate, which people bear risk, and whether the system changes power or merely automates existing choices. People affected by the decision should have meaningful information and recourse. The quote ultimately asks for intellectual humility. Progress becomes more credible when builders state what they know, invite scrutiny, and remain prepared to revise both the model and the institution around it. Barocas and Selbst make historical context part of technical analysis. Data are not a neutral mirror; they are evidence of how institutions have already treated people. That standard keeps history useful. It connects a memorable research claim to concrete choices about measurement, accountability, participation, and the people who experience the system outside the laboratory. The work continues with care.
Labels, features, sampling, and past decisions can encode unequal access and discriminatory institutions.
A model may learn those regularities as predictive even when protected attributes are removed or no one intended harm.
Location, spending, education, and work history can correlate with protected status because of social structure.
Audits should examine multiple groups, intersections, error types, selection rates, and outcomes after deployment.
A balanced prediction can still support an unjust policy or distribute too little opportunity overall.
Meaningful governance asks whether automation is appropriate and provides participation, explanation, correction, and appeal.
Explore more "Quotes of The Day"
Continue exploring how writers, thinkers, artists, and advocates have described the choices that shape an everyday life.