“The algorithm assigned Black patients the same level of risk at lower levels of health.”
Ziad Obermeyer and colleagues studied a widely used health-management algorithm that ranked patients for additional care. The system used expected healthcare spending as a proxy for health need. Because Black patients historically received less care for the same level of illness, spending understated their needs. The model therefore assigned equal risk scores to Black patients who were substantially sicker than White patients. The line has lasted because it captures a research turning point without pretending that one paper settled every scientific or social question around it.
The case demonstrates that a proxy can be statistically convenient and morally wrong for the task. Cost was easier to observe than health, but it reflected access, trust, insurance, location, and patterns of unequal treatment. Race did not need to appear as an input for racial bias to emerge. The target variable carried the disparity. Auditing only the feature list would have missed the central mechanism. The sentence also shows why definitions matter. When researchers specify what they are measuring, disagreement can become an experiment instead of a contest between impressions.
Healthcare organizations use algorithms for triage, outreach, scheduling, diagnosis support, and resource allocation. Each use should begin with a careful statement of the desired outcome and a review of whether the available label truly represents it. Validation needs clinical expertise, subgroup analysis, and comparison with direct measures of need. After deployment, teams must monitor who receives services and whether gaps narrow or widen. For present-day teams, the lesson is to translate broad enthusiasm into observable outcomes. A strong deployment plan names users, failure modes, review points, and the evidence that would justify changing course.
High average accuracy can conceal systematic under-service. Correcting one algorithm is not enough if access and quality remain unequal across the institution. Organizations should investigate the data-generating process, provide human review, and give patients routes to correct records or challenge decisions. Vendors must support independent evaluation rather than treating models and proxies as trade secrets beyond scrutiny. The most durable response is neither reflexive celebration nor blanket rejection. It is disciplined adoption: verify the benefit, disclose the limitation, and preserve a workable human alternative. Obermeyer’s finding turned an abstract fairness concern into a concrete diagnostic lesson: the choice of target can matter more than whether a protected attribute is visible. Responsible progress depends on maintaining that distinction. Technical possibility opens a door, while public value depends on careful implementation, meaningful oversight, and the ability to learn from failure. The work continues with care.
The algorithm predicted healthcare spending, using it as a stand-in for how much care a patient needed.
Unequal access meant spending was systematically lower for Black patients with comparable illness, distorting the risk score.
Removing race from inputs cannot prevent bias when the label itself reflects structural inequality.
Teams should examine how every target was produced and whether it measures the real outcome the decision is meant to serve.
Clinical expertise, subgroup analysis, direct need measures, and monitoring of actual service delivery belong in validation.
Patients also need correction and appeal mechanisms, while vendors should permit meaningful independent scrutiny.
Explore more "Quotes of The Day"
Continue exploring how writers, thinkers, artists, and advocates have described the choices that shape an everyday life.