A widely used software program that helps guide care for millions of patients is flawed by unintentional racial bias that leads to blacks getting passed over for special care, according to a new study.
The software predicts costs rather than sickness. It is used by U.S. insurers and hospitals to direct higher-cost patients into health care programs designed to help them stay on medications or out of the hospital.

Whites tend to be higher-cost patients even when they’re not as sick as blacks. The study found the software regularly suggested letting healthier white patients into health care risk management programs ahead of blacks who were less healthy because those white patients were more costly.
Fixing the software could more than double the number of black patients enrolled in these programs, said Dr. Ziad Obermeyer of the University of California, Berkeley, who led the research published Thursday in the journal Science.
“The problem was the algorithm was built to predict who’s going to cost money next year, not who’s going to need health care,” said Obermeyer, who studies machine learning in medicine.
The study was based on patient data from one large hospital where blacks cost $1,800 less per year than whites with the same number of chronic illnesses. That’s a pattern seen across the U.S.
The researchers had no financial relationship with the health data company and did not name the company in their paper, but they did share their findings with the company, Obermeyer said.

Some Experts Worry That Bias Will Be Further Baked Into the System

The company, Optum, acknowledged that its software was the subject of the study and responded Friday, calling the findings “misleading” because hospitals can and should supplement the company’s cost algorithm with their own socio-economic data.
“The cost model is just one of many data elements intended to be used to select patients for clinical engagement programs, including, most importantly, the doctor’s expertise and knowledge of his or her patient’s individual needs,” said Optum spokesman Tyler Mason.
Health insurers have been trying to identify patients with chronic or complex medical conditions and to enroll them in programs that help manage their care. The idea is to cut down on big bills from things like emergency room visits and hospital stays by making sure patients are taking their medicines and keeping up with their health.
A recent analysis of claims from nearly 6 million people covered by employer-sponsored health insurance found that 20% of the population accounted for 84% of the total spending in that group, according to the non-profit Employee Benefit Research Institute.
Other research has tied racial disparities in health care to a cluster of factors including doctors’ unconscious attitudes, blacks’ distrust in the health care system, lack of transportation and poverty.
As big data drives more health care decisions, some experts worry that bias will be further baked into the system. In an accompanying editorial, Ruha Benjamin of Princeton University wrote that older Jim Crow forms of discrimination are feeding into a “New Jim Code” source of bias in “automated systems that hide, speed and deepen racial discrimination behind a veneer of technical neutrality.”