If Algorithms Know All, How Much Should Humans Help?

Armies of the finest minds in computer science have dedicated themselves to improving the odds of making a sale. The Internet-era abundance of data and clever software has opened the door to tailored marketing, targeted advertising and personalized product recommendations.

Shake your head if you like, but that's no small thing. Just look at the technology-driven shake-up in the advertising, media and retail industries.

This automated decision-making is meant to take the human out of the equation, but it is an all-too-human impulse to want someone looking over the result spewed out of the computer. Many data quants see marketing as a low-risk - and, yes, lucrative - petri dish in which to hone the tools of an emerging science. "What happens if my algorithm is wrong? Someone sees the wrong ad," said Claudia Perlich, a data scientist who works for an ad-targeting startup. "What's the harm? It's not a false positive for breast cancer."

But the stakes are rising as the methods and mindset of data science spread across the economy and society. Big companies and startups are beginning to use the technology in decisions like medical diagnosis, crime prevention and loan approvals. The application of data science to such fields raises questions of when close human supervision of an algorithm's results is needed.

These questions are spurring a branch of academic study known as algorithmic accountability. Public interest and civil rights organizations are scrutinizing the implications of data science, both pitfalls and potential. In the foreword to a report in September, "Civil Rights, Big Data and Our Algorithmic Future," Wade Henderson, president of The Leadership Conference on Civil and Human Rights, wrote, "Big data can and should bring greater safety, economic opportunity and convenience to all people."

Take consumer lending, a market with several big data startups. Its methods amount to a digital-age twist on a basic tenet of banking: Know your customer. By harvesting data sources like social network connections, or even by looking at how an applicant fills out online forms, the new lenders say they can know borrowers as never before, and more accurately predict whether they will repay than they could have by simply looking at a person's credit history.

The promise is more efficient loan underwriting and pricing, saving people lots of money. But big data lending depends on software algorithms poring through mountains of data, learning as they go. It is a complex, automated system - and even enthusiasts have qualms.

"A decision is made about you, and you have no idea why it was done," said Rajeev Date, an investor in data-science lenders and a former deputy director of Consumer Financial Protection Bureau. "That is disquieting."

The concern is similar in other fields. Since its Watson computer beat human "Jeopardy!" champions, IBM has taken its data-driven artificial intelligence technology well beyond brainy games. Health care has been a major initiative. The history of "expert" decision-support technology in medicine has been disappointing; the systems have not been smart or fast enough to really help doctors in day-to-day practice.

But IBM scientists in collaboration with researchers at leading medical groups - including the Cleveland Clinic, the Mayo Clinic and the Memorial Sloan Kettering Cancer Center - are making progress. Watson can read through medical documents at a pace incomprehensible to humans: many thousands per second, searching for clues, correlations and insights.

The software has been used to help train medical students and is starting to be deployed in clinical settings in oncology, offering diagnostic and treatment recommendations as a kind of quick-witted digital assistant.

IBM has also developed a software program called Watson Paths, which is a visual tool that allows a doctor to see the underlying evidence and inference paths Watson took in making a recommendation.

"It's not sufficient to give a black-box answer," said Eric Brown, IBM's director of Watson technologies.

Watson Paths points to the need for some machine-to-man translation as data science advances. As Danny Hillis, an artificial intelligence expert, put it, "The key thing that will make it work and make it acceptable to society is story telling." Not so much literal storytelling, but an understandable audit trail that explains how an automated decision was made. "How does it relate to us?" Hillis said. "How much of this decision is the machine and how much is human?"

Keeping a human in the loop is one approach. The new data-science lenders are animated by data and software. But one of the startups in San Francisco, Earnest, has at least one of its staff members review the predictive recommendations of its software, even if the algorithms are rarely overruled. "We think the human element will always be an important piece in our process to make sure we're getting it right," said Louis Beryl, co-founder and chief executive of Earnest.

But such a stance, others say, amounts to a comforting illusion - good marketing perhaps, but not necessarily good data science. Giving a person veto power in algorithmic systems, they say, introduces human bias. The promise of big data decision-making is that decisions based on data and analysis - more science, less gut feel and rule of thumb - will yield better outcomes.

Yet even if optimism is justified, there is a serious challenge, given the complexity and opacity of data science. Will a technology that promises large benefits on average sufficiently protect the individual from a mysterious and wayward decision that might have a lasting effect on a person's life?

One solution, according to Gary King, director of Harvard's Institute for Quantitative Social Science, may be for the human creators of the scoring algorithms to adjust them to give somewhat greater weight to the individual, reducing the risk of getting it wrong.

In banking, for example, an algorithm might be tuned to reduce the probability of misclassifying a loan applicant as a deadbeat, even if the trade-off is a few more delinquent loans for the lender.

"The goal," King said, "is not necessarily to have a human look at the outcome afterwards, but to improve the quality of the classification for the individual."

In a sense, a math model is the equivalent of a metaphor, a descriptive simplification. It usefully distills, but it also somewhat distorts. So at times, a human helper can provide that dose of nuanced data that escapes the algorithmic automaton. "Often, the two can be way better than the algorithm alone," King said.