What Does a Fair Algorithm Actually Look Like?

In some ways, artificial intelligence acts like a mirror. Machine learning tools are designed to detect patterns, and they often reflect back the same biases we already know exist in our culture. Algorithms can be sexist, racist, and perpetuate other structural inequalities found in society. But unlike humans, algorithms arent under any obligation to explain themselves. In fact, even the people who build them arent always capable of describing how they work.

That means people are sometimes left unable to grasp why they lost their health care benefits, were declined a loan, rejected from a job, or denied bailall decisions increasingly made in part by automated systems. Worse, they have no way to determine whether bias played a role.

In response to the problem of AI bias and so-called black box algorithms, many machine learning experts, technology companies, and governments have called for more fairness, accountability, and transparency in AI. The research arm of the Department of Defense has taken an interest in developing machine learning models that can more easily account for how they make decisions, for example. And companies like Alphabet, IBM, and the auditing firm KPMG are also creating or have already built tools for explaining how their AI products come to conclusions.

But that doesnt mean everyone agrees on what constitutes a fair explanation. Theres no common standard for what level of transparency is sufficient. Does a bank need to publicly release the computer code behind its loan algorithm to be truly transparent? What percentage of defendants need to understand the explanation given for how a recidivism AI works?

Algorithmic transparency isnt an end in and of itself, says Madeleine Clare Elish, a researcher who leads the Intelligence & Autonomy Initiative at Data & Society. Its necessary to ask: Transparent to whom and for what purpose? Transparency for the sake of transparency is not enough.

By and large, lawmakers havent decided what rights citizens should have when it comes to transparency in algorithmic decision-making. In the US, there are some regulations designed to protect consumers, including the Fair Credit Reporting Act, which requires individuals be notified of the main reason they were denied credit. But there isnt a broad right to explanation for how a machine came to a conclusion about your life. The term appears in the European Union's General Data Protection Regulation (GDPR), a privacy law meant to give users more control over how companies collect and retain their personal data, but only in the non-binding portion. Which means it doesn't really exist in Europe, either, says Sandra Wachter, a lawyer and assistant professor in data ethics and internet regulation at the Oxford Internet Institute.

GDPRs shortcomings havent stopped Wachter from exploring what the right to explanation might look like in the future, though. In an article published in the Harvard Journal of Law & Technology earlier this year, Wachter, along with Brent Mittelstadt and Chris Russell, argue that algorithms should offer people counterfactual explanations, or disclose how they came to their decision and provide the smallest change that can be made to obtain a desirable outcome.

For example, an algorithm that calculates loan approvals should explain not only why you were denied credit, but also what you can do to reverse the decision. It should say that you were denied the loan for having too little in savings, and provide the minimum amount you would need to additionally save to be approved. Offering counterfactual explanations doesnt require the researchers who designed an algorithm release the code that runs it. Thats because you dont necessarily need to understand how a machine learning system works to know why it reached a certain decision.

The industry fear is that [companies] will have to disclose their code, says Wachter. But if you think about the person who is actually affected by [the algorithms decision], they probably dont think about the code. Theyre more interested in the particular reasons for the decision.

Counterfactual explanations could potentially be used to help conclude whether a machine learning tool is biased. For example, it would be easy to tell a recidivism algorithm was prejudiced if it indicated factors like a defendants race or zip code in explanations. Wachters paper has been cited by Google AI researchers and also by what is now called the European Data Protection Board, the EU body that works on GDPR.

A group of computer scientists has developed a variation on Wachters counterfactual explanations proposal, which was presented at the International Conference for Machine Learnings Fairness, Accountability and Transparency conference this summer. They argue that rather offering explanations, AI should be built to provide "recourse," or the ability for people to feasibly modify the outcome of an algorithmic decision. This would be the difference, for example, between a job application that only recommends you obtain a college degree to get the position, versus one that says you need to change your gender or age.

No one agrees on what an explanation is, and explanations arent always useful, says Berk Ustun, the lead author of the paper and a postdoctoral fellow at Harvard University. Recourse, as they define it, is something researchers can actually test.

As part of their work, Ustun and his colleagues created a toolkit computer scientists and policymakers can use to calculate whether or not a linear algorithm provides recourse. For example, a health care company could see if their AI uses things like marital status or race as deciding factorsthings people cant easily modify. The researchers work has already garnered attention from Canadian government officials.

Simply because an algorithm offers recourse, however, doesnt mean its fair. Its possible an algorithm offers more achievable recourse to wealthier people, or to younger people, or to men. A woman might need to lose far more weight for a health care AI to offer her a lower premium rate than a man would, for example. Or a loan algorithm might require black applicants have more in savings to be approved than white applicants.

The goal of creating a more inclusive and elastic society can actually be stymied by algorithms that make it harder for people to gain access to social resources, says Alex Spangher, a PhD student at Carnegie Mellon University and an author on the paper.

There are other ways for AI to be unfair that explanations or recourse alone wouldn't solve. Thats because providing explanations doesnt do anything to address which variables automated systems take into consideration in the first place. As a society we still need to decide what data should be allowed for algorithms to use to make inferences. In some cases, discrimination laws may prevent using categories like race or gender, but it's possible that proxies for those same categories are still utilized, like zip codes.

Corporations collect lots of types of data, some of which may strike consumers as invasive or unreasonable. For example, should a furniture retailer be allowed to take into consideration what type of smartphone you have when determining whether you receive a loan? Should Facebook be able to automatically detect when it thinks youre feeling suicidal? In addition to arguing for a right to explanation, Wachter has also written that we need a right to reasonable inferences.

Building a fair algorithm also doesnt do anything to address a wider system or society that may be unjust. In June, for example, Reuters reported that ICE altered a computer algorithm used since 2013 to recommend whether an immigrant facing deportation should be detained or released while awaiting their court date. The federal agency removed the release recommendation entirelythough staff could still override the computer if they chosewhich contributed to a surge in the number of detained immigrants. Even if the algorithm had been designed fairly in the first place (and researchers found it wasnt), that wouldn't have prevented it from being modified.

The question of What it means for an algorithm to be fair? does not have a technical answer alone, says Elish. It matters what social processes are in place around that algorithm.

• How the US fought China's cybertheftwith a Chinese spy
• Turning Californias weed into the champagne of cannabis
• Inside the secret conference plotting to launch flying cars
• Cities team up to offer broadband and the FCC is mad
• PHOTOS: The space shuttle program's golden age
• Get even more of our inside scoops with our weekly Backchannel newsletter