The problem with the EU’s A.I. strategy

This is the web version of Eye on A.I., Fortune’s weekly newsletter covering artificial intelligence and business. To get it delivered weekly to your in-box, sign up here.
Last week, the European Union issued its long-anticipated white paper on artificial intelligence. The document is a prequel to new legislation and regulations governing the technology that are likely to have global consequences.
That’s because, as with Europe’s privacy law, GDPR, any new A.I. rules are likely to apply to anyone who sells to an EU customer, processes the data of an EU citizen, or has a European employee. And, as with GDPR, any rules Europe enacts may serve as a model for other nations—or even individual U.S. states—looking to regulate A.I.

The paper says that the 27-nation bloc should have strict legal requirements for “high-risk” uses of the technology.

What’s high-risk? Any scenario with “a risk of injury, death or significant material or immaterial damage; that produce effects that cannot reasonably be avoided by individuals or legal entities,” especially in sectors such as healthcare, transportation, energy and government.

The week before the white paper’s release, Margrethe Vestager, who is best known as Europe’s tough anti-trust cop but whose remit now extends to both policing—and promoting—Europe’s digital economy, told The New York Times that she wasn’t interested in policing the algorithms that recommend Spotify tracks or Netflix movies. She was concerned about A.I. that determines who gets a loan or what diseases are diagnosed.

That all sounds reasonable. But in practice, lawmakers are likely to find it much more difficult to draw nice, fine-tipped Montblanc circles around high- and low-risk uses of A.I.

Geoff Hinton, the deep-learning pioneer who is an A.I. researcher at Google and a professor at the University of Toronto, highlighted one potential problem. In a viral tweet in response to the new white paper, he asked: “Suppose you have cancer and you have to choose between a black box A.I. surgeon that cannot explain how it works but has a 90% cure rate and a human surgeon with an 80% cure rate. Do you want the A.I. surgeon to be illegal?”

To be sure, the white paper does not say all A.I. in high-risk areas must be explainable. (Explainable A.I. is a fraught area, in which one always has to ask: Explainable to whom? To the software developer? To the doctor? To the patient?) But it does talk about the need to provide clear information about an A.I. system’s capabilities and limitations, and the need for human oversight.
Nick Cammarata, a researcher at OpenAI who works on explainability issues, tweeted a good retort to Hinton: “I’d take the 90% only if I knew the distribution it was trained on is very similar to me.” Otherwise, he’d take the human surgeon.

Some “low-risk” areas may present problems too. For instance, Vestager said she wasn’t worried about most recommendation engines. But what about targeted advertising, which seems like it might be a fairly similar use case?

Targeted advertising is combining with dynamic pricing in pernicious ways that may be difficult to police. For instance, a loan approval system would qualify as high-risk under the EU framework. One that discriminates against people with certain last names or who live in certain areas—a practice known as “digital redlining”—would likely be illegal.

But another way to accomplish the same thing, while possibly evading the “high-risk” label, is to simply never show a subset of people the ads for particular financial products. If people don’t know that a loan exists with favorable interest rates, they are much less likely to apply for one.

***

Speaking of digital redlining, I wanted to share the thoughts of a few readers who wrote in response to my newsletter on Lemonade CEO Daniel Schreiber‘s ideas for regulating the use of A.I. in insurance underwriting.