AI Is Being Used to "Custom-Tailor" Prices and Insurance, and Even the Appearance of Fairness Is Gone | AI: It Still Comes Down to People, Part 2

Every industry wants in on artificial intelligence. Along with the gains in efficiency come problems no one has faced before.

San Francisco, 9 a.m., June 1. More than 30 people sat in a circle in a room here, waiting for the AI Workshop, an event run by the tech news outlet GigaOM, to begin.

By the setting alone, this was a typical Silicon Valley event. The room was rented from Breather, a service like Airbnb but for stylishly furnished office space. Byron Reese, the futurist giving the opening talk, wore a casual shirt and jeans and stood in front of a huge projection screen. Two years ago Reese bought GigaOM, a publication well regarded for its coverage of enterprise technology but short of money.

"Everyone in this room could live to see artificial intelligence deliver all of this, across every industry: human immortality, the end of the hunger crisis, a huge drop in food prices, the end of poverty. Think about it, all these problems could be solved. In 120 years, this planet will be 1,000 times as productive as it is today."

For an hour and a half, Reese described a future in which artificial intelligence changes everything. He closed by borrowing a line from Marc Andreessen, the Netscape founder and well-known investor: "In 2011, Marc Andreessen said 'software is eating the world,' and he was right. Now it's artificial intelligence that's eating the world."

The audience applauded hard. Most wore suits; no one wore the T-shirts and hoodies favored by Silicon Valley engineers. Nearly all were middle-aged men, and apart from the QDaily reporter only one woman was in the room. Several had come from out of town, their suitcases standing against the wall.

Just as Byron Reese said, the audience came from all kinds of industries: real estate, law, machinery manufacturing. They were not technical people; most managed mid-size or large companies. Each had paid $995 and given up a working day for this beginner's course in artificial intelligence.

The session was built for them. Christopher Mohritz, the instructor, took the microphone and spoke for four hours on what artificial intelligence can actually do today. He pulled out an Edison chip and a small light bulb and built a voice-controlled lighting system on the spot.

Mohritz's talk ran like a proper introductory course in artificial intelligence: what the concept means, how it relates to data analysis, where the field is heading now.

Questions were allowed at any point, and once Mohritz got to how to choose an AI platform, they came faster.

"I'm in real estate. Which platforms would you recommend? Some of our data can't be stored in the cloud."

"Should I go and hire a team of data scientists?"

Instructor and audience alike clearly cared about one thing above all: how a company can use AI to speed up its business. Mohritz repeated it through the talk like a sermon, "AI accelerates your software, and your software accelerates your business."

This was only their first session of the year. After it, they planned to tour the United States, telling more business owners how to join the AI wave; June alone had three workshops booked. "The AI ship is leaving. Are you getting on board?"

The tech companies' ship has already sailed some distance. Now a much bigger one is putting out to sea too. But it may not be heading where you would like.

Artificial intelligence has been deciding your digital life for a while now

For an internet company, artificial intelligence is already everywhere. The only difference between one company and the next is how deep it goes.

"We're really competing with sleep. It's not Hulu, and it's not Amazon." That is how Netflix CEO Hastings, whose company made House of Cards and Daredevil, talked about competition on an earnings call this April.

What is really competing with your sleep is the algorithm that picks what to show you.

The world's largest video site, worth $66 billion, is currently worth more than Baidu. But Netflix is not satisfied with the share of your time it already holds. It would like people to stay up late watching its shows.

To keep more users on the platform, Netflix makes full use of your data: what you have watched, what you have searched for, when you watch, what device you watch on. All of it ends up training an algorithm to recommend the videos you are most likely to click and play.

"Ranking matters enormously. The first video recommended is the one most likely to be clicked and played," Netflix product designer Gomez-Uribe said in explaining the system.

Even if you never use Netflix, the logic will be familiar.

Take Toutiao.

Like Netflix, its news is picked by an algorithm. Toutiao recommends stories using a reader's past interests, location and other data. If a story is read by a larger share of the users it is shown to, its chances of being shown to similar users go up.

These methods do not look complicated. But as the services grow, the companies behind them get hold of more and more data, and the algorithms no longer have to be written by hand. Using the user data a service accumulates, machine learning can refine the original algorithm.

Amazon prices the goods on its platform with an algorithm, so the price you find when you search for a product there is always a few cents below the competition's. And you do not have to go anywhere: browse or buy on Taobao, and Alibaba collects that record and eventually uses it to place ads in front of you on Weibo and on portal sites.

Through the internet you get all kinds of services. In the end, the data those services pile up lets companies know who you are better than they used to.

But just as Netflix competes for your sleep, when a tech company uses AI to recommend things to you, its first concern is how to hold you a few minutes longer: one more video played, one more clip watched, one more feel-good story read.

As former Google product designer Tristan Harris put it, "Never before in history have a dozen or so people at a dozen or so companies had this much power to decide what billions of people think."

Harris did not mean it as praise. In a Recode interview this February he gave an example: at a design meeting for Google's Inbox email app, when the talk turned to whether new mail should trigger a push notification, most of the engineers and designers in the room said without thinking, "Of course. Why not?"

What most product designers chase are the numbers tied to engagement, things like active users and time spent.

Which explains why Harris's colleagues agreed so readily about the email alert.

But Harris thought the design was not necessary. A user does not need to spend that much time checking mail; from the user's side, the alerts waste too much of it. Users do not need video sites feeding them one clip after another either.

So after leaving Google he set up a nonprofit, Time Well Spent, to remind people using today's internet services to keep the mobile internet from eating all their attention. Not long ago he even suggested setting your phone screen to grayscale, which he said would cut how often people check it and reduce their dependence on it.

Harris is plainly not the mainstream view inside tech companies, which is why The Atlantic called him "the closest thing Silicon Valley has to a conscience".

With AI's "help," some people are already paying more for the same service

Zhang Yiming, Toutiao's CEO, has said repeatedly in interviews that technology is neutral.

But the metrics that make technology's decisions are not neutral. They are set by the people who choose them, and for most products the technology runs after revenue and engagement.

As the American economist Milton Friedman famously put it, "The only social responsibility of business is to increase its profits."

"The question isn't only who is using this new technology, but who is developing it," Wu Jing, a professor at Peking University's School of Journalism and Communication, told QDaily when we asked her about the harm technologies like AI can do to society.

Applying AI has made things faster. But the companies pushing it care about more than serving users. They want not just more of your time but the ability to adjust a price automatically according to who the buyer is, so that some people pay more than others for the same service.

Uber is doing exactly that.

Not long ago, after drivers complained repeatedly that the fare they were paid fell far short of what passengers paid, Uber offered a new explanation: it had built a new way of pricing. Unlike the traditional method, which sets a fare from distance, duration and similar factors, the new algorithm uses AI to estimate the most a passenger would be willing to pay for a given trip, and so maximizes Uber's take.

That means that for two trips of the same length, a passenger riding from a wealthy neighborhood to an upscale shopping district pays more than one heading to a poorer area. This is not robbing the poor to pay the rich: the driver's share does not change, and all the extra money goes to Uber.

Uber's head of product, Daniel Graf, said the price takes into account things like what users will accept and the trip itself. But he did not explain how it is calculated. How the algorithm actually works remains a black box.

And price insensitivity is not only a matter of a customer's income. Often it has to do with how urgent the situation is.

People riding to a hospital tend to care less about the fare. Does that mean they should be charged more?

When a person makes that call, there is usually some hesitation. An AI adjusting itself to preset parameters does not weigh much of that. Which is why, on several earlier occasions when disaster struck, natural or man-made, Uber's prices rose automatically, an adjustment to the changed ratio of supply and demand. Afterwards Uber's operations team would usually step in by hand and cancel the increase.

Uber clearly is not out to profiteer from disasters. But today's AI is not clever enough: when its goal is more profit, it still cannot learn to see that charging more is sometimes not acceptable, even though any normally educated five-year-old understands this. In the end Uber relies on people stepping in to keep it from happening.

But once AI starts setting the price of every single ride according to each passenger's situation, that kind of human intervention may no longer be possible.

The hospital is only one example; there are many more situations no one can foresee. An out-of-towner hailing a car does not know the going rate and is naturally less sensitive to price. Plenty of taxi drivers used to take such passengers the long way round and charge them more.

Can AI now simply raise the price to charge out-of-towners more? No engineer has to write that feature. The AI can find the pattern by itself.

When outsiders do not know what the algorithm does, who decides whether the fare is fair?

Unfair pricing has moved beyond tech companies' products and into more industries

"The internet is turning into an equation for predicting the future," said Patrick Tucker, a technology reporter at the American outlet Defense One, speaking about what AI is doing to the world today. "Computer-assisted prediction is already everywhere, from potential flu outbreaks to stock market swings. Over the next 20 years, the predictive power of machine learning and big data will affect far wider areas, including consumer choice, health insurance and educational opportunity."

There is no need to wait 20 years. AI's effect has already begun.

Uber is not the only one using AI to charge consumers different prices. Traditional oil companies took it up quickly too.

Before AI came in, gas stations belonging to different companies fought for market share by undercutting each other on price. Applying AI has begun to change that rule.

This May a gas station in Rotterdam, the Netherlands, raised its gasoline price by 3.5 cents. A competitor three miles away then raised its price by about the same amount. The average price of gasoline in the Netherlands is around $1.70, which makes that increase far from trivial.

Both stations that raised prices use a dynamic pricing algorithm from the same AI company, a2i Systems.

The algorithm runs the price data dozens of times a day. Besides analyzing past prices and rivals' prices, it also models how much people in the area are willing to pay before producing a final number.

After the first station raised its price, the second judged that residents nearby could afford it, and raised its own price too.

"It's not about taking more money from consumers as a whole. It's about charging more to people who aren't price-sensitive and less to those who are," said Ulrik Blichfeldt, CEO of a2i Systems, which supplies the algorithm to this chain of stations.

What it guarantees, in the end, is still the station's revenue.

Alireza Derakhshan, who designed the algorithm, told an AI conference in Seville, Spain, last year that the system had raised clients' profit margins by an average of 5%.

When several companies raise prices together, they are in effect forming a price cartel, a monopolistic practice governments regulate strictly.

When companies use AI to steer their prices, responsibility blurs. But the logic underneath has not changed. People forming a price cartel to protect profits and a program forming a price cartel to protect profits are no different.

Derakhshan gave the conference another example, this one from retail. A client using their dynamic pricing algorithm once complained to him that the system had gone wrong: every retailer nearby was cutting prices, and the system was advising a price rise.

Not a malfunction, Derakhshan explained. From its estimate of what people nearby would pay and its reading of competitors' prices, the system had predicted the store would get more customers, because they would not want to queue at the checkout just to save a little money.

And that is what happened. "The price went up, but the customers kept coming," Derakhshan said at the conference.

Changing prices in a physical store is still cumbersome; there is a price tag everyone can see. But if Taobao, JD.com or Amazon raised your price by a few percent based on your own spending habits, would you notice?

How AI "thinks" is opaque, and that opacity is now being used to decide employment and credit

It is not only prices. Credit rating and hiring are bringing in AI too. The advantages are obvious: it can weigh more factors, and it is faster.

But new problems are arriving with it.

ZestFinance, backed by JD.com and Baidu, uses AI to assess an individual's creditworthiness. From public data and data it buys, ZestFinance can produce a credit rating for a borrower.

Its founder, Douglas Merrill, started the company in 2009 with the line, "All data is credit data."

In 2015 ZestFinance launched Basix, a lending product for small businesses and individuals that estimates each applicant's ability to repay using a great deal of information that has nothing to do with financial data. Several internet companies now use the service, JD.com and Baidu among them.

Basix analyzes as many as tens of thousands of data points per applicant: whether they pay their phone bill in advance, whether they move house often, whether they make capitalization mistakes while filling in the loan application online, how long they spend reading it. All of it counts toward the applicant's credit rating.

ZestFinance says its data models found that applicants who fill in the form entirely in capital letters carry a higher credit risk. So how you capitalize text on a form has become one of the things that affect your credit.

Gild, another company, which helps tech firms assess programmers, judges a programmer's ability partly by whether they often visit a particular Japanese anime site, because many high-caliber programmers go there and the AI concluded the two were related.

This is no different from the much-discussed case of predicting criminals from photographs, where the AI looked at a set of pictures and came up with a rule: anyone not wearing a white-collar dress shirt is a criminal. The AI had only so much to look at, so it forced a pattern out of it.

Or a more recent example: a Meituan recruiter said the company would not hire people from the "Yellow River flood area." He was fired afterwards. But when an AI arrives at a similar conclusion on its own, who is responsible? Would anyone even know?

One basic reason for this is that the way most AI reaches a judgment today is a black box. Data goes in, a result comes out, and the designers adjust the parameters according to the result.

What AI mostly does is turn extremely complicated individuals into a series of measurable numbers, then judge the person by those numbers.

During the computation it is not only outsiders who are in the dark; even the designers can hardly say how the algorithm reached its conclusion. It is like AlphaGo's designers not understanding where each move came from, or the builders of an exam-taking robot being unable to follow their own machine's mathematical reasoning.

"In fact many algorithms are a black box, which makes it impossible for regulators and the public to see inside. That's a problem, because algorithms keep getting more capable, and right now we lack any form of oversight over them," Ben Wagner, a senior researcher at the Center for Internet and Human Rights, a European academic body that studies the relationship between technology and society, told QDaily.

Wagner does not think the people writing these algorithms mean any harm. "At least the tech startups I've dealt with are well aware of the problems the black box creates," he said. "They just don't know how to solve them."

Meanwhile, more sensitive data is being opened up to artificial intelligence

This March, the US House Committee on Education and the Workforce passed HR1313, a bill on employee health information introduced by the Republican congresswoman Virginia Foxx. It would let employers obtain health and medical information about their employees and their employees' families.

Under the bill, an employee may volunteer for a wellness program run by the employer, take a genetic test and let the employer use the genetic information to assess health risk. Employees who take part get a subsidy of 30% to 50% on their health insurance.

Supporters say it saves employees money on medical costs and offers a route to a healthier life.

More than 70 organizations, among them the American Academy of Pediatrics and AARP, signed a joint letter against the bill. Derek Scholes, director of science policy at the American Society of Human Genetics, said the policy in effect forces employees into the program and out of their genetic privacy, because an employee who refuses to hand over health data pays $5,443 more a year in insurance than someone who hands it over.

In most countries, China included, an employee's medical exam results are kept from the company. Once genetic data is tied to AI and used to judge how likely a person is to fall ill, the risk is far greater than a leak of hepatitis B status.

If insurers use genetic data to judge the chance of future illness and set premiums accordingly, people with genetic defects will have to pay more, even though nothing has actually happened to them and all that has changed is a "probability" a few percentage points higher.

For people without money, the cost of treatment can decide whether they live. And the reality, as things are going, is that genes may well decide the cost of treatment.

"Social management today atomizes every member of society completely, treats every citizen as a problem, and becomes the solving of one problem after another. It is an intensifying process. Since the late 19th century and the 1920s, and especially as globalization and mobility have grown stronger, this tendency has become more and more obvious, because it no longer treats the individual as a member of a community but as a separate, isolated unit. Social management, or social governance, is no longer aimed at social solidarity or the development of social culture; it has become a means of reducing social unrest through digital tools," Professor Wu Jing said of the trend toward applying algorithms to social management.

Is it too late to talk about regulation?

Andrew Tutt, a Yale scholar, has designed a "Federal Agency for Algorithms" (An FDA for Algorithms) modeled on the US Food and Drug Administration. Algorithms already reach into every corner of people's lives, he argues, so a system like this ought to exist for them too.

Today the FDA, or China's CFDA, decides which drugs may be used in the clinic, which ingredients may not go into food, which pesticides may be used on farms.

Tutt borrows a similar structure for his design, and his regulatory scheme likewise covers how transparent an algorithm is, how safe it is in testing, and how responsibility is assigned.

The FDA has existed in the United States for 111 years. Like AI today, it came into being because of a new technology; at the time, that technology was synthetic drugs.

Chemical drugs developed gradually through the 19th century, and morphine, opium tincture and the like were all created in that period. But right up to the 20th century the United States had almost no laws regulating the food and drugs it produced at home.

In the late 19th century Harvey Wiley of the US Department of Agriculture's Division of Chemistry began investigating the abuse of drugs and food on the market, and between 1887 and 1902 published a series of research reports titled Foods and Food Adulterants. The division also worked with the General Federation of Women's Clubs, medical workers, drug manufacturers and other bodies to lobby for a uniform food and drug standards law.

In 1906 the American writer Upton Sinclair published the novel The Jungle, which described in detail the filthy canning process in Chicago's meatpacking plants and left the public angry. That June, then-President Theodore Roosevelt signed the Pure Food and Drug Act. The next year, the Agriculture Department's Bureau of Chemistry was reorganized into the Food, Drug, and Insecticide Administration; three years after that, it was renamed the FDA.

Nearly 30 years passed between the moment chemical drugs came under wide public doubt and the founding of the FDA.

Only, facing artificial intelligence, do we have another 30 years?

Cover image: a still from Westworld.