AI Is Already Predicting Who Will Become a Criminal, and That's Not Good News | In AI, the Key Variable Is Still People ①

You see, it doesn't just play chess

"Thirty seconds left." As the futuristic cop played by Tom Cruise jumps down from the helicopter, a colleague's last reminder sounds in his ear.

At that moment, a man in his bedroom picks up the scissors from his nightstand and faces his unfaithful wife. Before he can do anything, a squad of police bursts into the room and pins him down.

"Look at me, look at me." Tom Cruise scans the man's eyes with an iris reader. "That's you, Howard Marks. I'm placing you under arrest for the future murder of your wife, which you were going to commit today, April 22, at 8:04."

"But I haven't done anything!" the man protests, but the police have already cuffed his wrists. The city's murder count stays at zero.

This is Steven Spielberg's 2002 science-fiction film Minority Report, in which the police can predict the details of a crime, arrive ahead of time, and stop it.

No one today can predict the exact details of a crime that hasn't happened. But predicting where a crime will happen, and who is more likely to commit one, is already being used in law enforcement — and in sentencing.

AI Is Already Deciding Where Police in Dozens of Cities Patrol, and Who They "Talk To"

KeyCrime is an Italian prediction-software company. It mainly serves the Milan police, making area-by-area forecasts of robberies and thefts aimed at shops; the system has been running for nine years.

Founder Mario Venturi told QDaily that plenty of experience and data back the logic: criminals follow a pattern. If they rob a place in a given area and get away with it, they tend to strike again near the same spot.

This March, guided by KeyCrime, the Milan police managed to "prevent a crime" — outside a supermarket, they caught two men just as they were getting ready to rob it.

KeyCrime pulls the police force's data on criminal suspects and combines it with the locations of the robbed shops and the movements and weapons of the suspect caught on camera. From that it works out how dangerous the man is and — more importantly — whether he is connected to other crimes nearby, and if he is, when and where his next one might come.

In the past few years, with large amounts of data and neural-network learning algorithms, linking separate crimes has become much easier for KeyCrime.

Plenty of police departments already use crime-prediction software like this.

This May, Chicago police chief Jonathan Lewin held a briefing on artificial intelligence. The department has installed acoustic sensors around the city that pick up gunshots and collect the data; combined with data from roadside cameras, machine-learning algorithms produced a "crime prediction system" that can forecast where robberies and shootings will happen, and even which people might commit a crime — so that officers can go and have a chat with a suspect in advance.

Chicago officers on patrol now use an app on their phones and tablets: the map shows small red squares, and that is where the next crime may happen.

Police have been using technology to analyze and predict where crimes happen for many years, and the reason is easy to see: efficiency.

The well-known criminologist Lawrence W. Sherman once set out "eight principles for reducing crime":

• 1. More officers;
• 1. Faster 911 response;
• 1. Shorter time from call to dispatch;
• 1. More random patrols;
• 1. More targeted patrols of crime hotspots;
• 1. More arrests;
• 1. More contact with suspects who have prior records;
• 1. More contact with the community;

Quite a few of these come down to how to send officers to a particular place. In practice, a local police department has only so many people and so many hours. What algorithms and machine learning do here is make the department more efficient. If an officer has only eight hours of patrol time a day, wandering around in the hope of running into a criminal is not an efficient way to spend it.

Los Angeles Police Chief Charlie Beck says much the same thing: "I'm not getting more money. I'm not getting more cops. I have to work with what I have — and if patrol officers can change how they see this technology, so will the commanders." Los Angeles says that a single precinct saves $2 million a year with this efficiency model.

A number of technology companies are helping out behind the scenes. All the crime-prediction software we have been describing at the Los Angeles and Chicago police departments comes from PredPol, a policing-software company in Santa Cruz, California.

The Chicago Police Department says that within a few months of adopting PredPol's algorithm, shootings and similar crimes were already down 13%, and it expects the number of murders to fall by 49%.

Chicago in fact started using the PredPol crime-analysis system back in 2009, and KeyCrime, which has proven its worth, has been in use by the Milan police for nine years.

Only in recent years, with more data available, have police shifted the focus of their development work — from analyzing and predicting "locations" to the upgrade above it, predicting "suspects."

The most contentious example last year was the Chicago Police Department's "predictive crime hotlist," which tells officers the names and photos of the 20 people in a neighborhood most likely to commit a crime, down to statements as specific as this: "This person has a 25% chance of being involved in a violent incident in the next 18 months."

According to the department's own disclosures, the list already holds 400 people. Being on it does not mean you have a criminal history: living in a high-crime area, or having friends or family who have committed crimes — these are all factors the algorithm behind the list weighs.

Police also prepare these algorithm-selected "future criminals" psychologically, in advance. Jonathan Lewin calls this early-intervention mechanism a "custom notification" — going to talk to the suspect before anything happens:

"We've noticed you. We want to get you out of the cycle of crime, and here are the social service programs we've lined up for you." What those "social service programs" are — anything from cleaning up the neighborhood and volunteering to job training — and whether the suspect alone has to do it or the whole family has to move, is never spelled out.

If the suspect refuses and wants to stay in the high-crime area the algorithm has mapped out, the police issue a warning: "If you commit a crime in this area, you may be punished more severely."

AI Has a Hand in Sentencing, Too

In 2013, Eric Loomis was sentenced to six years in prison by a Wisconsin court for stealing a car.

The car he stole had been used in a shooting, and there was a gun in the trunk. The police thought at first they had caught him red-handed, but it turned out to be a mistake: Loomis had not taken part in the gun crime; he had simply stolen a car with a criminal history. For car theft alone, Wisconsin law allows at most three years in prison followed by three years of supervision.

The court gave Loomis a heavier sentence anyway. Its sentencing drew on an artificial-intelligence algorithm called COMPAS. This ten-point "scoring" system is used by the U.S. Department of Justice to judge the odds that someone with a criminal record will offend again.

Before the car theft, Loomis had been in prison for a sex crime. During that earlier term, the COMPAS algorithm gave him a high score. This time, the Wisconsin court used that score as a reference figure.

Loomis appealed, demanding that the mechanism by which the algorithm scored him be made public; he believed the scoring rules included factors such as gender.

The court turned him down. It held that scoring and analyzing criminals is something that has gone on since 1923, and that the algorithm is the intellectual property of its developer, Northpointe, and therefore cannot be disclosed. It also held that Loomis had been judged by other means besides the algorithm, so the process was fair. On May 23 this year, the court rejected the appeal and upheld the sentence.

If you commit a crime and go to prison in the United States, the COMPAS algorithm will most likely take over your personal data. An inmate fills out a questionnaire about his circumstances; adding in the severity of the crime, COMPAS calculates a "future criminal" score. To avoid discrimination, the questionnaire asks nothing about sensitive dimensions such as race or income, but it does include questions like these:

• Where do you live?
• What is your level of education?
• Had you ever traveled by plane before middle school?
• Have any of your relatives or friends ever been involved in crime?

Last year the nonprofit investigative outlet ProPublica obtained 18,000 COMPAS scores from police and published an investigation that followed those people's reoffending records over two years.

The results:

• Poor people are more likely to commit crimes;
• People with less education are more likely to commit crimes;
• Black people are more likely to reoffend;
• Men are more likely to commit crimes.

As statistics, these may all be facts. The trouble starts when such data is turned around and used to judge how likely one individual is to commit a future crime, and to set his sentence.

That is no different in kind from the Meituan HR employee fired not long ago for writing "no applicants from the Yellow River flood region or the Northeast" into a job posting.

In the United States, COMPAS has been used by judicial bodies nationwide since the early 2000s, and the algorithm is now in its fourth version. Judges in various states use COMPAS scores as a reference when sentencing, and so do police when questioning suspects.

In 2014, then Attorney General Eric Holder said publicly that it is risky for American courts to rely on algorithms to determine guilt and predict reoffending. American judges, Holder said, lean too heavily on this scoring algorithm when they sentence.

An Algorithm That Looks Fair Still Carries Its Designer's Bias

There is a metaphor about scientific research: studying something new, scientists tend to over-analyze the information they already have and force conclusions out of it, and so overlook the unknown factors.

A man loses his keys in an outdoor parking lot, and the first place he looks is the pavement under the streetlight.

Not because the keys are more likely to be there, but because that is where it is easier to look.

An AI algorithm judging how likely a person is to commit a crime works much the same way.

MIT professor Ethem Alpaydin once explained the principle behind the newest machine learning like this:

In the past you needed to know what a particular set of data was meant to achieve, and then you hired a programmer to write the program. With machine learning, the computer now learns for itself how to process and recognize the data, writes the program itself, and produces the result you need.

To build a system that decides who is a criminal, the first step belongs to the designer of the machine-learning algorithm, who decides "what kind of person is more likely to commit a crime?" and then breaks that into separate factors and goes out to collect data.

There are thousands upon thousands of reasons a person might commit a crime, and the dimensions the designer feeds in are the pavement under the streetlight. The computer is forced to judge a person's odds of criminality inside the dimensions the designer thinks matter.

Worse, today's machine-learning algorithms are essentially black boxes — you put data in and wait for a result. How the machine recognizes anything in between, not even the designer of the algorithm can say for sure.

Last month, a paper by Shanghai Jiao Tong University professor Xiaolin Wu and his doctoral student Xi Zhang drew controversy. Their paper, “Automated Inference on Criminality Using Face Images”, used machine-learning algorithms and image recognition to scan 1,856 ID photos of adult Chinese men and had the algorithm judge whether each man was a criminal, claiming a success rate of 90%.

Wu and Zhang also summarized the facial traits of these "criminals": the distance between the inner corners of the eyes was 5.6% shorter than in ordinary people; the curvature of the upper lip was different; the nose-lip angle was 19.6% smaller than in non-criminals; and criminals' facial features were more pronounced than ordinary people's.

The paper drew accusations of racial discrimination as soon as it appeared. In early May, three researchers from Google and Princeton University wrote a rebuttal titled “Physiognomy's New Clothes”. In it they argue that Wu's method resembles the Italian "medical physiognomy" of 150 years ago, only dressed in machine-learning algorithms:

"In 1870 the Italian physician Cesare Lombroso opened the skull of an Italian criminal named Villella and found a distinct depression at the back of the skull, placed as it is in lower animals. The discovery gave him his idea, and from it he proposed the theory of the 'born criminal': that criminals differ physically from non-criminals, and that the difference can be found by measuring with instruments such as calipers. Lombroso also held that the criminal is a form of atavism, carrying many of the traits of primitive, lower man, and that these traits can be inherited."

Blaise Aguera y Arcas, the Google researcher among the three, explains that a neural-network algorithm "judges" an image quite unlike a human being: it puts different weights on millions of learned parameters. Ask an algorithm what era a photograph comes from, for instance, and machine learning is likely to have picked up all sorts of small clues, from low-level ones like "film grain and color gamut" to clothes and hairstyles, even car models and typefaces.

So what pattern did the machine learning actually take from those 1,000-odd ID photos?

Not necessarily any distinctive feature in criminals' faces. It could be the grain of the photograph, or some other shared trait — shirts, for example. In Wu's paper, all three of the "non-criminal" images show men in white-collar shirts, while none of the three classified as "criminals" do. Only three examples, of course; as Aguera y Arcas wrote, "we don't know whether this is representative of the whole dataset."

And while none of us knows how machine learning makes its judgments, we humans, as the designers of the algorithm, supply the "assumptions" and the data — and those may well decide which way the algorithm goes.

Alexander T. Todorov, a professor in Princeton's law department and the third author of "Physiognomy's New Clothes," told QDaily: "Some people think our piece is an attack on Wu, but that wasn't the intent. What we wanted to show is that the result isn't as 'objective' as they presented it. The assumptions Wu made were untested."

The same thing happened back in the 1980s — yes, the machine learning every technology company is hyping today already existed then.

At the time, the U.S. military wanted an algorithm that would let a computer tell American and Soviet tanks apart in photographs.

After a stretch of training, their algorithm was recognizing them quite accurately.

But the engineers later found this was not because the computer had learned the design differences between the two countries' tanks.

What the computer had actually decided was that a higher-resolution photograph meant "American tank."

That was because the Cold War was still on, and the photographs of Russian tanks fed into the database were blurrier.

"New technology has the power to do good and to do harm. If your assumptions haven't been carefully examined, machine-learning technology will only bring more unfairness and speed up the inequality we already have." Todorov ended our interview with that line.

And if you think this is a problem only American police have, take a look at an earlier report from Bloomberg.

China Electronics Technology Group, a technology company managed directly by the government, is also developing software that assembles data on citizens' behavior — work, interests, spending habits and so on — in order to predict a criminal's actions before a terrorist attack happens. Wu Manqing, chief engineer at the military contractor, said: "Checking after a terrorist act is important, but predicting it is more important."

The report is very vague about this new system. Where exactly the data will be collected from, who writes the algorithm, whether it is already running — the public will have a hard time finding out.

Algorithms and AI are only a digitized version of problems that human judicial institutions already have. They raise efficiency, but they may also do a better job of absolving ordinary law enforcers of responsibility for their biases about regions, groups and races — because the choice was made by an objective machine-learning algorithm.

Crime-Prediction Algorithms Reach Into Everyday Life, Too

"Can you really trust the people you hire?"

This May, at a fintech startup event in Sunnyvale, California, Husayn Kassai opened with that question and went on to introduce his company, Onfido.

Kassai used images on stage to show how an algorithm can help a company decide whether the person it is hiring is "reliable."

The method is a bit like the way Didi drivers upload a photo of themselves holding their ID card: you have the person hold up a document and take a picture.

Onfido then scans the employee's photo, runs facial recognition and a background check, and within tens of seconds can confirm whether the person is who he says he is, whether he has a criminal record, and whether he is an undocumented immigrant.

Onfido also started out doing business with Uber, but as Uber's growth slowed from last year, Kassai began turning the company around — from checking whether sharing-economy customers had criminal records to running background checks on loan applicants for banks. "We're really more of a RegTech company now, because the data takes a certain amount of cooperation with government."

Onfido raised close to $30 million in one go last year, its newest investors including Sequoia and Salesforce. Its rival Checkr has bigger names behind it — Google Ventures, YC, Accel — and raised $50 million last year.

Onfido's website offers no clear explanation of where the data goes after the identification is done. What employers do with the information once they have it is outside the company's concern as well.

I asked Kassai on the spot about data-privacy protection and how they got their first batch of data, and his answer was vague: working with institutions that already hold large amounts of data, such as banks and governments, is the fastest way to train the algorithm, and they also want to help ordinary people clear the checks faster and get the loans and jobs they want.

Beyond that, he would not say another word.

You can see the same caution at many companies selling crime-analysis products.

Northpointe, the company behind the COMPAS algorithm that scores criminals, took on a new brand, Equivant, and went back to selling its services after being written about by The New York Times, ProPublica and other outlets.

This May, a venture conference called SVOD was held in Mountain View, California.

Andrew Ng, the artificial-intelligence expert and former head of Baidu's AI lab who had left the job not long before, said from the stage: "AI is the electricity of the 21st century. The companies that seize this new technology will run faster in the next round of competition."

One startup after another walked on stage to present its project — facial-recognition solutions for businesses, voice-recognition systems. The investors below threw out question after question.

"How accurate is your recognition rate?"

"How are you thinking about your exit?"

……

Fairness and privacy are not necessarily on their list of concerns. After the startups finished their presentations, Ivan Crewkov, an entrepreneur building an AI assistant made for children, talked with me about the future of artificial intelligence. I asked him: if a machine-learning algorithm knows who a child is from an early age, his strengths and his weaknesses, doesn't that worry him?

"I'm an optimist," he said. "Even with more regulation, I think business is going to move forward."

After the conference, Andrew Ng told QDaily what he thinks about regulating artificial intelligence — a view typical of the industry: "I think it's too early. The technology hasn't fully developed yet, and too many limits would hold it back."

Cover image: RoboCop, Person of Interest