Microsoft's Chatbot Xiaoice Has Just Learned to Talk. We Met the Team Behind Her Voice

She speaks aloud, replies on WeChat and is going global: from version 3.0 on, Xiaoice is no longer just a side project for Microsoft

"Meow, take out the battery. Don't be angry, okay? It'll make you old."

"Japan's security authorities will carefully select bodyguards from the Security Police unit of the Metropolitan Police Department to protect Fukuda at close range."

Last Monday, in a recording studio on South Zhongguancun Street in Beijing, a long-haired young woman in a white T-shirt and a black skirt leaned back in her chair and spoke these unrelated sentences into the microphone fixed in front of her.

"I love Sagittarius, but Sagittarius doesn't love me... mm."

"Everything before that was good. Don't drag out that last 'mm.'" Lian Jie, recording director of Microsoft's Xiaoice voice team, pressed the intercom button in the control room on the other side of the wall.

Li Meng answered with a cheerful "Okay!", steadied her breathing and read the sentence again. (Li Meng is a pseudonym, at Microsoft's request.)

They had been working like this for two months to get ready for the release of Xiaoice 3.0 on August 20. At the launch event this afternoon, Microsoft's artificial intelligence, called "Xiaoice," talked to people by voice in public for the first time. She is also back on WeChat.

Xiaoice has drawn wide attention through a string of episodes: Zhang Zetian endorsed her, WeChat banned her, and she worked as a customer service agent for JD.com. Until today, though, she had talked to her 37 million users only in text.

How Xiaoice began to speak

Li Meng is a senior majoring in broadcasting and hosting at a Beijing university. To give Xiaoice a voice, she has come to the studio three times a week since June, working from ten in the morning until five or six in the afternoon. On her busiest day she recorded more than 700 sentences. Each sentence is saved as a WAV file and sent to Microsoft's engineers for processing.

Li Meng first heard about the project five months ago. She told QDaily that teachers in her department had asked students to forward a message in the class group chat: a company needed voice actors for an artificial intelligence project, and it wanted bright, lively young women.

She signed up with several classmates, and a teacher took them to a recording company to record audition samples. Li Meng, who hardly used Weibo, had not yet heard of Xiaoice.

More than a hundred people sent recordings to Microsoft, as she did. Xu Xiang, product manager of Microsoft's AI voice team, told QDaily that besides students in the field like Li Meng, Microsoft directly invited veteran voice actors and celebrities with large followings.

The celebrities were dropped first: their voices were too distinctive and would be recognized. Experienced voice actors tend to be older, and they found it hard to keep "putting on" the voice of a 17-year-old girl.

Being "17" makes Xiaoice an oddity among AIs. Because AI is still not very intelligent, tech companies that launch AI assistants tend to keep their persona vague. Apple's Siri, Amazon's Echo and Microsoft's other voice assistant, Cortana, have no stated age.

Xiaoice, by contrast, has called herself a "16-year-old girl" from the start. A Weibo post at the end of last year announcing that she had turned 17 drew more than 2,000 comments.

Lian Jie, the recording director, is in charge of shaping Xiaoice's voice. As she sees it, this 17-year-old girl "is afraid of nothing. She can act coy and play cute, and she can also put you in your place."

The technical team's requirements were far more detailed. In a PDF several pages long, Microsoft's product managers listed criteria such as "standard Mandarin," "no dialect" and "degree of warmth."

After two rounds of screening, five candidates were left for the third. At this stage, researchers and engineers from Microsoft's Internet Engineering Institute joined in alongside the subjective judges, using technical measurements to test whether each candidate's voice was stable.

"Take the first hundred sentences someone records and a hundred sentences after the three-hundredth. If you picked just anyone, those two sets could differ a lot," Xia Hairong, a researcher on the AI voice team, explained to QDaily. Only a stable voice can be processed by a computer and recombined into any sentence it needs.

Finally, the recording director and the engineers each scored the five candidates on a complex spreadsheet. Weighing subjective impressions against the technical data, they chose two students from the same school and the same major, numbers 102 and 143, to become Xiaoice's "voice."

While they were choosing the voice talent, the engineers also began building Xiaoice's corpus: the set of sentences they select for her voice talent to read, and the raw material from which her voice is made.

This is not the first time the Internet Engineering Institute has made a computer talk. As far back as Windows 2000, you could find the text-to-speech (TTS) function they developed in Beijing: one mouse click, and the computer read out all the text in a document. By today's standards, that voice is far too mechanical.

Cortana, the AI voice assistant Microsoft builds into all its phone and desktop operating systems worldwide, also learned to speak here.

Talking like a person

Unlike voice assistants such as Cortana and Siri, Xiaoice, Microsoft stresses, is a girl with emotions, able to reply in any register, from serious talk to anger, swearing and playing cute.

To make Xiaoice sound natural, Li Meng has to record many colloquial sentences and pieces of internet slang, even passages from web novels.

Xia Hairong explained where these sentences come from: "Xiaoice has been around for quite a while, so we have a lot of Xiaoice material we can use."

Because of privacy restrictions, Microsoft does not record what users say to Xiaoice, but it does record what Xiaoice says. Microsoft's engineers pulled out the sentences Xiaoice had used most over the past year or so, and from those, an algorithm chose the sentences that cover the widest range of sounds for Li Meng to read.

To make the voice truly natural, Li Meng will eventually have to read tens of thousands of these slightly baffling sentences.

Although Microsoft has stressed from the start that Xiaoice is an AI with emotions, Li Meng cannot overact when she voices her.

"When voice talents first get this kind of script, they are often very excited and know just how to perform it," Xia Hairong told QDaily. "But when we get the recordings back, we find we can't process them, because they are too different from the other texts. We can't (have the machine) use one set of standards to process completely different styles."

The result depends on several parties working together. In the studio, Lian Jie has to make sure Li Meng speaks more slowly than normal, pronounces every character correctly and to standard, and keeps her voice steady and consistent.

Emotion poses a challenge not only in recording but also in how the voice sounds when it finally speaks.

"When people talk, they can never say the same sentence the same way twice. Reproducing that is very difficult," said Wang Peng, another engineer who works on text-to-speech. Yet when they design the voice model, they need to reduce the complexity of the information. That is why, when a machine says the same sentence, the first time and the ten-thousandth sound identical to the human ear.

That sameness makes people feel, without knowing why, that the voice is unnatural.

Even back when Xiaoice talked to people only in text, she gave different answers to the same question. Now that she can speak, she will also say the same sentence in different ways, to sound more like a person.

"When we choose an answer, we don't just pick the single highest-scoring one; we pick several, ranked from high to low," Xia Hairong said, describing one approach. And when Xiaoice says the same sentence in different contexts, she chooses a different intonation depending on the mood that has been set. When she says she has been dumped, for example, she might sound teasing, sincere or consoling.

Not everything is ready, though. August 20 brings only a limited test, and the talking Xiaoice still does not sound natural enough. For now, her speech is built from a few thousand sentences recorded earlier; a more complete voice function will go live in a month.

This is Xiaoice's first step in speaking, and a stage in the slow work of perfecting AI speech. The first thing she learned was to pronounce each syllable and tone: syllables formed from Mandarin's initials and finals, combined with its four tones, make up more than 1,600 speech units.

In real speech, however, a character's tone shifts with its position in the sentence and the speaker's tone of voice, and the machine has to keep being adjusted to handle all of that.

"With 4,000 sentences, coverage is around 90%. From ten thousand sentences upward, you reach the splicing stage," Xia Hairong told QDaily. Beyond 90% coverage, each extra percentage point takes a great deal more recording.

So the recording will go on. The people who voiced Cortana, which launched last year, still go to the studio to record for her whenever they have time.

Once the corpus is large enough, the engineers use algorithms to stitch together speech segments cut into frames a few dozen milliseconds long, smoothing the transitions between syllables so the result sounds more natural.

Besides adding more recordings, the engineers have devised some technical shortcuts to make Xiaoice sound more human.

For certain special words, such as a dismissive "hehe" or fixed internet expressions, the engineers take complete speech units from the large recorded corpus and join them directly to the rest of the speech. Common words are handled the same way.

Suppose the database shows that Xiaoice says "I love you" to users thousands of times a day. The engineers will then have the voice talent record "I love you" as a whole and use that recording directly.

When shown a photo, Xiaoice uses image recognition technology that Microsoft continues to develop to identify the objects in it and offer her own comments.

Xiaoice as a platform

Most of the technology behind Xiaoice was not built by the AI voice team itself but drawn from Microsoft Research's stockpile. When a conversation mixes Chinese and English, for instance, Xiaoice speaks the English using Skype Translator's real-time translation technology.

Xiaoice has gradually become Microsoft's testing ground for new technology. Large amounts of experimental data flow back to the development team in real time, and developers can quickly revise the software based on the feedback and ship it to users in Xiaoice's weekly feature updates.

That is a world away from what Wang Peng experienced developing for Windows Phone: "Once, after we finished building something, we waited six months for the software update."

"Microsoft Research now has a dozen or so projects, all of them serving Xiaoice," said Qi Lu, the executive vice president in charge of all Microsoft's applications and services worldwide, at this afternoon's launch event.

The Xiaoice team has grown from an experimental project of 20-odd people a year ago to a team of 100, and it has a plan to expand worldwide. On August 7 this year, Xiaoice launched an AI called Rinna in Japan with Line. Because Xiaoice is built as a platform, moving it into Japanese took six months in total.

Released alongside Xiaoice 3.0 is a feature that embeds Xiaoice in WeChat official accounts. Her return to WeChat is part of a series of new partnerships, and next she will appear in the comment sections of NetEase News.

Born last April, Xiaoice has lived through headline-making episodes such as the celebrity gossip about her endorser Zhang Zetian and Liu Qiangdong, and the WeChat ban. She now works as a service inside products from Weibo, JD.com, Meitu, CooTek, China Eastern Airlines and other companies, offering their users everything from small talk to customer service. So far, 37 million people have left their phone numbers to claim a Xiaoice of their own.

In China, apart from pirated Windows, no Microsoft product reaches as many consumers as Xiaoice.

According to Microsoft's figures, each time users open Xiaoice they exchange an average of 18 messages with her, and even late at night more than 50,000 people are chatting with her at once.

It is hard to measure what any single use brings, whether small talk on Weibo, photo comments in Meitu Xiuxiu or customer service for China Eastern. But this rapid cycle of trials, data feedback and product improvement will be the foundation for helping AI find practical uses.

Apple and Google can build Siri and Google Now into their own mobile operating systems. Microsoft's phone business shows no sign of recovery, and the PC has become a pure productivity tool. Built into all kinds of internet products, Xiaoice is Microsoft's most promising bet in artificial intelligence, and even in search.