How Far Are We From the Future of Film?

At the International Advanced Imaging Conference, scientists and people from the industry discussed VR, AR, and what these technologies really come down to.

On December 1, the seventh Beijing International Advanced Imaging Conference and Exhibition opened at the Beijing Film Academy. Fifteen guests spoke at the forum: executives from visual-effects companies, research scientists, and working professionals such as cinematographers.

The event was co-hosted by the Beijing Film Academy and the China Society of Motion Picture and Television Engineers, and co-organized by the Future Visual Imaging Advanced Innovation Center and the society's Advanced Imaging Committee. Each speaker talked about his own field, under headings such as "high-quality image production," "advanced imaging workflows," "VR/AR content production" and "the future of imaging."

This year VR became a word everyone knows, and AR picked up plenty of attention too, thanks to the runaway success of Pokémon Go — a game nobody in China could play — and the "Japan's Eight Minutes" segment at the closing ceremony of the Rio Olympics. On the whole, though, these technologies still sit at some distance from ordinary consumers. They have barely touched daily life; mostly they have become concepts some companies use to talk up their market value.

Still, people in the industry agree almost unanimously that once VR and AR mature they will bring a new upheaval. And "advanced imaging" does not stop at these science-fiction-sounding technologies. The recently released Billy Lynn's Long Halftime Walk broke new ground by screening at 120fps, 4K and 3D, and behind that technology lie changes to shooting, to acting, to every stage of production. Once VR is widely used in film and similar settings, the filmmaking principles that today's filmmakers take for granted will have to be rewritten almost from scratch.

The guests came to China for the conference not only to talk shop with their peers, of course, but because they see a market here. John Dietz, a visual-effects supervisor who worked on the Harry Potter films — he also has a Chinese name, Zhang Dizhe — has been in China four years now. His talk was given over to explaining how Chinese films get made, and to inviting foreign companies to work with them.

How will technology change film? Nobody can answer that clearly today, but one thing is beyond doubt: technology has given filmmakers more choices. We may not see a VR film any time soon, but in the course of exploring VR, film has already been changed a great deal.

Will Billy Lynn Become a Milestone for VR Film?

While promoting Billy Lynn's Long Halftime Walk, Ang Lee was asked by a reporter what he made of VR. My VR — meaning the "top-spec" technology he had used — is better than theirs, Lee said. He has said more than once that the film was meant to give audiences a sense of immersion, and immersion is exactly one of VR's biggest selling points. Which is why some have called it "a milestone film in the 'VR-ification' of cinema".

High dynamic range, high brightness, high frame rate: these are the changes Ang Lee brought to film. Set against a true VR film, of course, Billy Lynn is still much closer to traditional cinema. Most important, the camera still fixes the audience's field of view; they cannot look freely at whatever they want to see. Jim Chabin, president of the International Virtual Reality Society, cited his own survey in his talk: "People think virtual reality is more exciting than the arrival of 3D and 4K."

Even so, Lee's new technology was enough to stir interest inside the industry. This is not just a change in how films are shot and projected; it also calls for a new set of cinematic aesthetics to match. With VR film still at the experimental stage, Billy Lynn doubles as a rehearsal for the future.

The cinematographer Shao Dan had just come back from shooting director Xu Haofeng's new film, Hidden Blade (《刀背藏身》), an action picture full of fight scenes — the kind of footage prone to motion blur and shake. "All we can do, when a character moves fast, is roughly hold his position in the frame, or even compromise by changing the shot size," he said. At 120fps, action footage gets far clearer. He thinks "this could be a new standard for film."

Still, Shao Dan finds Ang Lee's film "too clear." In the past, a director and cinematographer who wanted to show the detail of someone's face would naturally reach for a close-up — "with a close-up you can look all you like" — the director pulling you closer to the actor.

But Billy Lynn "may go beyond the image the human eye can take in." Every detail of an actor is there, down to the last hair, without the camera having to go looking for it. Shao Dan doesn't know whether that counts as a kind of "offense," or whether the principles of film really are changing. "In a lot of cases we change the thing or the person being filmed. We put makeup on the actor, we need to cover the flaws on his face. But now it's a different concept: we need everyone to see detail this startling. We can take it all the way to ten out of ten — but do we need to keep it there?"

Demetri Portelli, the 3D cinematographer on Billy Lynn, described some of the practical choices. "If we want technology to move forward we have to use it. But some people say, I just want to shoot 24fps, I don't want 120 — that's their choice too. When we see a beautiful actress, we may want to soften her face with 24fps, keep her less than sharp, emphasize her features. But sometimes we can work by contrast, and use 120fps to bring out her detail."

He also admitted that artists have to think hard about everything 120fps can do — a gory scene, for instance. A gory scene that lifelike may not belong on a screen at all. Or try this: if the orcs in The Lord of the Rings were rendered with this technology, would the film still be rated PG-13?

Makeup cannot be heavy, because under this technology audiences will feel the falseness it brings. Acting styles have to change too: if there is too much performance in a performance, you can see the actor acting. This is not the first time a technical advance has shaken up an aesthetic. Charlie Chaplin was a master of the silent era, but his method and style look overblown in the film era that came after.

Liu Gesan, head of the technology department at the Beijing Film Academy, poured cold water on the technology from another direction: "Pushing this film with technology didn't do much for the box office. Might there be other films that need this technology more, or that could put it to better use, so audiences enjoy watching more and feel something deeper? We're certainly hoping for a film like Avatar — science fiction, big production. This one is a bit more like an art film, which isn't quite the same thing."

Wang Chunshui, deputy director of the general office at the Future Visual Imaging Advanced Innovation Center, said new technology now arrives faster than anyone can take in: "The problem we face has changed. Being poor with nothing to eat used to be one kind of suffering. Now we suddenly find there's too much of everything, and for an artist the question becomes which thing you truly need, which thing you truly control. That may be the important question in front of us in the future."

Jim Chabin, for his part, thinks technology has to be proved out by the market: "When we talk about future trends, we need to know what future users are thinking, and then make our films to fit what they want."

Billy Lynn, which attempts a technical leap on the foundation of a traditional film, has already set off plenty of discussion and argument. So what will VR — the technology that might overturn film altogether — look like when it finally arrives?

In Virtual Reality, What Matters Is the Reality, Not the Virtual

Yu Jingyi, a professor at ShanghaiTech University, started a company in 1999. He and three friends from Caltech went out to shoot 360-degree panoramas of Las Vegas casinos — panoramic images to hand to tourists and pull them into the casinos. The venture failed, because a photograph is, after all, a fake thing. "In the course of doing virtual reality we've paid too much attention to the virtual part," he said. "We stressed the fake images that virtuality generates, and forgot that what matters more is the reality."

By Jim Chabin's account, consumers are drawn to VR because it "lets them go places they can't go, or do things they can't do in reality" — walk a tightrope between the Twin Towers, say, or have superpowers. Augmented reality works the same way. In a project run by Wang Yongtian, head of the School of Information and Electronics at the Beijing Institute of Technology, AR can help train pilots, and can let visitors see the Old Summer Palace as it stood before it was burned down.

The "reality" inside a film breaks down roughly into characters, settings and story.

While discussing what Billy Lynn's technology changes, someone suggested that to avoid the flaws a real person might show, you could simply build a character with CG. That technology is not out of reach at all: Hollywood motion capture on today's screens already passes for real. In Captain America: Civil War, released in the first half of this year, digital makeup gave Iron Man back his young face.

The biggest obstacle to it is the "uncanny valley."

Jim Foley, the "father of computer graphics," presented a body of his own research at the conference on what causes the uncanny valley and what follows from it.

The "uncanny valley" was proposed by the Japanese roboticist Masahiro Mori. Because robots resemble humans in looks and movement, people feel warmly toward them — up to a certain point, at which the reaction suddenly turns sharply negative. The smallest difference between a robot and a person becomes glaring; the whole robot looks stiff and frightening, as if you were facing a zombie. But as the resemblance keeps rising, until it is no greater a gap than the one between two ordinary people, the emotional response swings back to the positive, and the empathy people feel for one another kicks in. That is called "crossing the uncanny valley."

L.O.R.D: Legend of Ravaging Dynasties, released this year, is the cautionary example: for technical reasons its characters looked stiff and unnatural, which also hurt how audiences took to it.

We still don't know how to tell whether we have truly crossed the uncanny valley, so many films do the opposite and make things deliberately unreal. In Shrek, Princess Fiona was first designed to look very much like a real person, so the studio rebuilt her with bigger eyes and smoother skin, to stay clear of the valley.

Once VR or AR reaches film, audiences may even interact with characters and make their own stories; Wang Yongtian showed how AR lets visitors feed sharks in an aquarium. That makes the uncanny valley a particular bottleneck. A guess: when VR films truly arrive, many of the characters you can interact with in them will not be real people at all.

In a VR film the audience goes right inside the scene. Zhai Zhenming, director of the human-computer interconnection laboratory at Sun Yat-sen University, explained: "Once we're in VR it's impossible to separate the story space from the viewer's space, because vertically, horizontally, above and below, it merges directly. There's no need to trace back to the protagonist's first-person point of view, because you're already inside that environment, as the first person. We go from the bystander's viewpoint to the protagonist's."

He said many VR devices on the market today don't count as real VR, because they are only stitched-together 360-degree footage — "the first person and the third person aren't joined up." We are still watching as observers, not truly entering the scene. In the ideal state, once you enter the VR world only you are real, and everything else is computer-generated — "but when you look back on it, you won't know where the real world is." Kevin Geiger, a visiting professor at the Beijing Film Academy, said that when he tried VR, even knowing he was in a virtual world, "my mind, my hands and my eyes connected up automatically." "For instance, I opened a car door to get in, and when I made the motion I lost my balance and reached for the car seat. Everyone knows there was no real car there, so I lost my balance and fell."

Zhai Zhenming can't even picture whether a film with that kind of interactivity would still count as a film. "Developing your story in a virtual world is a completely different concept from shooting one. People haven't even gotten around to discussing this."

Person and point of view are central to how a story is told, and changing them is bound to change how films tell stories. Montage, a technique everyone knows, may vanish from VR film. Zhai Zhenming said: "You shoot a wheel, then a man on the cart swinging a whip, and we know he's driving the horses — that's montage. In VR you can't do that, because at every moment there has to be a continuous transition from the scene to me." Spielberg has made the same point: if people can look wherever they want, how am I supposed to direct?

"It's story, stupid, not technology." Scott Ross, a vice president of Lucasfilm, went through the relationship between technology and film and held up his own studio's Jar Jar Binks as the counterexample: technology should serve the story.

But in front of new technology, the rules of storytelling will change too.

What Might the Solutions Be?

Before taking up that question, we need to recognize that no mature, market-ready VR film exists yet, so we don't even know what problems such films will run into.

Leave the technical problems aside; the biggest problem for the story itself is "fragmentation." Spielberg has worried about it too: if I can chase my own storyline inside a scene, does that make for a fragmented experience? Will they end up fragmenting the experience?

Kevin Geiger thinks games have something to teach here. "If you play games you'll know there are certain key points you have to get through … before you can understand the whole plot." And: "If there's a bug on the ground, say, we can go and look at it, or look at the leaves on a tree nearby. In games you can do that. People can take in the environment naturally without missing the main plot."

Theater got there before film. Immersive theater has been very popular internationally in recent years; Sleep No More became a famous production and will soon be staged in Shanghai. In immersive theater the actors are not confined to a stage and the audience is not confined to seats: the actors move through the performance space, and the audience can watch in a more self-directed way, even take part. Depending on where you watch from, the plot comes out differently. The audience becomes part of the play.

A sense of distance will be an important narrative tool in VR film. In traditional cinema, because the screen and the seats are separate, the distance between audience and film is, in a sense, fixed. Once the audience enters the film, all of that changes. Kevin Geiger made a VR piece about refugees for the United Nations. "In one scene a lot of refugee children are gathered around you, and it's a very moving feeling, because you're right there in it. A lot of people were in tears afterwards, because on a television screen they might never see anything this real, anything they could be this far inside."

And distance carries different feelings: "Our closest family, our children, our lovers, wives, husbands, boyfriends and girlfriends — that's half a meter to a meter. Beyond a meter might be my friends, people who aren't that close but whom we know quite well. Around four meters is our colleagues, people we're friendly with but not necessarily close to. They might not come help you move house, but if you run into them outside they'll nod and smile at you."

Another principle is that interaction has to create a sense of presence. "Virtual reality is real interaction — you make a move and the environment gives you a response." (Kevin Geiger.) Only through that kind of interaction can an audience truly feel it is part of the film, rather than simply watching it from a different angle.

Before such interaction is used everywhere, it may first show up in the shooting of traditional films. Shao Dan said he had just come back from a freezing snowfield, and it had been brutal. "I hope that one day there'll be something like a database. When we want an ice field we won't have to go there to get one; if we need a palace, the palace can stand right there. The cinematographer wouldn't have to be on location. The key thing is, maybe one cinematographer could control two or three films at once, and collect two or three salaries."

Cover image credit: Samsung Gear VR, Wikimedia Commons