Autonomous Driving Is Here. Which Technologies Made Cars Smart? | CES 2016

More than a century after its invention, the car is finally becoming a computer on wheels

Over the past year or two, whether or not you care about cars, you can hardly have missed the news about Google's driverless car.

The day a driverless car comes to pick you up from work on its own is still some way off. But active collision avoidance, which brakes the car automatically in an emergency without the driver doing anything, is already a new selling point of the 2016 Volkswagen Passat. In the US, models with the feature start at just over 200,000 yuan. The pricier Volvo S60 can already inch along in highway traffic jams by itself, without the driver touching the brake or the accelerator.

Smart cars can now do far more than that. The Tesla Model S's autonomous driving feature, Autopilot, already changes lanes on the highway by itself.

Traditional carmakers have not followed, because the technology is not yet completely safe.

"Tesla's output is small, and its customers tend to be highly educated. Tesla isn't a mass-market product yet. If General Motors or Chrysler did this, they would go bankrupt very quickly," Robert Evans, vice president of sales at the automotive systems integrator ZF, told QDaily. At this year's CES consumer electronics show, ZF's booth displayed several autonomous driving systems, some of them already used in production cars.

True, you can find videos of a Tesla Model S misreading the road. But most of the time it already knows what to do. Within a few years, automatic driving, collision avoidance, lane changing and similar functions will be standard on new cars.

How did the car go from a machine that moves only as the driver's hands and feet tell it to, to something this "smart"? And what are the different companies in the industry doing?

How does a car read the world? Sensors gather its first-hand information

The first step in teaching a car to drive itself is getting it to understand the world humans see. That depends on sensors.

In November 2015, Toyota tested a driverless car converted from a Lexus GS450h on Japanese roads. At this year's CES, Toyota used the same car to show its near-production autonomous driving technology.

On site, QDaily counted 5 lidar units, 3 ordinary radars, 3 optical cameras and 2 GPS antennas on the car.

The front of the car carries the most sensors

The camera inside the car is the most basic sensor autonomous driving needs

Two more lidar units sit at the rear sides

Optical cameras

The most basic sensor for autonomous driving is the camera. Its hardware is not advanced; its resolution is lower than your iPhone's. But it lets the car see the road ahead the way a driver does.

A monochrome camera records the road as the car sees it. Image-recognition algorithms then pick out the cars, pedestrians and trees ahead, and even a cyclist hiding in the driver's blind spot.

Mobileye is probably the driver-assistance technology supplier that works with the most carmakers today. GM, BMW, Ford and others are its customers. The more than 4,000 Tesla Model S cars upgraded to Autopilot last year also use its technology.

Evans, ZF's vice president of sales, thinks a single camera has limits because its field of view is narrow, and that multiple cameras are indeed the better solution. But the camera is not the end point of Mobileye's approach. The Israeli technology company buys camera components from others; what it makes itself is a chip, EyeQ, loaded with optimized image-recognition algorithms.

In the end, the car does not look at video footage. It receives commands directly: if the system sees a car ahead, for example, it tells the car to stop.

Many startups offer similar software, such as ZMP, a Japanese startup developing driverless-car technology. It is the main supplier to a driverless-car project being prepared for the Tokyo Olympics, and Intel invested in it 2 years ago. But not every company does both software and hardware, as Mobileye does.

Doppler radar

This is the familiar parking radar. Its resolution is low and it can barely make out distinct objects, but it measures the distance to surrounding objects reliably in rain and snow. It works on the same principle as a bat's echolocation: objects at different distances reflect radio waves differently.

The autonomous car Lexus showed at this year's CES hides 2 short-range radars in its headlights. They reach only 30 meters at most, enough to see what is around the car when it turns left or right and keep it safe. They also make up for cameras, whose image quality drops sharply in rain, snow and darkness.

The Lexus badge on the car in the picture hides a long-range radar as well, which can detect objects up to 250 meters ahead. Longer-range Doppler radar can measure the speed of the car in front; adaptive car-following and lane changing both depend on it.

Lidar

Cameras and radar are a driverless car's basic equipment. But on the road a driverless car has to see much farther ahead, to give its system more time for precise calculation. The more advanced hardware for this is lidar.

The lidar on Ford's driverless car at CES 2016

Unlike Mobileye's driver-assistance approach, Google's driverless car does not need algorithms to turn a flat image of the road into a three-dimensional one, because the live picture captured by its roof-mounted lidar is already 3D. Google's Street View team uses the same method, with errors kept within 2 centimeters.

Lidar measuring distance

Invisible laser beams shoot out and bounce back to the lidar almost instantly. Light travels faster than sound, and this piece of common knowledge is what lets a driverless car analyze the road in real time. The system's GPS helps calibrate the bearings the laser works out. The inertial measurement unit (IMU) works like the gyroscope in a phone, tracking how fast objects are moving at every moment and helping judge relative distance.

On Lexus's concept car the lidar no longer stands out, but most driverless cars today still carry a big disc on the roof, as Google's does.

Velodyne, a Silicon Valley technology company founded more than 30 years ago, showed a lidar the size of a coffee can at this year's CES. Its first commercial product, the SC-600 IW, developed in 2007, was more than 30 inches (about 76 centimeters) across. And a mechanical lidar built with standard technology costs US$80,000.

Velodyne's lidar keeps getting smaller

This may be one reason Google's driverless car has been so slow to reach mass production: lidar accounts for half the cost of the whole vehicle. A year ago, Audi said officially that lidar could go into mass production only once its cost fell to US$200.

To bring driverless technology into everyday use sooner, Quanergy has followed Mobileye's "weak hardware, strong algorithms" approach. It replaced the 360-degree rotating camera and laser rangefinder with a solid-state image sensor, which should bring the cost down to about US$1,000 a set. The Canadian startup LeddarTech goes further and uses LED light to measure distance; its kit sells for only US$299.

Single camera or several, ordinary radar or lidar, these sensors in essence give the car a pair of intelligent eyes, letting it see the road at different ranges.

More sensors are turning the car into a smartphone on wheels

An ordinary car already contains quite a few sensors, or electronic components. In the 1960s and 1970s, as engine design moved from adapting to fuels, to raising horsepower, to controlling exhaust emissions, the whole design became steadily more modular.

An on-board diagnostics (OBD) system monitors how the engine runs and what comes out of the exhaust. There is also the anti-lock braking system (ABS). But the on-board computer stayed primitive for a long time, doing little more than watching for engine faults. And every new function required a separate electronic component with its own CPU. Today a car has at least 50-odd chips, and often more than a hundred.

That many processors makes no sense. Before they founded Apple, Steve Jobs and Steve Wozniak were developing games for Atari. Their boss, Nolan Bushnell, told Jobs he would pay a US$100 bonus for every chip removed from the machine. Wozniak used memory to cut 50 chips.

The phone maker Nokia once wore itself out upgrading its operating systems and chip performance to keep up with ever-changing hardware. Its new operating systems came out more and more slowly, and they still did not work well enough with all the different hardware.

Sync, the first-generation infotainment system Ford built with Microsoft, stayed on the market for a full 5 years. The iPhone's hardware and software are updated more than once a year.

Hardware integration

A computing platform that pulls together data from different sensors can lighten the load on car system design and leave more room for future technology.

At this year's CES, the automotive electronics maker Delphi unveiled its latest Multi Domain Controller (MDC), which will integrate the ABS, OBD and radar systems mentioned above. Audi will be its first customer, and it will appear in new 2016 models.

With the MDC, the car needs noticeably fewer electronic components

The chip companies are trying to do the same.

The jumbo chip Nvidia CEO Jensen Huang held up, the Drive PX 2, which he called an "autonomous driving supercomputer," can process data from up to 12 two-megapixel cameras running at 60 fps.

In March 2015, the Dutch semiconductor company NXP bought Freescale Semiconductor for US$11.8 billion. NXP is good at custom chips for automatic braking and cruise control, while Freescale's strength is the wireless technology that connects cars to the internet. The rest of the top five in the 2015 global automotive chip market were Renesas, Infineon and STMicroelectronics.

Meanwhile, to get into automotive chips, traditional semiconductor companies have been buying complementary technology faster. Last December, Intel bought Altera for US$16.7 billion. Wendell Brooks, Intel's vice president of mergers and acquisitions, said at the time that combining Intel's Atom processors with Altera's programmable FPGA chips would produce chips usable in areas such as automotive electronics.

The scene recalls the computer market of the 1970s and 1980s, when the biggest semiconductor companies were all Japanese: NEC, Toshiba and Hitachi. Intel was still fighting another American company, AMD, for business, and the phone maker Motorola had its own chip business. Today "Intel Inside" dominates the vast majority of laptops and server rooms.

And this time? More startups will appear, and acquisitions will follow. No one has won yet.

Operating systems

As cars need more and more sensors and processors, a single operating system to run them all becomes a necessity. An operating system lets software developers ignore the hardware. An Android developer building a camera app, for example, does not need to know which camera module each phone uses: Android talks to the hardware, and the developer only has to talk to Android.

Kerry Johnson, a product manager at QNX, BlackBerry's automotive operating system, also told QDaily: "As cars use more and more technology and the software gets more complex, the auto industry needs an advanced operating system, not the traditional kind that can do only one thing."

In computers and smartphones, the most basic logic is to hide complex hardware behind an operating system so that developers can concentrate on software. Once cars become smart, they will have to do the same. That is exactly what BlackBerry's QNX is doing.

At this year's CES, QNX brought an autonomous driving operating system. It makes only the underlying operating system and stays entirely out of hardware manufacturing and out of writing the software algorithms for specific functions. The autonomous driving algorithms come from the Israeli startup Adasworks; the Russian company Itseez supplies the image recognition. Its processor partners include Texas Instruments, Renesas, NXP, Intel and Nvidia.

QNX, which puts its emphasis on system safety, was already installed in more than 50% of the world's cars. But it is not the only company making automotive operating systems. The chipmaker Nvidia is building one, and the autonomous driving company Mobileye is trying too.

Then there is the Open Automotive Alliance (OAA), launched by Google. As with phones, Google wants to control every link in the making of smart cars: hardware manufacturing (carmakers), software development (technology companies), processors (chip companies) and electronic components (parts suppliers).

But carmakers have seen what happened to Android phone makers. Ford's chairman, Bill Ford, has said he does not want car companies to become hardware manufacturers working for more innovative companies.

Google's camp already includes several carmakers, among them Audi, GM, Hyundai and Bentley. Apart from Nvidia's chips, they use almost none of the alliance's other products.

Still, Google or Apple may one day build a complete car system of its own. What system Google's driverless car actually runs is still something of a mystery. Apple's car plans are even more so.

Who glues the hardware and operating systems together?

The parts suppliers ranked in the top ten by market share, such as Delphi, Continental and Bosch, have become the companies that finally integrate hardware and operating systems, not the carmakers.

At this year's CES, the German drivetrain maker ZF put a concept car on its booth, just as a carmaker would, and invited visitors to sit inside and experience, in front of a big screen, what the world will be like once autonomous driving arrives. The image-recognition video playing on the booth's TV carried a watermark: "Licensed by Mobileye."

On one hand, these suppliers buy technology startups or partner with large technology companies, such as Mobileye, for software algorithms. They are also QNX's customers. On the other hand, because safety matters so much to autonomous driving, parts makers have started merging with one another. In September 2014, ZF acquired the debt-laden American company TRW Automotive, which specializes in car safety systems such as anti-lock brakes. Safety was also the keyword of ZF's autonomous driving solution at this year's CES.

As the auto industry stands today, carmakers, to cut costs, handle only the overall vehicle design. They buy parts from suppliers through tenders or strategic partnerships, and the supply chain is tightly knit. "We work with companies like Mobileye, integrate all the systems, and then bring them to the carmakers," ZF's vice president of sales, Robert Evans, told QDaily. "That is the advantage parts suppliers like ZF, Bosch and Continental have in autonomous driving."

Like phones and computers, cars have to be connected

Maps, this time not for humans

In daily life you may already be lost without map navigation. Cars are even more dependent on it.

The world as a driverless car sees it

HERE, the European mapping company that BMW, Audi and Daimler jointly bought in the second half of last year, is working on exactly this. At CES it launched HERE HD Live Maps. The name sounds ordinary, like yet another Google Maps or Baidu Maps, but the way it works is entirely different.

On the maps you use every day, the finest line may be an alley. A map made for cars has to show whether a lane marking is dashed or solid. If a lane is marked "bus only" or has restricted hours, the map must recognize that too.

A HERE mapping car

HERE's mapping cars, with a disc on the roof, also use more precise lidar to record the positions of all road signs. When a car sees those signs, it can work out its own position from where they are. This gives the car a more precise location than a GPS map can.

Perhaps you can finally consider using the car's built-in map instead of the one on your phone.

As with phone maps, mapping companies want data from every car. Three companies have announced they will supply data to HERE. With so many Google Street View cars, and even driverless cars, on the world's roads, it is hard to say how far Google Maps has come. Until Tesla decided to make its own maps, the navigation data on the Model S's central display came from Google Maps.

Cars need to connect not only to the internet but to a local network

A smart car on its own is not enough. Once cars are networked, data will flow far faster between cars, traffic lights, pedestrians and more.

When a neighboring car suddenly cuts into your lane, a collision can be avoided. With a city's traffic-light data, a car can also predict when a light will turn yellow or red and speed up or slow down automatically.

The technology is similar to electronic toll collection (ETC) on highways. Every car carries an electronic tag. Within a range of a few dozen meters, a management system can identify vehicles moving at high speed, and cars can identify one another as well, including the driver's identity, speed, relative distance and relative speed.

Four videos QNX played at CES explain V2V (vehicle-to-vehicle) technology clearly:

This dedicated short-range communications (DSRC) technology is also used to track goods on production lines. But in real life, little hardware supports this wireless standard. For cars to truly connect to everything, governments and carmakers may have to push together. It will not happen within three to five years.

Beyond autonomous driving, the car's interface is getting better too

The car's glass is a ready-made display

In 2007 the first iPhone replaced the fiddly physical buttons of mobile phones with a touchscreen. As cars get smarter, the hard-to-use instrument panel and central screen will merge into the few panes of glass the car already has.

The Tesla Model S's 17-inch central screen may not be the best answer. The Vision concept car Jaguar Land Rover unveiled 2 years ago put car information on the windshield, but this kind of technology has been slow to reach production. Phone apps such as HUDWAY also want to project a live navigation map onto your windshield.

A virtual windshield

So the parts maker Visteon added a screen above the instrument panel: a head-up display (HUD). The HUD began as the optical gunsight of fighter jets and was later used to show flight information so that pilots did not have to look down. Because it took up precious cockpit space, it was dropped in favor of AR helmets. But the AR helmet BMW showed at last year's Shanghai Auto Show looked more like brand marketing, there only to create a sense of the future.

The all-glass cockpit is now standard on modern fighter jets. When it will reach cars is hard to say, but that is the direction.

Voice recognition

With the voice assistant built into CarPlay, Android Auto or Ford Sync, you can already start a car or turn on the air conditioning to warm it up or cool it down in advance.

Phone voice assistants are still not smart enough. But in a smart car that still needs a steering wheel, voice control is the safest way to work other functions, such as playing music or sending texts. BMW is working with the voice recognition company Nuance Communications to help its voice assistant understand accents.

Tracking eye movements to catch drowsy driving

Using infrared cameras to capture the driver's eye and face movements, the Australian company Seeing Machines has shown a built-in real-time monitoring system on the Jaguar F-Type that tracks the driver's attention.

It can follow your eyes even through glasses or sunglasses. It is linked to the emergency brakes: if the system finds your attention has drifted from the wheel while danger appears ahead, the car brakes automatically.

Biometrics

Synaptics, which makes laptop touchpads, launched its Natural ID technology at this year's CES. It embeds a fingerprint reader between layers of glass, so fingerprints can be read through the glass. When people no longer need to buy cars, but rent them or ride in driverless ones, how do you make a car feel like your own? Using biometrics to call up your usual playlist and adjust the seat automatically sounds like a sensible answer.

Smart cars today look a lot like the computer business more than 30 years ago. Back then Apple, Microsoft and Intel were all small companies. As with cars today, operating systems, CPUs and even the internet each had several competing standards. A 95% share of installed operating systems once made Microsoft's online service MSN one of the earliest contenders on the World Wide Web.

Not all of the many companies working on smart-car technology today will survive; one by one they will be absorbed into bigger companies. The buying has already begun. In 2015 Delphi bought 2 software companies in one go and invested in Quanergy, which makes cheap lidar. Continental acquired the autonomous driving software company Elektrobit Automotive.

Once the market players settle, autonomous driving will more or less have truly arrived.

Whoever wins the autonomous driving race, carmakers are losing their grip on the car. Building cars used to be the core of the auto industry, and the carmakers that held the engine patents controlled the whole business. Now the companies that control autonomous driving are chip companies, software companies, internet companies... Even parts integrators showed driverless cars of their own at this year's CES.

Dan Akerson, the former CEO of GM, once said that Apple has badly underestimated how hard it is to make cars: "If I were an Apple shareholder, I would not be happy. I doubt there is much to gain from adding a low-margin, complex manufacturing business."

Once autonomous driving arrives, carmakers that cannot break into the system and service layers may hold on to little more than manufacturing, reduced to assembling hardware, like Foxconn, which builds the iPhone.

Cover image from Pixabay