Before Jobs Knew About It: How Apple Secretly Built the iPhone — a New Book Tells the Prequel to 'One More Thing'

No, nobody has found Steve Jobs's secret notes yet

Every important invention comes with one especially simple story: a visionary genius has a flash of insight, adds some hard work, and the world changes.

Newton's apple and Edison's thousand filaments are told that way.

This is human nature. Our brains are bad at holding on to facts and much happier stringing information together with simple logic, which is what helps us remember.

On the question of how this century's most important product came to be, Jobs offered one of those simple stories in 2010:

"iOS actually started with the tablet. I had this idea: a glass display, a multi-touch glass display you could type on directly with your fingers. I told our people about it. Six months later they came back with this fascinating screen. I gave it to our user-interface experts, and they came up with inertial scrolling and all the other interactions. And I thought, 'God, we could build a phone out of this.' So we put the tablet aside for a while and started building the phone."

Court records from the Apple-Samsung patent case already show the iPhone's birth wasn't as simple as Jobs made it sound. But Apple's culture of secrecy has left what actually happened blurry.

The end of last month marked 10 years since the first iPhone went on sale. On June 20, VICE editor Brian Merchant published a new book, The One Device: The Secret History of the iPhone (English only so far).

Drawing on interviews with more than a dozen people who were there, plus details that had leaked out through other channels over the years, Merchant tells the story of how the iPhone came to be. Unfortunately, it is still only part of the story.

It All Started With a Regular Meeting Jobs Knew Nothing About

The iPhone got countless things right, but it began with multi-touch.

In 2002, nothing you did on a computer was direct. You clicked through layer after layer of drop-down menus with a mouse or a trackpad. Enlarging a photo usually meant clicking "View," then clicking "Zoom In," which blew the image up 25 percent at a time. If you wanted a little less than that, you had to type in the exact percentage yourself.

Precise, but not direct in the slightest. That is why schools have computer classes, and why a three-year-old can use a phone.

Pinching a photo with two fingers, swiping through a web page, typing fast on glass — the instinctive gestures that even the cheapest smartphone has today all depend on multi-touch. Before 2007, none of them existed on a phone.

Just as Jobs said, multi-touch was where the iPhone started.

But when a group of Apple engineers and designers began tinkering with the technology, Jobs had no idea.

The group called itself ENRI, for exploring new ways to interact.

Its first two organizers were Bas Ording and Imran Chaudhri, both interface designers on the human-interface team run by Greg Christie.

When Jobs returned to Apple in 1997, he killed most of the product lines, including the company's only touchscreen product, the Newton tablet. Apple refocused on the Mac, and the human-interface team's job then was to redesign every detail of the Mac operating system. The Dock, still in use on the Mac today and recently carried over to the iPad, was Ording's design.

However much the Mac changed, it was still built on the indirect interaction of trackpad, mouse and keyboard. These few didn't want to stop there, and went looking inside the company for people who wanted to explore new forms of interaction. Brian Huppi, an input engineer who joined Apple in 1998 and worked on the iBook laptop, joined the little group.

Huppi then pulled in Duncan Kerr, an industrial designer from the same laptop project. By Kerr's account, he was the person in Jony Ive's dozen-strong Industrial Design Studio most interested in software interaction.

Kerr went to his boss, Ive, and suggested setting up a standing discussion group to study new ways of interacting. Ive backed him. Over the next six months, ENRI — a mix of experts from different fields — began meeting in the building that housed the Industrial Design Studio.

Huppi later said he knew from the start that he had to win over the Industrial Design Studio, because they were the people with the most say in the company and the ones Jobs thought most highly of.

Others who joined the discussions included Mike Culbert, a chip architecture expert who had worked on the Newton; Steve Hotelling, Huppi's boss; and Joshua Strickon, a sensor engineer fresh out of the MIT Media Lab.

For the next six months, ENRI met every two weeks.

Jobs did not know they existed.

ENRI Had Its Own Trip-to-Xerox Moment

In those six months ENRI discussed plenty, some of it sound and some of it not: spatial-sensing cameras like the ones Microsoft would later use in the Xbox Kinect, new mouse technologies. Fifteen years on, Huppi said he still can't disclose every project, because some of what they discussed then is still in development and may yet end up in an Apple product.

The Industrial Design Studio already had a number of phone mockups in different shapes at the time, many of them flip phones.

By Huppi's account, ENRI never discussed phones at all. Their free-ranging conversations kept circling back to one subject: navigation — zooming, scrolling.

That naturally narrowed to controlling a screen directly with several fingers. There were plenty of touchscreen products around, including the Newton that Jobs had killed. The trouble with all of them was that none was direct enough.

One day software engineer Tina Huang brought a FingerWorks touchpad into the office. FingerWorks counted as a kind of medical device: a touchpad about the size of a mouse pad. Plugged into a computer, it let you use dozens of complicated gestures to do things that would otherwise take a great deal of clicking and dragging. Huang used it because she had used a mouse so much that her wrist hurt.

Several ENRI members remember that FingerWorks device as the spark. They even contacted the company behind it directly — much as Jobs once toured Xerox's lab, saw the graphical interface and put it in the Mac. A few years later, as Apple began preparing the iPhone, it simply bought the company. Its founder, Wayne Westerman, came to Apple too, and became the author of several important interaction patents.

But FingerWorks was only a touchpad; it couldn't show anything. What ENRI wanted was a screen you could touch.

They got hold of a Mac and hooked it up to a projector and the touchpad, so that whatever the Mac displayed was thrown straight onto the pad. The projector's focal distance was too long, so Christie went home for a macro lens and modified it; they also laid a sheet of white paper over the touchpad to make sure the image came through clearly.

Strickon then rewrote the finger-sensing algorithm, and several interaction designers worked out a simpler logic for zooming. In the end they built a handful of demos with real appeal, including a map you could zoom and rotate, and a photo that bounced back when you flicked it to the edge of the screen.

Secrecy Began With Keeping Jobs in the Dark

ENRI demoed it for Ive first. Ive was thrilled, but decided not to show Jobs yet. "I never demo something to him with other people around. He might say, 'This is shit,' and kill the idea. I think ideas are very fragile. You have to treat them gently while they're being worked out."

ENRI kept improving the multi-touch demo software. In the summer of 2003, Ive finally showed it to Jobs.

By Ive's account, Jobs didn't think it was any big deal. He thought it was a worthless technology.

Ive hadn't figured out what to do with it either. All he suggested was putting it on the back of a digital camera, so the whole back could be one big screen instead of a small screen plus a pair of buttons. Plenty of cameras today are designed exactly that way.

But as Jobs often did, he changed his mind a few days later.

Christie, an ENRI member and head of the human-interface team, was called into a meeting. When he came back he joked: "First of all, Steve invented multi-touch, so every one of you needs to go back and revise your notes."

"As far as I know, Ive demoed it for him and the idea got into his head. Steve does this a lot — by the time he comes back, it's his idea. Nobody can convince him otherwise," Huppi recalled with a laugh 15 years later. "It doesn't matter."

This was routine at Apple under Jobs. Walter Isaacson cited similar stories more than once while writing Steve Jobs.

The First Piece of Hardware Was a Tablet, and It Was Good for Nothing

The idea had a new owner, but that also meant multi-touch got more attention and more resources. It was no longer a projector rig nobody knew what to do with. It became Apple's secret internal tablet project, code-named Q79.

Interaction designers went on refining the details of finger control, Apple's hardware team started hunting for a usable multi-touch screen, and the Industrial Design Studio started drawing what the tablet would look like.

It didn't take long. By 2004, about 50 tablet prototypes had been built. These ancestors of the iPad had white shells, but they were thick and heavy, and the battery lasted an hour.

Some important details were settled then. Take the elastic scrolling you see on the iPhone: drag any page to the bottom and it springs back. Nothing like it existed at the time. On a computer, only the scroll bar told you that a page had hit the bottom rather than the machine having frozen.

A lot of designs that look obvious today were worked out in Q79.

But nobody knew what the thing was or when you'd use it. Apple's marketing department was thoroughly skeptical of a portable touchscreen computer, and suggested a tablet aimed at real-estate agents, who could use it to show clients photos of houses.

It was also too expensive. Going by the parts involved, Apple reckoned it would have to cost at least $1,000, about what a mid-range laptop cost then.

Inside Apple the multi-touch tablet was still an experiment, not the future iPad. Developing it was no fun either. From 2003 to 2004, Apple's finances were only beginning to recover. A few engineers, tired of waiting for new Macs, built themselves a PC to work on.

Then Jobs fell ill and had to have surgery to remove a tumor. Q79 stopped moving, and Strickon, who saw no future in it, left Apple in October 2004.

What Apple Wanted to Build First Was an iPod Phone

Jobs had said publicly that phones were controlled too heavily by the carriers, and that it wasn't a business Apple would go into.

But at the end of 2004 he called Ording and said, "We're going to make a phone. No keyboard, just a touchscreen."

There are several accounts of what changed his mind:

• The Rokr, the iTunes phone Apple made with Motorola, came out in early 2005 and became the biggest failure of Jobs's second act. Tony Fadell, who worked on it, said Apple didn't even wait for the Rokr to flop; it decided to build its own phone on the day of the launch event.
• MP3 phones were multiplying. Mike Bell, an Apple executive who had worked at Motorola, looked at Ive's new iPod design and thought it could be turned into a phone. On November 7, 2004, he emailed Jobs, arguing that Apple should put its own software into an iPod and make a phone, rather than put its software into somebody else's phone.
• During the Motorola collaboration, executives at Cingular — then the second-largest mobile carrier in the US, and later AT&T — privately told Apple that in exchange for exclusive rights, they would keep their hands entirely off Apple's hardware and software.

Whichever participant's memory you take, one thing stays the same: everyone at Apple thought the phones then on sale were junk.

Walking into a market other people are bad at and redefining it is something Apple has done many times. Before the Apple II there were already plenty of personal computers you had to solder together yourself. Before the Mac, Xerox had put out its own graphical-interface computer. Before the iPod there were all sorts of MP3 players that held 10 songs and came with software that was painful to use.

There were two options on the table. Jobs had set ENRI to work on a phone design early, but Tony Fadell, who ran the iPod, caught wind of it and prepared something of his own: the iPod phone.

The iPod phone really was like an iPod. It had an iPod-style click wheel.

The iPod phone was the iPhone's P1 design. It had two modes. Playing music, the wheel showed play, pause, next track and previous track, lit in blue backlight. Switched to phone mode, the wheel showed the digits 0 to 9, like an old rotary dial, with the 26 letters A to Z glowing around the edge.

The other design, called P2, went to Apple's Mac software team under Scott Forstall. P2 was essentially a shrunken Q79 tablet.

Both Fadell and Forstall were famously hard-nosed, each with his own followers and his own enemies. Fadell's nickname was "Tony Baloney"; Forstall's was "Star Fucker" — the same initials as his own name.

Ording and the rest of the human-interface team worked on both projects at once.

At first Jobs leaned toward P1, because a multi-touch phone looked too hard. To get around P1's clumsy text entry, Ording built a smart predictive keyboard that guessed the next word from what you'd already typed.

"We tried everything, and none of it worked. Steve kept pushing us anyway. Put it this way: I think he knew it was a dead end. He just didn't want to give up, so he pushed until there was nothing left to push," Fadell said of P1's failure.

The work wasn't wasted, though. Apple's hardware team finished the cellular system, and the first iPhone used exactly the system developed for P1.

In the end every executive came around to P2. The lone holdout was marketing chief Phil Schiller, who insisted Apple should have a physical keyboard like BlackBerry. Jobs eventually forced him to change his mind.

After Fadell's P1 lost, the iPod team still wanted control, and the opening they chose was the operating system.

Richard Williamson of the P2 team recalled Fadell arguing that the operating system didn't matter, and that Apple should take a version of Linux and draw in support from the open-source community — the way Android later did. It was the easier path: Linux already supported the ARM processors the iPhone would use, and judging by how Google tightened its grip on Android step by step afterward, a system Apple fully controlled could have grown on top of Linux just as well.

Forstall's plan was to cram Mac OS X into the phone, so that from the outset the iPhone would be a complete computer in your pocket rather than an accessory to one.

Essentially, one was an iPod that could make calls; the other was OS X that could make calls.

Jobs picked OS X that could make calls. The whole company then lined up behind a single plan: Ive on industrial design, Fadell's iPod team on hardware, Forstall's Mac software team on software.

What Followed Looks Like the Birth of Any Other Hit Product

Once they knew what they were building, things got simpler. They did not get easier.

Apple's deal with AT&T took more than a year to settle. Code-named Project Vogue, it let Apple do whatever it wanted and came with a subsidy from AT&T. The price was that for the next four years Apple could sell only AT&T-locked phones in the US.

Getting from P2 to the iPhone was hard going too. During the 2012 Apple-Samsung patent suit, Apple released photos of more than a dozen prototypes. One of them was the rough form of the later iPhone 4.

Six of those prototypes were more or less usable, each different in shape, internal hardware and software detail.

In the early 1980s, Jobs, who had handed over the CEO job to concentrate on products, gathered the best people in the company and sealed them off to build the Mac.

Twenty-odd years later the same method was used on the iPhone. Roughly 200 people worked on it. The executives running each part of the phone took the best engineers off their teams. Those people could tell no one what they were doing, and worked punishing hours for years on end.

By January 9, 2007, the iPhone in Jobs's hand still crashed from time to time; the whole demo had to launch its software in a strict order.

On June 29, 2007, the iPhone went on sale.

In the months after it reached the market, a good number of the people who built it were spent, and left Apple. For what happened in that stretch, see Fred Vogelstein's Dogfight.

Star projects tend to be built the same way. Bill Gates said in an interview that within two years of the Mac's release, almost everyone had gone from both Microsoft's and Apple's Mac teams (yes, Microsoft helped prepare the Mac too). Microsoft's most successful project, Windows NT, burned through the energy of hundreds of engineers in much the same way.

From ENRI's first meeting to the iPhone becoming the iPhone, less than five years passed.

Jobs Didn't Invent the iPhone, But He Built the Company That Did

Many blanks are still unfilled, yet Merchant's book makes the story of the iPhone's birth more complete than it has ever been.

The book is nothing like the iPhone in its economy. At least seven-tenths of it has nothing to do with how the iPhone was made.

Long stretches trace the history of one technology after another, from IBM's failed smartphone of the 1990s to the beginnings of GSM.

All of it serves the argument that Jobs did not really invent the iPhone. Even the technologies the iPhone used had been around a long time.

All of that is true, and you don't have to go back as far as the multi-touch pad born in a European quantum lab. By 2007 Microsoft already had a tablet that did multi-touch and displayed images. Many of its interactions looked a lot like the iPhone's.

The difference was that Microsoft's tablet was an entire table, with the image rear-projected onto its surface.

Like Apple's iPhone, Microsoft's Surface tabletop can be traced back to an internal experiment. But Surface — later renamed PixelSense — went on sale still the size of a table, and hardly sold at all.

At Microsoft, nobody pushed the concept of multi-touch from a huge, crude demo rig to something like the Q79 tablet, and then into a phone.

By Ording's account, once the iPhone was under way he met Jobs every two weeks, on subjects as fine-grained as what color copy-and-paste should be and how you should trigger it.

The more complicated the story of the iPhone's birth gets, the more astonishing its success looks.

The difficulty wasn't only technical.

From 1997 to 2003, Apple shipped some successful products, but the financial crisis never really ended. ENRI members remember going years without a raise, and the stock crashed after the dot-com bubble burst. None of it stopped them from getting together and turning multi-touch from an idea into a product.

Apple did use a structure unlike other big companies' to soften the effects of departmental politics: individual departments don't carry their own sales numbers. But it has no shortage of departments, and relations among the managers pushing the iPhone forward were tense enough.

After Jobs was gone, Fadell, Forstall and Ive could no longer work together. Fadell left to start a company; Forstall was fired by Cook.

Greg Christie, the ENRI participant who ran the human-interface team, left in 2015.

Reports said he and Ive couldn't get along. The human-interface team, which used to report to the software division, reported to Ive after Christie's departure.

There was a time when these people stood as equals and built the iPhone together.

Today, among the Apple executives who touch the product, Ive is the only one who matters.

No wonder Apple could make the iPhone in 2007.

Appendix: Where the ENRI Members Went

Bas Ording:

User-interface designer, at Apple from 1998 to 2013. Ording now runs user-interface design at Tesla.

Copy-and-paste in iOS 3, the keyboard used before iOS 7, and the OS X Dock were all his work.

Ording says he left Apple mainly because of the patent litigation. "My name was on those patents, so I had to testify in court over and over. In the end I was spending more time in court than designing. And I really missed working with Jobs, back when we met every 14 days."

Imran Chaudhri:

User-interface designer, at Apple from 1995 to 2016.

Chaudhri talked Jobs out of keeping two buttons on the iPhone. Jobs had insisted at first on a back button. The iPhone that shipped had a single Home button.

At last summer's WWDC, Chaudhri demonstrated the new iMessage app. Several outlets remarked at the time that putting this Black presenter on stage was meant to make Apple look more diverse.

Joshua Strickon:

R&D engineer, at Apple from March 2003 to October 2004.

Before Apple, Strickon worked at the MIT Media Lab, where his master's thesis was on touch interaction. After leaving he changed jobs several times, including nearly two years of technology exploration at The New York Times; he is now trying out toy-robot ideas at Ocean View Lab. Strickon may have been the first Apple engineer to tell the press about the iPhone's development.

Brian Huppi:

Hardware designer, at Apple from 1998 to 2006, after which he founded two companies and then joined Fadell's Nest. He left before Google bought Nest, returned to Apple in February 2012, and stayed until January 2016.

Duncan Kerr:

Industrial designer, joined Apple in 1999. Like most of the designers under Ive, he is still there.

Greg Christie:

Head of the human-interface team, at Apple from 1996 to 2015. Christie is the author of the slide-to-unlock patent, and is said to have left because he and Ive didn't get on.

Mike Culbert:

Chip architect, at Apple from 1988 until his death from cancer in 2013. Culbert was an early participant in the Newton project. His name is on several patents, including rotating the screen image according to the iPhone's orientation and adjusting screen brightness to ambient light.

Cover image / Mike Matas, TheNextWeb

The speaker in the long cover image is Scott Forstall, former senior vice president of Apple iOS software and the software lead for the first iPhone.