Tim Cook laid out the playbook for Apple's next big thing
The Apple Watch, which has been selling poorly for Apple, isn't going to do it.
Apple needs a new platform. And Apple CEO Tim Cook knows it. That's why he's been dropping hint after hint that Apple's next major technology initiative is augmented reality - basically, integrating software into the real world.
What is AR?
AR is a close cousin of virtual reality. For now, augmented reality means you can hold up your phone and, for example, identify landmarks, discover a hidden Pokemon, or magically translate signs using your device's camera.
The end game of augmented reality is a pair of computer glasses or perhaps a contact lens that can superimpose computer graphics into the real world seamlessly.
Apple CEO Tim Cook has talked about augmented reality essentially every time he's been called on to make public remarks recently.
"This is something you know it's coming," Piper Jaffray managing director Gene Munster told Business Insider. "The only other times they've talked like this is before the iPhone came out, they started to indicate they could do something in the phone market. Before the Apple Watch came out they talked about wearables and the wrist being a better option."
"It's pretty rare that Apple has been this clear that something is coming," Munster said.
If you parse Cook's most recent comments on AR, given at an event in Utah in October, he seems to lay out Apple's strategy for attacking this market, according to analysts and people in the AR industry.
'Three meals a day'
Cook said about AR:
"It will be enabled in the operating systems first, because it's a precursor for that to happen for there to be mass adoption of it. I'd look for that to happen in the not-too-distant future. In terms of it becoming a mass adoption [phenomenon], so that, say, everyone in here would have an AR experience, the reality to do that, it has to be something that everyone in here views to be an 'acceptable thing.'
...
I do think that a significant portion of the population of developed countries, and eventually all countries, will have AR experiences every day, almost like eating three meals a day, it will become that much a part of you, a lot of us live on our smartphones, the iPhone, I hope, is very important for everyone, so AR will become really big.
...
AR is going to take a while, because there are some really hard technology challenges there."
Here's what Cook is signaling:
- Adding key AR building blocks, like the ability for a camera to identify objects, into iPhone software and Apple apps in the near future.
- Adding AR-specific hardware to the iPhone, like a 3D sensor. This could come from Primesense, a company Apple bought in 2013, and would allow the device to understand its surroundings even better than a basic camera.
- Opening up the platform to developers, so that programmers can make augmented reality apps without dealing with the complicated computer vision and physics algorithms currently being developed by people at Apple and other companies with Ph.Ds.
- Releasing an AR-specific product, perhaps a pair of glasses, when the technology is ready. It will be able to take advantage of all the AR apps already developed for iPhone.
"AR is related to all Apple's current businesses; the key is that AR is an innovative human-machine interface that could be used in various devices & applications," KGI Securities analyst Ming-Chi Kuo wrote in a note distributed to clients last week seen by Business Insider. "All of Apple's past successes were related to human-machine interfaces."
'Enabled in operating system first'
"We believe Apple could embed early AR features in iOS 11," the version of iOS expected to be unveiled next June, UBS analyst Steven Milunovich wrote last week in a note distributed to clients.
There are already signs that Apple has experimented with integrating some headset-related features into its iOS operating system. In 2015, it purchased Metaio, a company that had developed a mobile AR browser as well as a whole suite of AR tools.
Last month, developer Steve Troughton-Smith uncovered some interesting features currently included in Apple's iOS.
These features seem to have been abandoned for the time being, Troughton-Smith told Business Insider.
"I think the kind of VR you build for something dumb and simplistic like Google Cardboard is very different from what real VR needs. I imagine they got to the point where they realized they were going down the wrong path if they want to do this seriously," Troughton-Smith said.
This isn't the first time that Apple has built an augmented reality application that has yet to see the light of day. 9to5Mac reported last year that the company had been experimenting with a feature in Apple Maps, based out of the Points of Interest system, that would show an augmented reality view for local listings.
"The augmented reality feature allows a user to hold up her iPhone in the Maps application, and point her camera toward a particular business or an area. Pointed towards a cafe, for example, the screen could show a virtual view of menu items or daily specials," 9to5Mac reported.
Apple is also reportedly working on a Snapchat-style messaging app that could use augmented reality technology to apply filters or manipulate faces in real-time.
"We expect Apple to generate preliminary results for AR in the next 1-2 years at the earliest and working with iPhone may be the first step," Kuo wrote.
'A lot of us live on our smartphones'
"Our best guess is next year, AR is going to be one of the selling points," Munster said.
The next step would be to add depth-sensing capabilities to the iPhone. Depth-sensing is critical for AR - if the computer can tell how far away something is, it's much easier to manipulate on a screen.
Apple purchased PrimeSense, a 3D-sensing company based in Tel Aviv, Israel, in 2013. Many of Apple's chip designing operations are based in Israel.
PrimeSense technology, specifically its 3D sensor, has yet to appear as part of an iPhone.
But Apple did hint at AR-specific hardware with the iPhone 7 Plus.
The iPhone 7 Plus has a dual-lens camera. Basically, the two lenses can act like a pair of eyes, and with some math and triangulation, can give some rough distance estimations. But there are issues with the lenses currently on the iPhone 7 Plus.
Experts say that the lenses are too close to each other to do much useful depth sensing. Imagine if there was no distance between your eyes, for example.
Fast Company talked to a developer who thinks the max distance that it could accurately identify distance is perhaps 50 centimeters - or slightly farther than you typically hold your smartphone away from your face.
Developers are already eager to make AR apps for the iPhone - as soon as Apple releases an AR platform, say, ARKit, there will be a slew of new apps and experiments.
Lenovo launched a phone this week that includes a 3D sensor using Google's technology.
'There are some really hard technology challenges there'
The next step would be to make a pair of glasses, but the timeline here is completely uncertain. "The future of AR is a wearable, but that's probably 5 years down the road," Munster said.
Some glasses are already on the market - Microsoft sells a version of its Hololens AR headset to developers already, and it will go on sale to consumers in the next two years. Magic Leap, a Google-backed startup with $1.39 billion in venture funding, is also gearing up to launch a product in the next year or two.
Apple has been designing headsets in its prototype labs, according to the Financial Times.
Apple is rarely an early adopter, though. And there are several issues with headsets as they currently exist:
- They're significantly heavier than modern glasses.
- It's difficult to integrate a battery into a pair of glasses without getting chunky. Magic Leap's prototype has a cord attached to a battery pack, for example.
- Currently available AR optics have issues with resolution, color, and field of view.
- Headsets are more difficult to store and less portable than something compact like a cell phone.
- There's still the big question of whether society will accept head-worn devices.
But why?
Bank of America Merrill Lynch estimates that AR could have 75% of a $90-$117 billion market by 2020, although that seems bullish.
It's easier to see what Apple's competitors see in the field. Microsoft launched a new computer last month, but instead of focusing on Windows, it spent a lot of time talking about 3D modeling and its Hololens product.
Microsoft missed the boat on mobile. Apple took over 100% of the profit in the smartphone sector this past quarter. So if it wants to become a major platform player again, it needs to find the next one. Smart people think it's going to be AR, even though the market is only nascent.
"Today, Microsoft is focusing on business applications with Hololens, and that is not what Apple focuses on," Blau said. "They really only ever focus on consumer technology for everyone."
"If Apple does anything, it is going to be an every-person's technology," Blau said.
What makes Apple a special company in Silicon Valley is it can design both the unique hardware and software for the project, and it remains one of the best companies at user interface and design. If it were to launch consumer AR based on that, it would be ahead of nearly every other major tech company in this space.
If Apple nails this transition, it could continue to be the world's most valuable company for years to come. If it doesn't, Apple becomes a less exciting company, Cook suddenly starts to look a lot like former Microsoft CEO Steve Ballmer - a caretaker who missed the boat on the next big computer platform.
Email the author at kleswing@businessinsider.com. Alex Heath contributed reporting.