The inside story of how Amazon created Echo, the next billion dollar business no one saw coming

When Amazon executive Dave Limp first saw the pitch for the product that would become Echo in 2011, his main reaction was doubt.

"This is going to be hard," Limp recalls thinking. "It foretold a magical experience. But it would require a lot of inventions."

The reaction was understandable given the lofty goals outlined in the Echo's original plan: it envisioned an intelligent, voice-controlled household appliance that could play music, read the news aloud and order groceries - all by simply letting users talk to it from anywhere in the house.

Since that time, the Echo has emerged as Amazon's sleeper hit, a hot-selling gadget that's being hailed as the standard-bearer for an entirely new computing paradigm in which Amazon suddenly has an edge on rivals such as Apple and Google.

But the Echo's path into consumers' homes was hardly a sure thing. The gadget was stuck in Amazon's in-house labs for years, subject to the perfectionist demands of Amazon CEO Jeff Bezos and lengthy internal debates about its market appeal. And in the wake of the high-profile failure of Amazon's smartphone, the industry rumors that circulated for years about a speaker product languishing within Amazon's labs seemed like more confirmation that the e-commerce giant lacked the chops to create a game-changing hardware device.

The story of the Echo's origins, recounted by several insiders, reflects the ambitions and challenges within Amazon as it quietly set its sights on the tech industry's next big battleground.

"It was an evolution of a product that was hard," Limp, who is Amazon's SVP of Device, told Business Insider. "I can't underestimate the amount of work and invention that went into this."

Upfront pain

Amazon's Bezos was a huge supporter of Echo from the get-go. But he also held the team to extremely high standards, given the unusual nature of the product the company was trying to invent.

One major concern for Echo was latency, or the time it took for Alexa - the name of the talking virtual assistant that powers Echo - to respond to any query. The average latency of existing voice-recognition technology at the time was around 2.5 to 3 seconds, so the Echo team initially set the goal at 2 seconds, according to an early team member.

But when the team presented its plan to Bezos, Amazon's CEO countered with a much more ambitious target.

"I appreciate the work, but you don't get to where it needs to be without a lot of pain," Bezos told the team in a meeting, according to one early team member. "Let me give you the pain upfront: Your target for latency is one second."

The team was "shellshocked" because even companies that worked on voice recognition for decades were only able to bring latency down to 3 seconds at the time. But at the same time, Bezos' directive also motivated the team to go for what seemed like an impossible goal.

"You did such a good job convincing me that latency matters, that I want to make sure you believe in yourself," Bezos said, according to the team member.

The key to getting latency down was to collect as much data as possible and constantly apply them to improve the product. The team did thousands of internal testing and weekly data analysis with speech scientists. Eventually, the team was able to bring latency down to below 1.5 seconds, far exceeding the speed of its competitors.

"That's just Jeff - he characterized the goal in a way that was crystal clear. It was a big, bold, top-down challenge, and I expected nothing less," another early Echo employee said. "It was hard engineering, but we just did it."

Amazon's Limp acknowledged that reducing latency was a big focus, but did not recall a specific 1-second target.

Wizard of Oz testing

Echo's biggest challenge was that there was no product of its kind in the market to refer to. Sure, Apple's Siri and Google's Voice Search were there, and Microsoft had the Kinect, but the concept for Echo was fundamentally different in that it had no screen for users to interact with.

To succeed, the Echo and its built-in Alexa virtual assistant would need to be very responsive and conversational. It really had to feel like talking to a human-being.

That's where the Wizard of Oz experiment came in.

The test involved a human "wizard" sitting in a separate room and responding in real-time to any voice query a human testing subject would make to the Echo, often without telling the tester in advance. For example, if the subject asked Echo, "What's the weather like in New York?" the wizard in the other room would quickly type and send out an answer through Echo's voice.

The goal was to collect information on what types of responses worked and what didn't work, as the subject would later fill out a satisfaction report on what they liked. For example, 50 people would go through the same script, but with varying response times or sentence structures.

"It was almost a psychology experiment to figure out what does it take to really make people excited?" one of the early member of the team recounted. "We focused maniacally on quality of speaking. It was a constant science project."

Another reason Amazon put so much focus on Alexa's quality was because it saw the voice system eventually being used in other devices, serving as a broader platform. Already, Amazon allows third party hardware makers to integrate Alexa into their own products, like Samsung's doing with its refrigerators and Ford with its own cars.

Not a music device

Even as the Echo team made progress solving technology problems, there were questions about the best way to market the gizmo and about what the right "hook" would be to get consumers to buy it.

That hook was found in internal data: more than 40% of the early Echo testers said their main intent was music-related, according to the early team member. Echo doubled-down on making music a more prominent feature. The very first prototype resembled a smaller hockey puck-sized device, but its size soon expanded to include a more powerful speaker, the person said.

"We needed stuff that people would use a lot and build frequency of engagement," the early Echo team member said. "Music was going to be that thing, the reference point that people know they like."

But music was a double-edged sword in a sense because it could also give the false impression that Echo was simply a music player, and cause people to overlook some of its unique features. Bezos was particularly wary of the Echo being perceived as just a music player, according to the early employee. He understood the importance of using music as an entry point to consumers, but sometimes questioned why the team spent so much time on improving music features.

Part of that attitude might have to do with Bezos's lack of music passion, the early employee theorized. When asked to play music on Echo for the first time, Bezos ordered a song from the soundtrack of the movie "Battlestar Galactica," referring it as one of his favorite songs, the person recounted.

Regardless, Echo ended up improving its speaker quality and song selection, and it now accounts for roughly 25% of the wireless speaker market, according to research firm 1010 data. It's also been successful at not positioning itself as just a music player. In recent months, it's added new functionalities like checking bank accounts, ordering pizza, or calling an Uber taxi by just talking to it. It's also compatible with internet-connected appliances made by companies Nest and IFTTT, making it a central hub for controlling other connected home devices.

The next billion dollar opportunity

Echo's resounding success seem to have surprised even people internally. Amazon's Limp said the team realized within minutes of taking pre-orders that it picked way too low a number for initial estimates. One of the early employees said Echo hit 1 million pre-orders in less than 2 weeks, a far better pace than the iPhone, which took about 70 days to reach the same milestone.

Amazon declined to confirm any of Echo's sales numbers. But all signs point to Echo being one of the biggest hits in Amazon's hardware history. It's been in the market for less than a year, yet it's been met with high demand, consistently ranking in Amazon's best sellers list. Echo was also the top-selling device priced over $100 on Black Friday, and was one of the top 5 items ordered on Christmas Eve through Prime Now, Amazon said.

Perhaps the biggest signal that Amazon is banking on Echo's future potential showed up earlier this year when the voice-controlled device was prominently featured in the online store's first-ever Super Bowl ad. It's why some people are convinced that Echo will end up creating Amazon's next billion-dollar business.

Limp says the sales numbers aren't a priority at this point. Rather, he's more focused on making the product better and smarter. "At Amazon, we believe the next big platform is the voice," Limp said at a recent press event for Echo. "What we're trying to do is to build a computer in the cloud that's completely controlled by your voice."

And that's perhaps what makes early team members proud of their work - they reached for the stars and invented what could be the next big technology platform that only seemed possible in science fiction.

"It really felt like working on something you didn't know whether you would ever achieve, but it was worth trying because it was a cool vision," one of the early employees told us. "What nerd kid didn't grow up wishing they had a Star Trek computer?"

Disclosure: Jeff Bezos is an investor in Business Insider through hispersonal investment company Bezos Expeditions.