How Walmart hired 2,200 developers and made 'magic'

Okay, so maybe the deals weren't so great on Amazon Prime Day or Walmart's same-day online sale. Still, perhaps take a moment to consider everything that goes on behind the scenes to make online shopping happen.

Consider this:

In addition to all the web traffic that a company like Walmart has to deal with, there's all kinds of complex wizardry that goes into wrangling a tremendous amount of data - millions of items, their prices, customer recommendations, warehouse availability - into a website that you can actually search and buy stuff from.

This is why four years ago, Walmart opened its Walmart Labs divison in the heart of Silicon Valley's Mountain View, not far from Google, LinkedIn, and Yahoo.

It started from Walmart's acquisition of a 65-person search and analytics startup called Kosmix in 2011, but now it employs 2,200 "technologists" - developers, engineers, data experts, and the like - to solve the retailer's biggest digital problems.

"When you build a product for Walmart, millions of people see it," says Walmart SVP and Chief Technology Officer Jeremy King, who heads up Walmart Labs.

Walmart.com sees "millions of transactions" every day, he says. And when you give smart people access to the data and resources they need, you get "magic," King says. It's not a word he uses lightly.

"I'm a big Ayn Rand fan, and magic is a bad thing in that world," King says.

Online and offline were separate worlds

Walmart has had a technology presence in northern California since the year 2000, but at that point, Walmart's site and Walmart's stores were a total mismatch: Not every item from the stores was available online, and even Walmart's internal item numbers weren't consistent between the stores and the website.

It made life difficult for customers and Walmart alike, especially as Amazon's star continues to rise. Around 2011, Walmart CEO Mike Duke knew it was time to take action and attack e-commerce sooner rather than later, or else risk getting overshadowed.

He wanted developers.

King, then at eBay, joined after a personal recruitment pitch from Duke, who was looking for someone a little more entreprenurial to shake things up.

It was an aggressive approach - Duke was supportive and wanted to give Walmart Labs whatever it needed to work, but his attitude was something like "don't call home until you get results," King recalls.

It wasn't long before King realized that stuff had to change on the back end, he says. The company was using a mishmash of different technologies, some of it outsourced to outside vendors and some of it not.

They were collecting massive pools of customer data, but it was just sitting there, unused.

"We have to dump these vendors," King recalls thinking early on in his Walmart career.

And that's what they did, opting to build their own systems that could connect developers to data (though a Walmart spokesperson was quick to note that it still engages with vendors on stuff like storage and networking).

King boasts that Walmart is one of the earliest and biggest users of Apache Hadoop, the hot data-crunching technology that businesses big and small are making use of to build better, smarter applications.

Similarly, when it came to building out a cloud to efficiently support all of the traffic coming to the site, Walmart Labs became one of the first big users of OpenStack, a free cloud technology that's still finding its way into the traditional enterprise.

The important part, King says, is that whatever they build, they build it themselves - often contributing the things they've learned back to the developer world as open source. Plus, the promise of working with hot technologies at huge scales is a great recruitment pitch, he says.

"You can't outsource innovation," King says. "It's important to own and build it at that scale."

The advantage is that Walmart's developers are closer to the data and closer to the systems that actually support Walmart's customers every day.

Walmart data scientists have come up with complex math that can algorithmically ensure that the website meets the retailer's famous low price guarantee - if the software detects a lower price somewhere, it automatically lowers it, King says.

Plus, the data means that they can optimize Walmart's logistics, too, King says. If a lot of people in one area are ordering, say, diapers, Walmart.com's software can detect that and allocate more diapers to that area's warehouses.

"In the end, you need to build a team that has daily access to the life of the customer," King says.

Walmart Labs has no direct place in the brick-and-mortar operation of the company, King says, but he has a "great relationship" with Walmart CIO Karenann Terrell. And as the lines between e-commerce and in-store shopping get blurrier, they're working together closely.

"To make that kind of change, it's not Walmart Labs sitting in the corner," King says.