Dropbox had a 'no mistakes' policy with a ticking clock as it moved off Amazon

When Dropbox finally finished its two-and-a-half-year-long "Magic Pocket" project in October of 2015, the team paused for a quick champagne toast. And then it was right back to work.

Over those two-plus years, Dropbox had designed and built its own storage systems - codenamed Diskotech (get it?) - and moved 500 petabytes (that's 500 million gigabytes) of stored data from the $8 billion Amazon Web Services cloud into its own data centers.

It wasn't any ill will towards Amazon, says Dropbox VP of Infrastructure Akhil Gupta, and in fact they're still a happy cloud customer. But with Dropbox usage only accelerating, they needed a system that could handle exabytes of data (that's one billion gigabyes).

And while Amazon Web Services is finding great success with many business customers, Dropbox's needs are at the far end of the spectrum, beyond what Amazon can provide.

As Gupta asks: "How many companies have 500 million users?"

All the while, it literally couldn't fit enough servers into the data center loading docks to handle the explosion of data it was dealing with. With Amazon unable to suit their needs, and their own data centers not enough to suit their needs, Dropbox had to think way outside the box.

So given the scale of what the company was dealing with, that toast was maybe an anticlimactic finale.

But for Dropbox, there was no other way it could have ended: To slow down would be to accept that they had done everything they could do. And that kind of complacency breeds mistakes and oversights. That would be unacceptable, given that those 500 million users are relying on Dropbox in their everyday personal and professional lives.

"We are never, ever losing sight of how important it is to not make a mistake," says Gupta.

Magic Pocket

The name "Magic Pocket" is kind of a company in-joke, taken from a promo video Dropbox released in 2009, trying to explain the then-new service in plain English.

"Putting something in the magic pocket means it's always there, no matter what you wear, or where you are," promises the video. "The same thing is true for computers."

So it seemed like an appropriate name for a huge exodus of data for Amazon, says James Cowling, who led up Dropbox's Magic Pocket team under Gupta. The data would have to be moved, and no customer could ever notice the difference. Like magic.

Starting in the summer of 2013, the Magic Pocket project had four phases, Cowling says: Building the system; getting it correct; scaling it up to the right size; and making sure it's "the most efficient system in the world."

Building the team was almost like the setup to a heist movie, with everybody on the team calling in contacts and old colleagues, like "I know a guy, he's the best guy," recalls Cowling.

This wasn't supposed to be something built fast and rushed out the door, says Cowling. Indeed, it was faster for Dropbox to write the software and build the hardware than it was to run enough tests to convince themselves it was ready for prime time.

"It's more like building a bridge than releasing some software," he says.

The final countdown

To that end, Cowling and his team actually signed a contract with Dropbox cofounders Drew Houston and Arash Ferdowsi, announcing a target date of October 31st, 2015, with provisions for doing the right amount of testing.

If they could get a beta version of Magic Pocket running for 180 days, without any loss of data consistency, said the contract, it'd go live. If not, they would start over as many times as it took. The team even set up a countdown clock to keep themselves honest with the rest of the company.

The stakes were high. Between the manpower and the R&D dollars spent on building the system, there was some obvious pressure from all corners to get it up and running as soon as possible. But by the same token, they couldn't release a sub-par system - if they cut any corners on testing, users would inevitably feel that pain.

"If we reset the clock, it would cost the company," says Cowling.

About two months in, a bug reared its ugly head. That bug never touched the data itself, but it affected how the data was getting organized and filed away. That made for a huge dilemma for the Magic Pocket team, Cowling says.

Technically speaking, the bug had never affected the integrity of the data stored with Magic Pocket. And it was a bug easily quashed. The team could have swept it under the rug without violating the letter of their contract with Houston and Ferdowsi.

But it wouldn't have been true to the spirit of the agreement. Cowling went to Gupta, hat in hand, to explain that they had to reset the clock. Gupta, for his part, was actually happy to hear it: The Magic Pocket team had come to the right conclusion, without the need for his managerial prompting.

On to the next one

On October 7th, 2015, a few weeks ahead of schedule, the testing finally came to an end. In all, it had taken eight months.

Nowadays, Dropbox hosts 90% of its data in its own data centers, on infrastructure built by Magic Pocket (with the other 10% still on Amazon). But the team still isn't slowing down. It's already thinking of the next evolution of the Magic Pocket concept, with new gains to speed, efficiency, and storage capacity.

"All our big milestones, as soon as you hit it, you're thinking of the next one," says Gupta.

Disclosure: Jeff Bezos is an investor in Business Insider through hispersonal investment company Bezos Expeditions.