How Microsoft spent $1 billion on a simple mistake with the Xbox 360

Former Xbox head Robbie Bach made hundreds of important decisions in his time at Microsoft. He also made plenty of mistakes.

But only one mistake cost $1 billion.

In May 2007, Bach writes in his new book, "Xbox Revisited: A Gameplan for Corporate and Civic Renewal," a group of Xbox team leaders met in his basement to argue about a huge problem: widespread reports of dead Xbox 360 game consoles.

"At the end of a long, difficult conversation, we took a deep breath and decided to extend our warranty to three years and repair or replace every console affected," Bach writes.

That last bit - the warranty extension to three years and the promise to "repair or replace" Xbox 360 consoles - is the $1 billion part.

"It was the biggest decision I made in my 22 years at Microsoft, no question," Bach tells Tech Insider in a recent interview.

It was Bach and his team at Microsoft's Xbox division that were ultimately culpable for fixing a huge defect with the Xbox 360 game console, known as the "Red Ring of Death." Maybe you've heard it referenced recently on Dr. Dre's new "Compton" album and wondered what King Mez was talking about?

The "Red Ring of Death" problem was a massive hardware issue that caused a huge number of Xbox 360 consoles to stop working. The system would flash a ring of three red lights around its power button on the front face (seen above). Thus the nickname.

Here's Bach on the situation, also from his book:

During the first year of Xbox 360 sales, we began to receive customer complaints about Xboxes shutting down and refusing to start back up again. Three of the four ringed lights on the Xbox faceplate would light up red, and the machine would just stop dead in its tracks.

The problem was widespread, but that wasn't initially clear.

The Xbox 360 launched in November 2005. Bach says that around nine months later - around August 2006 - returns of the Xbox 360 began trickling in. "It's a three red lights thing, which unfortunately for us doesn't tell us anything," Bach tells Tech Insider. "It doesn't tell you squat about why, but the hardware is saying to us 'I'm finished and I'm not going to start again.' It could've been almost anything."

Bach's team monitored the situation, but the problem took time to build. And besides, the three red lights were only meant to signal a dead Xbox 360. Even if the folks at Microsoft could see the coming wave, they wouldn't have been able to navigate it with no information.

Bach describes the sentiment at Microsoft in late 2005, leading up to the eventual $1 billion decision in May 2007:

At the beginning, we don't know if we have a trend or you've got two different errors. So you start going through manufacturing batches. You start looking at production numbers, were these two units produced at the same time on the same line at the same factory with the same components. You do all that math: no trend. Then you start to look at the engineering, and then the numbers start to grow. And by six months in, we're struggling to root cause what the problem is.

So, what caused the problem? Bach says it was a measure of putting design over engineering.

In short: The original Xbox was ugly as sin. "It was our ugly, but it was still not a gorgeous thing," Bach says.

Bach's not wrong. The original Xbox had a built-in hard drive, which helped make it far larger than the PlayStation 2 competition from Sony. It came with a controller dubbed "The Duke" because of how large it was. Regardless of the games on the original Xbox, of which there were many good ones, it was built like a tank and looked as such.

With the Xbox 360, Microsoft took a design-first approach. Here's how Bach describes it:

We started with design at the front of the process, and we said, 'This has to be designed with a designer's sensibility.' So the enclosure work we did was done relatively early. Not locked in stone, but we have a shell under which we want to fit. So then the engineering team goes and puts things in the shell.

More clearly: Microsoft designed the look of the Xbox 360 and then figured out how to fit the console's guts inside, which can be risky. Though game consoles are designed to be pretty enough for a living room home theater, their design is also based on heat management. These things are basically computers. If you pack a computer in a tight box, it will eventually overheat.

Worse, it might loosen parts of the system's internals or cause other havoc.

Microsoft had run the console through various tests, from heat to longevity to cold to movement, and plenty of others, and the Red Ring of Death problem was apparently something they didn't come across. It was only when consoles started coming in as returns that Microsoft began to see the scope of the issue.

"I was hugely disappointed and humbled by the entire episode, and that angst left a mark on me that exists today," Bach writes in his new book.

It wasn't until the release of the remodeled Xbox 360 "S" model in 2010 that the console was finally really fixed, despite years of attempts with behind-the-scenes chip changes. "It wasn't really until we shipped the next form factor of the product that the [Red Ring of Death problem] was completely gone," Bach tells Tech Insider.

Amazingly, despite all the problems, the Xbox 360 remained a massive success throughout. It's a testament to how good the Xbox 360 console was, and how adept Microsoft was at fixing the problem, that the console managed to succeed in the face of a $1 billion disaster.