Catastrophic software errors doomed Boeing's airplanes and nearly destroyed its NASA spaceship. Experts blame the leadership's 'lack of engineering culture.'
- NASA is investigating Boeing after software coding errors jeopardized its Starliner spaceship's crucial test flight.
- One error caused the spaceship's thrusters to fire too early. A second - fixed mid-flight - could have caused a catastrophic crash between two closely maneuvering modules of the spaceship.
- Software flaws also caused two Boeing 737 Max plane crashes that killed 346 people. NASA officials mentioned those problems when they announced the new investigation.
- Together, multiple government agencies are now questioning whether Boeing's newest planes or spacecraft are safe for people.
- One analyst said "there is a clear lack of engineering culture, aeronautical culture, and long-term business culture" on Boeing's board.
- Visit Business Insider's homepage for more stories.
Just 31 minutes after Boeing's CST-100 Starliner spaceship launched into space, Mission Control knew something was wrong.
In the early stages of that crucial test flight on December 20, the Starliner's engines were supposed to fire automatically, setting the ship on a course toward the International Space Station - but they never did.
Mission controllers soon realized the problem: The Starliner's clock was 11 hours ahead. It was following the steps of a phase of the mission it had not yet reached, firing small thrusters to adjust its position.
The uncrewed flight was meant to demonstrate that the spaceship Boeing designed to ferry NASA astronauts to and from the space station could successfully get there and dock, proving that the Starliner was ready to fly astronauts.
Instead, the spaceship wound up burning through about 25% of its fuel before engineers on the ground could get it back under their control. By then, there wasn't enough fuel to reach the space station - only to maneuver the ship into a stable orbit and prevent it from crashing to Earth.
As Boeing workers frantically checked hundreds of thousands of lines of the spaceship's code, they found a second error - one that would have caused the wrong thrusters to fire after two modules of the spacecraft separated. That could have led to a disastrous collision in space.
Boeing and NASA didn't publicly reveal that second error for another seven weeks.
The doomed Starliner flight, of course, wasn't the first time in recent years that issues with Boeing software caused a crisis. Software flaws in the 737 Max airplanes were the reason two of those planes crashed - the Lion Air flight in October 2018 and the Ethiopian Airlines plane in March 2019 - killing 346 people.
NASA announced on February 7 that it would investigate Boeing because of the flaws revealed in the Starliner test flight. That puts the company under intense scrutiny across sectors, with experts at many government agencies questioning whether Boeing's spaceship or its newest planes are safe enough for people.
"The question is now: Is this a software problem, or is it a deeper culture problem?" Bjorn Fehrm, an aeronautics industry analyst at the Leeham Company, told Business Insider. "Part of why they do these [flight] tests like they did is to check everything. And then they find bugs, and that's not extraordinary, but it comes in the backwater of this big debacle of [the 737] Max."
Boeing hastily re-wrote code to prevent 'catastrophic spacecraft failure'
If people had been flying the Starliner, NASA and Boeing officials later said, they would have seen that it had missed a critical engine burn, and could have used manual controls to take over, saving the fuel.
"Had we had an astronaut on board, we very well may be at the International Space Station right now," NASA Administrator Jim Bridenstine said shortly after the flight.
But it turned out to be a very good thing that nobody was onboard.
As Boeing engineers combed through the Starliner's code to make sure it could safely re-enter the atmosphere and land on Earth, they found that it was missing the proper instructions to separate two units of the spacecraft: the crew module, which returns to Earth, and the service module, which gets discarded in space.
The code, as it was written when the spaceship launched, would have instructed the wrong thrusters to fire just after the two modules separated. That could have propelled the service module back toward the crew module, and the two units could have crashed. Such a collision could have sent the crew module - the one astronauts fly inside - tumbling or damaged the heat shield that protects it while reentering Earth's atmosphere.
"Nothing good can come from those two spacecraft bumping," Jim Chilton, senior vice president of Boeing's space program, said later in a call with reporters.
So Boeing coders worked through the night to rewrite the software and send it to the spacecraft.
"I don't think we'd have found [the second error] if we hadn't gone looking after that first one," Chilton said.
Around 5 a.m. on Sunday, the Starliner got its new code.
"This actually was a pretty simple coding sequence," John Mulholland, vice president and manager of Boeing's Starliner program, later told reporters. "Within a couple hours, they were able to re-code it and run it back through the verification."
At 7:23 a.m., the spacecraft began its return to Earth - a process that could have led to disaster just three hours earlier.
Journalists say Boeing downplayed the second coding error
If Boeing hadn't caught the second error, it might have led to "catastrophic spacecraft failure," Paul Hill, a member of NASA's Aerospace Safety Advisory Panel, said at a February 6 meeting, according to SpaceNews. It was the first time the agency had openly discussed its investigation into the software patch.
Two journalists reported that they'd heard about the second software problem before that meeting and had asked Boeing about it, but they said the company had downplayed the issue.
NASA, however, acknowledged the gravity of the problem when it announced plans for a deeper investigation into Boeing's culture and software processes.
"Software defects, particularly in complex spacecraft code, are not unexpected. However, there were numerous instances where the Boeing software quality processes either should have or could have uncovered the defects," the agency said.
'We want to understand what the culture is at Boeing that may have led to that'
NASA said in early February that a preliminary investigation had uncovered deep flaws in the Starliner code.
The two errors discovered in-flight "are likely only symptoms," Doug Loverro, a NASA associate administrator, said in a call with reporters. "They are not the real problem."
The bigger issue, NASA found, was that Boeing's testing team didn't catch several defects before the flight.
"We want to understand what the culture is at Boeing that may have led to that," Loverro said.
The Orlando Sentinel reported on Wednesday that a critical end-to-end software test ahead of the Starliner launch could have caught the two coding errors, but Boeing didn't conduct that test at all.
"It's pretty exhaustive. You gotta do that," Christopher Saindon, a former member of NASA's safety panel who helped review Boeing's test-flight results, told the Sentinel. "That was somewhat surprising to us on the panel. There were certainly gaps in the test protocol."
Boeing told the Sentinel that it had followed all of NASA's testing requirements.
"I wouldn't characterize it that the team did not do extensive testing, because they did," Mulholland said in a call with reporters on Friday.
He did not directly answer questions about why the company chose not to do the end-to-end test, but announced that going forward, Boeing will run tests of its software systems in larger chunks (from launch to space-station docking, then from docking to landing). Such testing would have caught the clock error, he said.
Some of NASA's concerns stem from the deadly history of the 737 Max
Boeing is still trying to get the 737 Max back into the air after software flaws caused two deadly crashes.
Investigations found that the pilots on both flights struggled to control a malfunction in the automated maneuvering system that forced the planes' noses toward the ground. In a recording from the cockpit of the Lion Air flight, the pilots can be heard desperately reading an emergency handbook looking for a way to control the plane.
Boeing faces investigations related to those crashes from the FAA, Department of Justice, Congress, and Securities and Exchange Commission.
The company reported a $636-million loss in 2019.
Boeing revealed on Monday that its new CEO, Dave Calhoun, could get a $7 million bonus if he fixes the company's biggest problems, including returning the 737 Max to the skies and launching astronauts for the first time.
"I think it's probably 60/40 that Boeing commercial aircraft will recover from this," Christine Negroni, a journalist who specializes in aviation safety, previously told Business Insider. "But I definitely think that they've been shaken up enough to know they have to change their ways."
Loverro said he considered all of this when he decided NASA would launch a deeper inquiry.
"There were several factors that were in my mind when I asked the boss if we could do this," he said. "Obviously press reports that we've seen from other parts of Boeing, as well as what seem to be characterized as software issues, and obviously the [orbital flight] test."
In an email to Business Insider, Boeing spokesperson Mike Friedman said the company was working to address these concerns.
"While the exact nature of the issues are on unrelated systems, we are taking action across the company to strengthen our safety culture, elevate our engineering focus, improve transparency and rebuild trust," Friedman said. "For example, we are realigning and strengthening our engineering team, as well as establishing a new Product and Services Safety organization that reports directly to our chief engineer. We're also adding new safety roles and expanding safety-reporting systems both inside Boeing and within our supply chain."
'That culture actually produced the Max disaster'
Following the second 737 Max crash, The New York Times reported that Boeing had developed the plane in a "frenzy" and that the goal was to avoid extra pilot training. Fehrm said the ultimate problem was the company's leadership.
"There is nothing wrong with the technical side of Boeing, but the top management and the board has had a focus on [keeping] costs down for a long time," Fehrm said. "That culture actually produced the Max disaster. I don't think there is any question about that."
However, planes and spaceships come from separate branches of Boeing, and Fehrm added that he doesn't know whether the same cost-cutting attitude trickled into Boeing's spaceship program.
Saindon told the Sentinel that the NASA safety panel "would never tie those two together, it's a completely different arm of the organization."
But he added: "That doesn't mean it's not a business right? And they're trying to do things efficiently and cost-effectively."
'The problem is sitting at the top'
Fehrm said it's reasonable for NASA to question Boeing's culture.
"They start to take notice and they ask themselves, 'Are we having the same problem in our corners of the Boeing company?'" Fehrm said. "There is a clear lack of engineering culture, aeronautical culture, and long-term business culture in the board."
"There's nothing wrong at the middle layer and the lower layers of the company. The problem is sitting at the top," he added.
Boeing has made major leadership changes since the 737 Max crashes, Friedman said. Its former CEO, Dennis Muilenburg, was fired in December. In the months before that, the company added Admiral John Richardson, a former Naval Operations chief with a master's in electrical engineering, to its board and established an Aerospace Safety Committee on the board as well.
"Going forward, safety and engineering experience will be key factors in any future board appointees, as the board amended the company's Governance Principles to include safety-related experience as one of the criteria it will consider in choosing future directors," Friedman said.
On Monday, Boeing nominated two new board directors, one of whom also has an electrical engineering background.
Calhoun said on a recent earnings call that "leaders have a massive role to play in setting culture, setting the stage for how to fix a culture."
He added, "I have to demonstrate that one step at a time, every inch of the way."
SpaceX will likely launch astronauts first
Until the Starliner problems arose, Boeing had aimed to launch its first astronauts in mid-2020. It's not yet known whether the spaceship will have to repeat that uncrewed test, but Boeing seems to have budgeted for the possibility: An earnings report released January 29 included a $410 million expense to cover a potential re-do.
SpaceX, meanwhile, is poised to win the race to launch the first astronauts on a commercial spacecraft. Elon Musk's company is NASA's other partner in its Commercial Crew Program, which has funded the development of new spaceships to ferry astronauts to and from the space station. SpaceX aced its own test flight in March 2019.
"The Commercial Crew program is broader than a single provider, and that's intentional," NASA Administrator Jim Bridenstine said.
Loverro recently told Ars Technica that a 'load' of paperwork is pretty much all that's left before SpaceX can launch astronauts.
Boeing, meanwhile, has to revisit 1 million lines of code.
"Only after that will we be able to say what we would do going forward," Chilton said.
Dave Mosher contributed reporting.
Get the latest Boeing stock price here.