NASA is reviewing Boeing Co.’s software engineering, and it doesn’t like what it sees.
Lurking behind 1 million lines of code for Boeing’s CST-100 Starliner spacecraft lies a deficient development process that led to two software flaws during a failed test flight, the U.S. space agency said Friday.
The “critical software defects” — either of which could have caused the uncrewed Starliner’s destruction — prompted NASA to open a broad review of Boeing’s quality control.
“The two software issues you all know about are indicators of the software problems but they are likely only symptoms, they are not the real problem,” said Doug Loverro, the National Aeronautics and Space Administration’s associate administrator overseeing human spaceflight.
The assessment further pressures Boeing as it tries to show NASA it can fly people safely into space. The aerospace giant, already reeling from a deep crisis because of crashes of its 737 Max jetliner, is one of two contractors hired by NASA to ferry astronauts to the International Space Station.
The other, Elon Musk’s Space Exploration Technologies Corp., completed an uncrewed test flight in March and aims to fly astronauts soon.
Boeing’s coding skills have been under intense scrutiny because of software implicated in two Max crashes that killed 346 people. NASA officials conceded that the high-profile problems of Boeing’s best-selling jet suggested the need for a broader look into the company’s culture — and why systems designed to find coding faults had failed.
Risk of Loss
The errors “could have led to risk of spacecraft loss,” NASA said, though engineers were able to compensate during the test flight and return the vehicle back to Earth undamaged.
The space agency and Boeing are now verifying all past programming work on the vehicle. Boeing also plans to check all software that controls the Starliner’s flight, company officials said Friday at a news conference.
Boeing’s problems cast a negative light on NASA’s oversight of contractors, the agency acknowledged. That could complicate efforts to land humans on the moon with systems that NASA purchases from the private sector.
“Our NASA oversight was insufficient” of the Starliner software, Loverro said. “That’s obvious. And we recognize that and I think that’s good learning for us.”
The joint NASA-Boeing news conference was convened one day after a NASA safety panel disclosed the second software glitch, which wasn’t discovered until hours before the Starliner’s descent and landing Dec. 22 in New Mexico.
“In this case, what we wished we’d done better was software so there’s a lot of learning there,” said Jim Chilton, a Boeing senior vice president, who thanked the independent review team for “excellent work.”
The Starliner suffered a problem with its mission timing software shortly after reaching space on Dec. 20, with the craft showing an elapsed time 11 hours different from the actual mission time.
That caused it to fire multiple thrusters too early, burning too much propellant to allow for the craft to continue its flight to the space station and forcing its return after 48 hours.
The second flaw — which NASA and Boeing had not disclosed at the time — involved a “valve mapping software issue,” which Boeing said was also diagnosed and fixed in flight. “That error in the software would have resulted in an incorrect thruster separation and disposal burn,” Boeing said.
The consequences of that mistake might have made the crew and service modules bump into each other after separation, Boeing officials said Friday, although it’s still unclear.
The review teams are also investigating a problem with data linking between the Starliner and U.S. satellites during the flight due to “ground noise” in certain locations, which Boeing engineers believe was generated by cell phone towers.
The signal loss prevented flight controllers from communicating with the vehicle during the early part of its ascent to orbit.
NASA hired Boeing and SpaceX to develop new orbital vehicles to carry astronauts back and forth from the space station. Russia’s Soyuz spacecraft have been the sole crew transport since the Space Shuttle was retired in 2011.
Boeing took a $410 million charge to cover the cost of a second Starliner uncrewed test flight if NASA deems one necessary before it allows astronauts to fly on the craft.
A decision will be made after a full investigation is completed by month’s end, said NASA Administrator Jim Bridenstine.
NASA and Boeing declined to say when the Starliner’s next flight may be.