- A NASA safety panel revealed on Thursday that it is investigating a potentially "catastrophic" software error that occurred during a crucial test flight of Boeing's CST-100 Starliner, a spaceship designed to carry astronauts.
- NASA administrators said software errors were "only symptoms" of deeper problems with Boeing's coding and possibly the company's overall culture.
- Two journalists said Boeing downplayed or denied the software error when asked about it.
- NASA is launching a broader investigation into Boeing's software integration and verification processes.
- Visit Business Insider's homepage for more stories.
NASA is investigating a previously undisclosed software issue that arose during Boeing's test flight of a spacecraft designed to shuttle astronauts to and from the International Space Station.
The Boeing spaceship, called the CST-100 Starliner, is part of NASA's Commercial Crew Program, which asked private companies to develop new astronaut-ready spacecraft. Boeing and SpaceX came out on top in the competition, and the two companies are now racing to launch their first astronauts into space.
But during a crucial orbital flight test in December, a clock software error (it was set 11 hours ahead) led Boeing's Starliner to initiate a phase of the mission it had not yet reached. That caused the spaceship to burn through 25% of its fuel, so Boeing had to skip docking with the space station — the primary goal of the mission — to save the Starliner from total failure.
Now, NASA has revealed a second software issue with the Starliner, which ground controllers had to fix in the middle of that test flight. In a call with reporters on Friday, Boeing and NASA officials said the error could have caused a collision between two units of the spacecraft: the crew module and the service module.
If Boeing hadn't corrected the software error mid-flight, a collision could have sent the crew module tumbling or significantly damaged its protective heat shield. That might have led to "catastrophic spacecraft failure," Paul Hill, a member of NASA's Aerospace Safety Advisory Panel, said at a meeting on Thursday, according to SpaceNews.
Before that meeting, neither NASA nor Boeing had publicly disclosed the error.
No astronauts were onboard for the test flight, but the error has prompted NASA to launch a larger investigation into Boeing's coding and culture.
Symptoms of a deeper problem
Boeing said it found the second issue after the first error with the clock, since that prompted engineers to review the spaceship's code while it was flying.
If they hadn't caught the second error, it could have caused the wrong thrusters to fire just before the spacecraft re-entered Earth's atmosphere. That's the point when the crew module separates from the service module.
Boeing and NASA officials said that incorrect thruster-firing could have bounced the service module back toward the crew model, potentially causing a crash.
"Nothing good can come from those two spacecraft bumping," Jim Chilton, senior vice president of Boeing's space program, said in the call.
Following the failure of Boeing's test flight, NASA convened a team to examine the issues that occurred.
"They are likely only symptoms. They are not the real problem," Doug Loverro, a NASA associate administrator, said in the Friday call.
The heart of the problem, the NASA investigators found, is several coding defects that Boeing's testing team didn't catch before flight.
"We want to understand what the culture is at Boeing that may have led to that," Loverro said.
The team is still investigating some other intermittent issues that disrupted communications between the spacecraft and ground control as well.
"Software defects, particularly in complex spacecraft code, are not unexpected. However, there were numerous instances where the Boeing software quality processes either should have or could have uncovered the defects," NASA press officer Marie Lewis wrote in a blog post on Friday.
2 journalists say Boeing downplayed or denied the glitch
Ars Technica reported that a source had tipped the publication off to the second Starliner software error in mid-January. But when reporter Eric Berger reached out to Boeing about it, he said, a company spokesperson "downplayed the gravity of the situation."
"According to the source, Boeing patched a software code error just two hours before the vehicle reentered Earth's atmosphere. Had the error not been caught, the source said, proper thrusters would not open during the reentry process, and the vehicle would have been lost," Berger wrote.
But the Boeing spokesperson, he said, told him that the software patch "had nothing to do with Crew Module reentry."
Chris Gebhardt, of NASASpaceflight.com, said on Twitter that he'd faced a similar situation: Boeing and NASA had denied a second software issue altogether when he inquired.
NASA administrators on Friday defended themselves and Boeing for not discussing the second error while the investigation was ongoing.
"Had we had the discussion back then, we probably would've gotten it wrong. I think it's not that we were not revealing something," Loverro said. "We didn't want to speculate on that at the time."
This failed orbital flight test was not the first time that Boeing has come under fire for a software glitch with catastrophic consequences. On Thursday, Bloomberg reported that the company uncovered yet another software error in its 737 Max airplanes, which have been grounded since March 2019. Software errors caused the plane to crash twice in five months, killing 346 people.
An 'even broader' assessment of Boeing's software practices
The NASA safety panel called for a broad assessment of how Boeing integrates, verifies, and tests its software.
"The panel has a larger concern with the rigor of Boeing's verification processes," Hill said. "Further, with confidence at risk for a spacecraft that is intended to carry humans in space, the panel recommends an even broader Boeing assessment of, and corrective actions in, Boeing's [systems engineering and integration] processes and verification testing."
NASA appears to be following this recommendation.
Lewis said NASA has asked its independent investigation team to do "a more in-depth analysis as to why the anomalies occurred, including an analysis of whether the issues were indicative of weak internal software processes or failure in applying those processes."
Citing "potential for systemic issues," panel chair Patricia Sanders said Thursday that the investigation will be similar to an internal safety review the agency did after SpaceX CEO Elon Musk smoked marijuana during a podcast recording.
Part of that investigation will involve reviewing all of the Starliner's software, which is comprised of roughly 1 million lines of code, officials said.
"Our NASA oversight was insufficient. That's obvious," Loverro said. "And I think that's good learning for us."
Boeing and NASA declined to say whether Starliner will re-do the orbital flight test. In an earnings report released January 29, Boeing included $410 million expense to cover a potential second uncrewed flight.
SpaceX's orbital flight test, meanwhile, went smoothly. The company is expected to launch its first astronauts — probably the first people ever to fly commercial spacecraft, given likely delays for Boeing — in the coming months.
"The Commercial Crew program is broader than a single provider, and that's intentional," NASA Administrator Jim Bridenstine said. "We have two providers, SpaceX and Boeing, that are going to take American astronauts to the International Space Station."
Dave Mosher contributed reporting.
Join the conversation about this story »
NOW WATCH: NASA is running out of spacesuits and it nearly caused one astronaut to drown in space
https://ift.tt/2Slo29K