When NASA was comparing designs for their post-shuttle rocket recently, they literally used four significant figures in the risk estimates [1], and used this estimate to pick the design. It seems crazy to me to have four sig figs of reliability for designs that were basically at the PowerPoint stage.
NASA started with the requirement that their new rocket have less than 1 in 1000 odds of loss of crew (LOC). They concluded that using an existing Atlas V had LOC odds of 1 in 957 (unacceptable), while the paper design of putting a capsule on top of a Shuttle booster had LOC odds of 1 in 1918 (totally acceptable). They then quoted this 1,918 number in a lot of places to justify the program.
This rocket was the Ares-I [2], which turned into a fiasco and was canceled four years ago.
My conclusion is that NASA's current risk assessments are as bogus as the ones for the space shuttle. They start with an unrealistic goal (1 in 1000 risk), make totally unjustifiable estimates to meet the goal, and then make bad decisions based on these estimates. Coincidentally, the decisions based on these estimates line up with the politically-desirable outcome.
The 1 in 1918 risk assessment turned out to be totally wrong, of course. The Air Force pointed out that the launch escape system wouldn't work since burning fuel would melt the parachute and everyone would die. [3]
My personal view is that NASA needs to admit that rockets are dangerous and you probably can't get the risk below 1 in 100. Then NASA can focus on doing the best job they can. [4]
NASA started with the requirement that their new rocket have less than 1 in 1000 odds of loss of crew (LOC). They concluded that using an existing Atlas V had LOC odds of 1 in 957 (unacceptable), while the paper design of putting a capsule on top of a Shuttle booster had LOC odds of 1 in 1918 (totally acceptable). They then quoted this 1,918 number in a lot of places to justify the program.
This rocket was the Ares-I [2], which turned into a fiasco and was canceled four years ago.
My conclusion is that NASA's current risk assessments are as bogus as the ones for the space shuttle. They start with an unrealistic goal (1 in 1000 risk), make totally unjustifiable estimates to meet the goal, and then make bad decisions based on these estimates. Coincidentally, the decisions based on these estimates line up with the politically-desirable outcome.
The 1 in 1918 risk assessment turned out to be totally wrong, of course. The Air Force pointed out that the launch escape system wouldn't work since burning fuel would melt the parachute and everyone would die. [3]
My personal view is that NASA needs to admit that rockets are dangerous and you probably can't get the risk below 1 in 100. Then NASA can focus on doing the best job they can. [4]
[1] See for example http://www.nasa.gov/pdf/140649main_ESAS_full.pdf figure 1-26 [2] http://en.wikipedia.org/wiki/Ares_I [3] http://archive.is/YD1sh [4] See "Safe is not an option" for discussion on how NASA's focus on safety is harming the space program: http://www.thespacereview.com/article/2435/1