Henry Spencer got it right (no big surprise):
The gap between engine cut off and staging was 1.5 seconds – which was fine for the ablatively cooled engine on Flight 2. But on Flight 3, with the regeneratively cooled engine, there was some residual thrust after engine shut down and this caused the first stage to be pushed back toward the second stage after separation and there was a recontact between the stages.
One of the big mistakes that people make in writing requirements is not writing proper verification statements for them. One of my rules, that I came to late in life, is to not allow a requirement to be accepted unless it has an accompanying verification statement (i.e., how you verify that the requirement has been satisfied). If you can’t write a verification statement for it, it’s not a valid requirement. The other reason is that verification is where most of the cost of a program comes from. Test is very expensive. If you can come up with ways to verify early on that don’t require it (inspection, demonstration, analysis), you can control and estimate costs much better.
One of the key elements of a proper verification statement is the environment. It’s not enough to say, “Verify, by test, that engine thrust is less than TBD Nt TBD seconds after engine shutdown.” It has to be “Verify, by test, in vacuum, that engine thrust is less than TBD Nt TBD seconds after engine shutdown.”
AMROC had a similar problem on SET-1 back in 1989, because the propulsion system testing had all taken place in the desert at Edwards, and the actual launch occurred in the humid October weather of Vandenberg, at the coast. The LOX valve iced up. The vehicle ended up catching fire and fell over and burned on the pad. There was no explosion, but it was a launch failure.
This is why systems engineering processes were developed. I’d be curious to know what kind of SE processes SpaceX had in place. And what they’ll have in place in the future…
[Late evening update]
Here’s the official statement from Elon Musk:
Timing is Everything
On August 2nd, Falcon 1 executed a picture perfect first stage flight, ultimately reaching an altitude of 217 km, but encountered a problem just after stage separation that prevented the second stage from reaching orbit. At this point, we are certain as to the origin of the problem. Four methods of analysis – vehicle inertial measurement, chamber pressure, onboard video and a simple physics free body calculation – all give the same answer.
The problem arose due to the longer thrust decay transient of our new Merlin 1C regeneratively cooled engine, as compared to the prior flight that used our old Merlin 1A ablatively cooled engine. Unlike the ablative engine, the regen engine had unburned fuel in the cooling channels and manifold that combined with a small amount of residual oxygen to produce a small thrust that was just enough to overcome the stage separation pusher impulse.
We were aware of and had allowed for a thrust transient, but did not expect it to last that long. As it turned out, a very small increase in the time between commanding main engine shutdown and stage separation would have been enough to save the mission.
The question then is why didn’t we catch this issue? Unfortunately, the engine chamber pressure is so low for this transient thrust — only about 10 psi — that it barely registered on our ground test stand in Texas where ambient pressure is 14.5 psi. However, in vacuum that 10 psi chamber pressure produced enough thrust to cause the first stage to recontact the second stage.
It looks like we may have flight four on the launch pad as soon as next month. The long gap between flight two and three was mainly due to the Merlin 1C regen engine development, but there are no technology upgrades between flight three and four.
Good Things About This Flight
- Merlin 1C and overall first stage performance was excellent
- The stage separation system worked properly, in that all bolts fired and the pneumatic pushers delivered the correct impulse
- Second stage ignited and achieved nominal chamber pressure
- Fairing separated correctly
- We discovered this transient problem on Falcon 1 rather than Falcon 9
- Rocket stages were integrated, rolled out and launched in seven days
- Neither the near miss potential failures of flight two nor any new ones
were presentThe only untested portion of flight is whether or not we have solved the main problem of flight two, where the control system coupled with the slosh modes of the liquid oxygen tank. Given the addition of slosh baffles and significant improvements to the control logic, I feel confident that this will not be an issue for the upcoming flight four.
Elon
And it was probably a mistake to try out his new first-stage engine without first getting a successful flight or two with the old engine! The engine switch might have looked like it would speed things up in the long run, but major design changes in the middle of debugging are seldom a good idea.
While he is careful to say probably and seldom and I wholeheartedly agree with the sentiment here (it’s true in software design and well as hardware and systems) I’ve been thinking about the decision process Musk has shown.
Assuming they’d stayed with the ablative motor and had a success to orbit; then suppose the fourth flight was this third flight with the new engine and results. Then they could say they were 1 for 4 which in some ways is a silly metric. Wouldn’t they really be one flight behind where they are now? Now they can have the problem solved on the 4th flight where in this alternate scenario it wouldn’t be resolved until the 5th flight.
None of this should in anyway take away from the awesome insight shown in the article. I’m just wondering out loud.
In some ways, it is like the accounts of the Ranger program, where they were launching a series of dead TV cameras to crash land into the Moon.
It turns out that “staging” of the Atlas had something to do with it, the BECO or booster-engine cutoff, where the tankless half stage of the two booster engines get separated, followed by continued firing of the center sustainer engine.
That event would spill some non-trivial amount of rocket fuel from the disconnection of the propellant feed pipes from the booster stage. The aerodynamic conditions were such that a flash of flame would envelope the entire vehicle.
The Ranger TV camera had an exposed plug for an umbilical connection when it was on the pad. The flash of flame was enough to melt something to bring a pair of wires into contact that wrecked the TV camera. Every time. On a rather largish number of failed Ranger missions.
Or perhaps that one unmanned Saturn V test that gave the Apollo programmer managers heartburn. They had a J-2 engine on the S-IVb fail to relight. Very bad for an actual Moon mission. It turns out that cryogenic propellant went through a flexible bellows pipe connection that vibrated into metal fatigue and failure. Never happened on ground test, but on ground test, enough ice formed on the bellows to damp out the dangerous vibration. Through some positively brilliant evaluation of telemetry and flash of insight, they fixed that one after one try.
I guess the question I have is what kind of staging setup do they have? On Titan-II, they had this famous “fire in the hole” where they would light the second stage engine still connected to the first stage and I guess fire some explosive bolts for the separation — the ullage motor for second stage ignition was the first stage.
On Saturn, they didn’t do anything that crazy. They had some pretty high-thrust solid motors on the interstage ring between S-I and S-II that got them some goodly clearance between stages before along with positive G’s on the propellant in the tanks before lighting the S-II.
Both Titan and Saturn had a good light show at staging — for Titan, the destruction of the first stage of second stage ignition, for Saturn, those solid rocket motors on the interstage ring lighting off.
What are they doing on Falcon that the second stage can hardly get out of its own way? Yeah, yeah, residual thrust, but are they really counting on separating these stages by simply coasting? What provides positive separation of the stages and what settles the propellant in stage 2?
“1957 Sept. 7, 14:39: The R-7 (Number M1-10) was launched from Baikonur. Flight went normally, but the warhead section apparently collided with the core stage during the separation and disintegrated again during the reentry.”
BTW, Soyuz’s 3rd stage (Blok I) separates by ignition.
Good news that it was easy to solve and Flight 4 is coming just next month. I look forward to YASXL (Yet Another SpaceX Launch) to being boring.
Just to clarify, how would the re-contact translate into the actual failure? Did the contact itself damage the second stage, or did the second stage exhaust impinging on the too-close first stage and being directed back at the second stage cause the damage?
To answer my own question:
“There are several data points showing a proper start-up before the plasma blow-back destroyed the base of the second stage,” Musk said.
From here.
I’m wondering how this new information may impact on other configurations. The F1 uses a Kestrel engine on it’s second stage, but the F9 will use a Merlin (I assume the 1C.) The second stage will have shutdown and re-ignition events to my understanding. Are there potential problems with residual thrust during shutdown that could impact on various mission types?
My thinking is that a clean shutdown with no residual thrust is much preferred in these situations and perhaps the Merlin 1A, Kestrel or another engine might be better for the second stage?
For all those wondering how the recontact translated into failure, watch the video at the bottom of the article Rand linked, it makes it obvious to the point of…well, just obvious.
You see the separation occur, then you see the 1st stage come right back at you and literally get held against the second stage just as the second stage engine lights. You even hear the “clunk” transmitted through the rocket body as it recontacts.
I really enjoyed the video. It’s got sound all the way through, you can hear the atmospheric change as the air gets thinner. Neat stuff. Right up until that sickening “clunk” at the end. It’s a bit like when you hear a batter get hit with a pitch on TV.
The particular failure here points up one of the potential advantages of RLV testing mentioned in a previous post. An honest RLV testing program would have the first stage vehicle flying several times before attempting staging. With the shut down transients well known before an orbital attempt, it seems likely that Falcons 2 and 3 would both have made orbit.
An RLV should be able to fly many test flights compared to an ELV. The difference is that testing an RLV does not destroy a vehicle unless something goes seriously wrong. Money spent to reach market could be less for RLV development than for a fully tested ELV.
Henry Spencer says, “SpaceX is doing things the hard way.”
http://www.newscientist.com/blog/space/2008/08/spacex-rocket-failure-due-to-new-engine.html
Is he right?
He makes his argument, but I think the crux of the matter is are they getting the information they need to make the adjustments that need to be made. For that reason, focusing on whether it’s reusable or not is a red herring.
Yeah, I’d like to see some rocketplane that just needs the windows washed and the tanks filled between excursions to orbit. But until the ‘easy way’ is demonstrated I think it’s a little too early to accuse a company that appears to be making historic progress to be doing it the hard way.
john hare wrote: The particular failure here points up one of the potential advantages of RLV testing mentioned in a previous post. An honest RLV testing program would have the first stage vehicle flying several times before attempting staging. With the shut down transients well known before an orbital attempt, it seems likely that Falcons 2 and 3 would both have made orbit.
Actually John I think this failure shows the opposite. Private sector rocket development is no longer a big “what if.” The last several years have seen serious efforts by Armadillo, Masten, LBSU, SpaceX, Paul Breed, etc. We’re starting to see an actual base of private industry experience to determine rules of thumb on. What that experience shows is that universally, independent of funding profile or, suprisingly, total energy increment, serious vehicle testing with flight-worthy propulsion systems, has been a destructive process. I don’t believe any of these efforts has had even a 50% success rate in the first 5-10 launches. (Is there an example I’m missing?)
I actually fail to see an advantage to going for an RLV until you have a well-characterized propulsion system anyway, based on the evidence to date. And every time you change something major on an RLV or expendable, like an engine, you introduce new uncertainties. Just read some basic Armadillo over the last 5 years to get a flavor. Luckily they started as small as possible, as SpaceX is doing with their orbital Falcon 1 before going to the big ones.
SpaceX appears to be bleeding credibility right now, but that’s just why they’re staying private. Tough engineering problems always seem to be farther from being solved than ever… until they very suddenly break through.
Let me go ahead and predict that SpaceX will have a working reusable orbital launcher long before Blue Origin, Armadillo, or Scaled does.
I don’t believe any of these efforts has had even a 50% success rate in the first 5-10 launches. (Is there an example I’m missing?)
XCOR.
Let me go ahead and predict that SpaceX will have a working reusable orbital launcher long before Blue Origin, Armadillo, or Scaled does.
——————————————
Tom,
I’m not trying to slam SpaceX. There are several anon people doing that over at Clarks’ place. I hope they make it. I don’t think they are idiots as the anons are saying.
I need to think about replies longer than most people seem to if I am to contribute to the discussion rather than the noise. Of the companies you mentioned, most have not done launches. I think Armadillo might have a higher percentage of successes than 50% on their actual flights as opposed to tethered hardware development tests.
Your point about destructive process stands of course. They have destroyed a lot of hardware.
A lot of this destruction is by choice rather than happenstance though. They would rather (I assume) break dozens of test models under controlled conditions than one full up operational vehicle. As Rand mentioned, XCOR has yet to lose a vehicle. You make good points which we will be argueing for a few years yet.
Ken, you wrote:
Assuming they’d stayed with the ablative motor and had a success to orbit; then suppose the fourth flight was this third flight with the new engine and results. Then they could say they were 1 for 4 which in some ways is a silly metric. Wouldn’t they really be one flight behind where they are now?
No, I disagree. You need to have a successful flight so that you can test every part of the flight system. Currently, there’s still untested hardware on the Falcon 1. For example, how accurately can it place something in orbit? We don’t have a demonstration that the avionics is good enough for SpaceX’s paying customers. It’s very possible that there’s still a mission killing problem in what’s left. So we could be going for 4 failures out of four flights. Meanwhile if the ablative engine had been used, that error would be known and dealt with. SpaceX probably should have continued to use ablative engines until it had a successful flight.
And it is important for SpaceX to have successful flights somewhere along the line. Or they’ll go out of business.
The one positive aspect of AMROC’s SET-1 failure was that we proved the safety of hybrids. It just burned — it didn’t explode. There’s never a large-scale fuel/oxidizer mix that an explosion requires.
It was still very sad, though. I was there.