First (as he told me last week he planned to do), Charles Miller rolled out his proposal for a return to the moon, using commercial launchers, at the National Press Club. I haven’t read it yet, but I’m sure it will contain a lot of good info to inform my own project.
Second, Elon had a press telecon to present preliminary findings on the accident. It was apparently a broken strut on the helium tank in the LOX tank, which failed at 20% of the rated strength (it seems to have failed in tension). Return to flight no sooner than September, depending on customer willingness to fly, redesign of strut (and new supplier). Falcon Heavy first flight delayed until Q2 next year. He admitted that it may have occurred due to “complacency” after long string of successful flights. Most current employees had never seen a failure.
[Update a few minutes later]
Alan Boyle (who seems to have retired from NBC, congratulations) has the story at Geekwire.
[Tuesday-morning update]
Here’s a description of what went wrong from the Space Access Society.
[Update a few minutes later]
Here’s Stephen Clark’s takehttp://spaceflightnow.com/2015/07/20/support-strut-probable-cause-of-falcon-9-failure/ over at SpaceflightNow.
Where is the link to Charles Miller’s website? I think that we should return to the Moon with commercial launchers. I think that NASA should purchase launch flights to the Moon, and rent base facilities on the Moon.
“In the future, every strut will be individually tested.”
OK, now what about everything that isn’t a strut? Some of you may recall my talk on mission assurance from the latest Space Access, but for those who didn’t make it, a recurring theme is that if you can’t afford to lose the rocket – and SpaceX certainly can’t afford to lose another rocket any time soon – you test everything. Not just the part that failed last time, but all the parts that will kill you if they break next time.
If you are dealing with an established aerospace supplier, you can maybe get away with reviewing the paperwork from their tests. Maybe. But anyone who doesn’t do aerospace or milspec work – and I think that’s most of Elon’s supplier base – is in the business of selling stuff that occasionally breaks and saying “Yeah, OK, we’ll give you a refund on that one”.
There’s no requirement that this be prohibitively expensive. It can be, if you’re interested in padding a cost-plus contract. But if you can’t figure out a cheap, simple way to verify that a strut will hold the specified load, that a tank won’t burst, that a length of wire will carry current and hold voltage, go ahead and get a refund on your engineering degree. Because the universe does impose a requirement that you get the tests done, and it will keep blowing up your rockets if you don’t.
You can test every part. Or you can establish the processes to ensure every part is built to the required specifications. For a part to fail at 20% of the design load, something radically wrong had to take place in the production process.
As I understand it, SpaceX make most of the critical parts themselves. I think we can be pretty sure they don’t make them to fail at 20% of their design load. They may need to do more testing on hardware they buy in from third parties, but I don’t see why they’d need to do more testing on those they make internally.
Besides, in future, when they’re reusing stages, they’ll know they’re likely to work since the same stage already flew ten times without failure. It’s not like they’re going to take the stage apart before every launch and test it all again.
> “In the future, every strut will be individually tested.”
> OK, now what about everything that isn’t a strut? Some of you may recall my talk on mission assurance from the latest Space Access, but for those who didn’t make it, a recurring theme is that if you can’t afford to lose the rocket – and SpaceX certainly can’t afford to lose another rocket any time soon – you test everything. Not just the part that failed last time, but all the parts that will kill you if they break next time.<
Would you consider it fair (or at least not totally unfair) to say that's the approach that the EELV program has taken, that's produced something over a hundred flights in a row with no mission loss?
Also, alas, the approach that has left ULA with costs high enough that their per-flight price is several times higher than SpaceX's. Oversimplified, yes, but in general?
What SpaceX seems to be trying to do is trim back the test-EVERYTHING overhead in a way that retains reliability while drastically cutting costs. A very tough balance to strike, I'm sure we would agree. (My understanding is that the test-everything quality approach evolved over time in response to repeated incidents where someone messed up and let bad parts or procedures slip past.)
I tend to agree with SpaceX's overall goal and general approach, FWIW. What I think we're seeing here is an instance of failing to set the balance properly.
Given a design where single He-bottle holddown struts are fairly obviously single-point loss-of-mission failures, given the relative ease and cheapness of setting up a calibrated pull-test rig to see they all can handle 1.5x flight loads without nearing their 3x design limit (and without cracking or stretching that might indicate an actual strength of 1.6x), given the odds against proper steel metallurgy ALWAYS being correctly present (as opposed to almost always) in any bulk-purchase part no matter what the certs, were I the F9 MFWIC, it would seem obvious to me that adding pull-test to incoming strut inspections is cost-effective. So, someone has to spend an hour per-rocket putting a couple dozen struts through a test rig then marking them "tested". Not a big cost impact for a $50M product.
Where the question gets interesting is, do you do the same thing for ALL the "couple hundred" similar struts (and attach points, and attach bolts) in each vehicle? (I infer they use these struts for more than just the He bottles.) Ease and cheapness of testing still seems a given, but that's absent knowledge of specific uses. Are they all as single-point critical as the He-bottle holddowns though?
Multiply that by every part in the rocket, some likely far harder to test than a relatively simple tension strut, and I think what we'll see is a modest dialing back of the "failure is not imaginable" complacency Musk alluded to at SpaceX rather than a wholesale adoption of "test everything".
Interesting thought: If they get reusability going routinely, I could see a reliability discount eventually developing for *first* flights of a given stage, with peak pricing for already flight-tested hardware.
Even as an oversimplification, I don’t think it is fair to say that ULA is building a $50 million rocket and then putting $150 million of mostly-unnecessary testing into it. Testing is generally cheaper than manufacturing, but ULA works cost-plus and so they figure out how to spend three times what they need for each. By this point, are so deeply in the habit that they probably won’t be able to break it when their corporate survival depends on it.
But if you’re shooting for say four nines of reliability (which if SpaceX uses a hundred of these struts per F9, means a 1%/flight loss rate from broken struts alone), you probably don’t get it by specifying some perfect blind manufacturing process that gets you 99.99% reliability straight off the production line. As you say, that’s unrealistic for anything involving real-world metallurgy. But 99% manufacturing reliability and a test that catches 99% of the failures? That’s picking the low-hanging fruit at both ends rather than trying to push one to ultimate, expensive perfection.
Tweak the numbers as needed if it turns out you need three nines or five nines or whatnot of part-level reliability. And maybe your engineering judgement says for this part you shoot for 99.9% manufacturing yield and a cheap 90% screen to get four nines reliability. I think the only time you’d really go with no testing, is if you have enough redundancy that you are willing to assume one failure out of the box, and you are using different manufacturing lots (or designs) for each redundant string.
To address a few other points: stock parts, expendables, and raw materials you do lot acceptance testing. Cut the first fifty feet of wire off each new spool, hook it to a power supply and see how far you can turn the dial before you see smoke (crudely speaking). If it’s a high enough value, staple the test results to the spool and you’re good.
Yes, you need to test the parts you built yourself. More precisely, you need someone else to test them. The guy who builds stuff, always overestimates how well he built it. You need a separate test engineer.
“Confidence in your supplier” means if they stapled a copy of their own test results to the spool of wire, you don’t need to redo the test. There’s no manufacturer I’d be so confident in that their brand name on the box would assure me the tests had been done. Even major aerospace and milspec suppliers have let us down too often in that regard, and I know SpaceX is stepping a level down in the supply chain by using automotive hardware in at least some applications. Ask for the actual test results, or a discount.
MIL-STD-1540D, or -E if you can get an advance copy, is a good guide to the testing you want to do. It isn’t a matter of just throwing out every test anyone imagined and seeing what would stick; that was plain 1540. The later versions have real experience behind them. But they are meant to be tailored by engineering judgement, not applied by rote.
And true reuseability is going to make things interesting in ways we haven’t yet figured out. On the one hand, flight is the ultimate test experience. On the other hand, if you fly one time too many you will inevitably blow up a worn-out rocket. What sort of testing you need to do, and when, to know what needs replacing and when the bird should be retired, that’s going to require a lot of engineering judgement, analysis, extra testing up front, and probably some spectacular explosions as learning experiences.
I’d take Musk’s mention of unwarranted complacency building up among the (company majority now) new hires since SpaceX’s last vehicle loss seven years ago, plus the facts he presented, to say that someone somewhere in SpaceX assumed that the strut’s design plus a manufacturer’s cert of correct metallurgy provided significantly more nines than it actually did.
Figure, as you say, a hundred such struts in each F9 that are individually flight-critical, and .9999 per-strut reliability means you lose one flight in a hundred to the struts alone. I would guess they convinced themselves they had at least one and more likely several more nines… There’s a hint as to this thinking in Musk’s mention that they only tested a modest batch of the struts in inventory initially; reading between the lines there was a bit of a painful conceptual leap involved in testing really large numbers of the things – some thousands is my impression – to find a handful well below spec and at least one bad enough to have failed.
One specific takeaway is that the actual reliability of that design-plus-metallurgy-cert approach was on the rough order of one in two-to-five thousand, IE .9995-.9998, for a flight failure rate from that cause on the VERY rough order of 2% to 5%. As Popeye would say, “how embarraskin’…” I’d want a few more nines, which partially explains why Musk said they’re both redesigning and adding individual testing. And the embarrassment factor – I seriously doubt anyone has enjoyed explaining this to the various customers – probably explains at least one more nine added…
My meta-takeaway is something I’ve seen before elsewhere: Engineers are occasionally prone to embrace their glorious vision of how a system *should* work to the exclusion of alternate visions of how it might fail. The best engineers are professional paranoids who spend a lot of mental energy envisioning how their baby might break, and forestalling these. The rest benefit from a knowledgable outsider to play Devil’s Advocate. (They often hate it – but they benefit.)
John I would at a minimum hope that SpaceX suppliers have at least MILSPEC experience. Granted SpaceX is a vertically integrated company but you can take that only so far. At some point you have to able to rely on the competence of your suppliers. You can’t build out to test every screw, every bolt, nut, wire, electronic component etc. I would think that yes, good paperwork is required. But you have to have some degree of confidence in your suppliers no?
Dave
Did anyone say if that strut was composite?
Just saying that is probably your problem. That will be the same problem people will discover when everyone starts 3D printing engineered parts. A part that is assembled particle by particle could have some of those particles poorly attached without looking any different from a correctly built part. Especially if you are pushing your engineers to work 60+ hours a week, like they do at Space-X.
It was probably stainless steel. Most LOX-system components are.
Steel is what Musk said. He also said that what the vendor was certifying was the material’s metallurgy. I listened carefully; there were no further relevant details in the telecon.
Take a look at the discussion of steels in cryo applications at http://steel.keytometals.com/Articles/Art61.htm and it’s clear that either the wrong alloy or incorrect processing – heat-treatment, cold-working, etc, depending on the alloy – could cause exactly this sort of problem in cryo apps.
Further, depending on the material chosen, testing might need to be done at cryo temps to be meaningful. (Not impossible, or even necessarily expensive – LN2 is cheap and not that hard to handle – but a complication.)
Elsewhere I saw it described as a “steel strut”. But one of the above articles mentions a failed bolt.
It would be nice if they would provide a picture or diagram of the helium tank mounting structure.
.
September is two months or eight weeks. I said it would be weeks.
SpaceX continues to meet my expectations. I suppose those claiming return to flight would take many months are also meeting expectations. They, for reasons I do not understand, continue to hope things go badly.
Previous launch webcasts included video from a camera inside the second stage LOX tank (at main engine cut-off the floating LOX would create trippy “stargate” imagery). I’ve read that CRS-7 did not have such a camera, and even if it had the accident might have happened too quickly to leave a video record of the event (depending on the latency of the camera and video telemetry system). It sounds like it hasn’t been easy to pin down the cause of the accident, and I assume that the investigators wish they had more direct evidence. I wonder whether the LOX tank camera will be back on the next launch, and what other sorts of additional telemetry might be added to future rockets.
Infant mortality. As Rand likes to say, IIRC: Every vehicle is a new vehicle and every flight is it’s first flight.
The ultimate solution is full reusability.
That analogy loses me. Of course there are the financial benefits of reuse, but that analogy isn’t about that.
Every reusable has a first flight. Thereafter it has fatigue.
So does an airplane.
The failure points tend to cluster at the beginning and the end, the end point may be further down the road than you think.
It’s really easy to add more process to address failure. It’s harder but ostensibly better to redesign to be robust against failures.
People who care about designing for cost should be deeply skeptical when “add more process” is proposed as a solution to anything. I’m worried about the process creep SpaceX could experience after this.