I missed my 7AM flight to LA from Vegas this morning, because I’m a dumbass, and went to the wrong concourse, and by the time I figured it out, it was too late to get to the right one (Note to self: in the future, look at your boarding pass). They put me on standby for the next flight that was scheduled for 1130 (and all flights are 100% full, because they’re still trying to recover from the CRWD update that had the same effect as a cyberattack). At 10, they rescheduled the flight for 1936. So I’ve now been here for ten hours, with a couple to go before I board. If I board. If not, I’ll go back to the hotel I’ve been staying and try for the 7AM flight tomorrow that I missed today.
So how’s your day going? Besides Biden ending his campaign. Or someone writing a letter ostensibly from him saying that he’s ending his campaign, without really explaining why. I wonder if he knows that he’s ending his campaign? Anyway, as Joe would say, anyway.
Thoughts from Mark Steyn.
[Monday-evening update]
Sorry for the lack of update. Yes, I caught a (slightly) earlier flight and got home last night.
[Bumped]
Now don’t you wish that we have hypersonic airships, power by microwaves, beamed from a SPS?
Yes, please.
Travel by airline kinda sucks.
I traveled across the Pacific to the US on Wednesday. Ended up arriving about 10 hours later than planned despite best efforts due to flights being rebooked by airlines (not in any way our fault) TWICE in under 12 hours, the second time after having being notified by e-mail that our reservations were cancelled with no explanation – while in flight on the first of three legs!
Finally reached our final destination at 0530.
Oh, and we are a party of 6 – my wife and I and 4 kids between 1 and 12yrs.
Now while much of this sucked, I don’t know of any better way to get here!
If the mitigation for the Crowdstrike bungle is as was described on the radio during the commute home on Friday: it involves rebooting the effected computer into Windows SAFE-MODE. Perusing into the file system for a corrupted file and deleting it by hand. Then rebooting the Windows computer normally. During invocation of Crowdstrike software at initialization it sees that the file is missing and automatically downloads a new (and corrected) copy from the Crowdstrike server and all is well again rather than a BSOD.
Because of the manual intervention step, roll-out of the fix will take some time.
What this says to me is that Crowdstrike needs to do some work on their update system. It boggles the mind they don’t use large SHA checksums of files they upload to their customers. This would allow their update system to reliably detect a corrupted file and automatically replace it or revert the update altogether.
Even if it introduced an infinite reboot cycle until a non-corrupted version of the effected file is pushed, it would be preferable to requiring manual intervention to fix, as a single push could rapidly fix the effected machines.
But to be honest, this should never have happened in the first place. I have to snicker a bit when recently banned Kapersky Labs (Russia) published a PR saying this isn’t possible with their software…
Life in the monosphere….
From what I’ve read, it wasn’t a corrupted file that caused the problem. It appears a variable had a null pointer value instead of a memory address.
Larry the null pointer had to come from somewhere. It was either a corrupted file or a file or files of complied instructions that had a bug which introduced the null pointer.
Seems clear this was an “update” that never got adequate testing. If this is the case an SHA signature would not have caught the problem.
Testing is an interesting field of study. I spent a significant fraction of my life on it. There is obviously a threshold effect. But when there is an issue that effects at scale such as this, there was obviously something that was missed.
Corruption of a file means its contents were scrambled, either partially or totally. A null pointer is a programming error. Someone, either a human programmer or some automated process, didn’t assign a value to that pointer. When the computer was trying to start, it read the value of that variable. Instead of being directed to a specified memory address, it was directed to zero (null). Bad things happen when a computer program tries to access address zero.
File corruption is usually stopped by cyclic redundancy checks in the network protocols. Programming errors are much harder to catch, although there are tools that can scan code to try to catch such mistakes.
Programming errors are much harder to catch, although there are tools that can scan code to try to catch such mistakes.
There are. Lint in the Linux world, or the commercial product called Coverity is one of the more famous name brands. I have used both at one time or another in my career. Coverity was great at spotting memory leaks. (Allocated memory that is never free’d).
But even static code analysis has it limits, esp. if the code is Object Oriented and modifies objects on the fly during execution that can subsequently lead to trouble.
Most of the work I do doesn’t lend itself well to O-O designs. Round peg, square hole type situations that are better handled algorithmically.
File corruption can lead to null pointers but it is rare that they would effect just a variable assignment and nothing else.
So, in other words, I think we are pretty much in agreement.
The situation here seems to be the BSOD occurs before Crowdstrike could detect the broken code and take preventative measures.
Such critical code areas need special attention and thorough testing.
To further clarify, the guy on the radio said “corrupted file” but didn’t say corrupted how. So in his world (IT security not necessarily software engineering), a file with a programmatic bug would also be considered “corrupted”.
Here’s an explanation of the CrowdStrike fiasco by a retired Microsoft programmer with a vast experience base (and a really first rate YouTube channel, though it is challenging). I’ve only watched it through once, and with Dave, that’s usually not enough.
I don’t know, it really looked to me like a dress rehearsal for a larger attack. The kinds of infrastructure and services whose absence would paralyze a country were shut down, though not entirely. For example, 911 was shut down…in Alaska, not a high-impact event. The New York City subway train arrival and departure signs were disabled, though the trains still ran. Delivery services, including Amazon and FedEx, were shut down. Three U.S. airlines were pretty much shut down, but not all of them. Two airports in the UK, Gatwick and Heathrow, were paralyzed, but only those. Doctor’s office computer systems were shut down – I had a cardiology appointment that day, and the office called me to say that they had to cancel because of the IT problem, and would have to call me later to reschedule. Hospitals cancelled elective surgerys, though evidently others still went ahead.
If it was, in fact, a dress rehearsal for a future attack, it is sobering. It would indicate a degree of control that is scary, given the effects it had.
A pessimist would look at this, and think that it was prelude to a pre-election attack which would cause so much social disruption that the Democrat executives would “have to” impose martial law, and cancel the elections…maybe indefinitely. But, hey, could that really happen?
Unsigned drivers. Gotta love that…
Harris is the female Ghostbusters version of Obama, except much dumber. Per Wikipedia, her mother was on the faculty of McGill but she attended Howard University, which is currently ranked 115th by US News, and then needed affirmative action to get into UC Hastings, currently ranked 82nd among law schools.
Personally, I’d make that trip by car. I figure three hours of airport time, plus flight time, is pretty close to the drive time, and you have your car.
Depending on traffic of course… but heading to Vegas Sunday morning should be a breeze. A hot Mojave breeze – you definitely want to do that one in the am this time of year.
As has been repeatedly said by the Republicans (among others) if he (Biden) is ending his campaign because he isn’t competent to run for re-election/serve 4yrs more, how can he be competent to serve as president until Jan ’25?
He isn’t the one using the powers of the President, so it doesn’t matter any more now than it did in 2021 when he was sworn in
I have seen demands for “proof of life” on comment threads.
The claim is that we haven’t seen Mr. Biden get on the tee-vee and look into the camera lense to make this announcement. It was on the Social Media Platform Formerly Known as Twitter, which could have been hacked, and “they” have President Biden zip-tied to the rails of a hospital bed while being treated for COVID?
We didn’t even get the John Gill treatment.
Paul, obviously the AI that ggggenerates *eep* the zzzzzz Biden image on TTTTTTV, was eeeEfected by the Crowd *roar* strike-three buuuuug.
The WH twitter account makes an uncharacteristic tweet, “I’m sick” Then a few days later announced Biden is withdrawing via statement that someone wrote for him. It is odd, suspicious.
During the same time, Democrats are playing all kinds of games to force him out, even writing articles claiming he said he is dropping out only to have Biden deny those claims.
Certainly looks like a rigged primary followed by a coup. Will Kamala be the candidate? Too early to say what happens at their convention but I think it is more important to see who Obama selects for VP.
…and they say Twitter/X is on the way out…
I’m sorry my dear, but I’m afraid she’s worse than the other one.
Rand, did you make it home or are you still in Vegas? A one-way car rental would be looking pretty good to me …