At least a few 1’s I imagine.

Well, you see, the front fell off.

Ah, makes sense. I guess a driver would completely freak out if that file gave no instructions and was just like “…”

Great layman’s explanation.

Maybe. But I’d like to think I’d just say something clever like, “says here that this year the pummel horse will be replaced by yours truly!”

Except “freak out” could have various manifestations.

In this case it was “burn down the venue”.

It should have been “I’m sorry, there’s been an issue, let’s move on to the next speaker”

Nice analogy, except you’d check the script before you tried to use it. Computers are really good at crc/hash checking files to verify their integrity, and that’s exactly what a privileged process like antivirus should do with every source of information.

I’m nominating this for the “best metaphor of the day” award.

Well done!

The funny bit is, I’m sure more than a few people at Crowdstrike are preparing 3 envelopes right now.

This guy ELI5s

Ah yes. So Windows is the screaming in terror version and other systems are the “oh, sorry everyone, looks like there’s an error. Let’s just move on to the next bit” version.

I disagree. I’d rather things crash than silently succeed or change the computation. They should have done better input and output validation, and gracefully fail into a recoverable state that sends a message to an admin to correct. A divide by zero doesn’t crash a system, it’s a recoverable error they should 100% detect and handle, hot sweep under the rug.

IFERROR(;0)

Maybe they should use a more appropriate development tool for their critical security platform than Excel.

The front of the file fell off

This was not the driver, it was a config file or something read by the driver. Now having a driver in kernel space depending on a config on a regular path is another fuck up

What about the Mac and Linux PCs? Did Microsoft sign those too?

the smart ones probably do

I mean yes, but one of the issuess with “state of the art av” is they are trying to roll out updates faster than bad actors can push out code to exploit discovered vulnerabilities.

The code/config/software push may have worked on some test systems but MS is always changing things too.

Companies don’t like to be beta testers. Apparently the solution is to just not test anything and call it production ready.

That’s certainly what we do in my workplace. Shocked that they don’t.

When I worked at a different enterprise IT company, we published updates like this to our customers and strongly recommended they all have a dedicated pool of canary machines to test the update in their own environment first.

I wonder if CRWD advised their customers to do the same, or soft-pedaled the practice because it’s an admission there could be bugs in the updates.

I know the suggestion of keeping a stage environment was off putting to smaller customers.

this is what I suspect also. I mean it’s easy to point fingers at George Kurtz as he was CTO at Mcafee when they had their “little” snafu but…well…yeah. I strongly suspect many of his “policies” he had while CTO at Mcafee carried over to Crowdstrike. dude isn’t exactly known for being a fan of testing or vetting processes. in fact he’s all about quick development/crunch.

“We are confident that only our engineers can fuck up so much, instead of our competitors”

“Foreign” in this context just means “not Crowdstrike”, not like a foreign government.

Holy hell

Oooooooo this one again thank you for reminding me

That one turns out to have been largely Microsoft’s fault for repeatedly ignoring warnings of a severe vulnerability relating to Active Directory. Microsoft were warned about it, acknowledged it and ignored it for years until it got used in the Solar Winds hack.

lesson not learned

You mean it’s going to cost corporations a pretty penny. Which means they’ll pass those “costs of operation” on to the rest of us. Fuck.

Where’s my fuckin raise

All the more reason for companies to ignore security until they’re affected personally. The companies I’ve worked for barely ever invested in future cost-savings.

That trick doesn’t work for B2B as organizations tend to do their research before buying. Consumers tend not to.

I don’t think they will lose any big clients. I am sure they will have insurance to take care of compensations.

I thought it was a security definition download; as in, there’s nothing short of not connecting to the Internet that you can do about it.

Thank God someone else said it. I was constantly in an existential battle with IT at my last job when they were constantly forcing updates, many of which did actually break systems we rely on because Apple loves introducing breaking changes in OS updates (like completely fucking up how dynamic libraries work).

Updates should be vetted. It’s a pain in the ass to do because companies never provide an easy way to rollback, but this really should be standard practice.

Years ago I read an study about insurance companies and diversification of assets in Brazil. By regulation, an individual insurance company need to have a diversified investment portfolio, but the insurance market as a whole not. the diversification of every individual company sum, as a whole of all the insurance market, as an was exposed market, and the researchers found, iirc, like 3 banks that if they fail they can cause a chain reaction that would take out the entire insurance market.

Don’t know why, but your comment made me remind of that.

Properly regulated capitalism breaks up monopolies so new players can enter the market. What you’re seeing is dysfunctional capitalism - an economy of monopolies.

.

Apply the ones in a star shape to distribute pressure evenly

I mean, joking aside, isn’t that how parity calculations used to work? “Got more uppy bits than downy bits - that’s a paddlin’” or something.

I once had a very noisy computer… It was the disk! I just balanced the data and all problems went away.

Perfectly balanced

At least SELinux doesn’t crash on bad config file

crowdstrike has caused issues like this with linux systems in the past, but sounds like they have now moved to eBPF user mode by default (I don’t know enough about low level linux to understand that though haha), and it now can’t crash the whole computer. source

You mean you don’t update your kernel?

with great power, don’t lay off the testing team (force return to office or get fired ultimatums)

This was a binary configuration file of some sort though?

Something along the lines of:

IF (config.parameter.read == garbage) {
     Dont_panic;
}

Would have helped greatly here.

Edit: oh it’s more like an unsigned binary blob that gets downloaded and directly executed. What could possibly go wrong with that approach?

Could also be that the Windows versions they tested on weren’t as problematic as the updated drivers around the time they released.