$5.4 Bn so far, not including lost worker productivity or damage to brand reputations, so thatâs a very conservative estimate. And Cybersecurity insurance will supposedly only cover up to 20% of that (but good luck getting even that much). What a clusterf***
And that $5,400,000,000 loss estimate is only Fortune 500 companies!
11111one11111@lemmy.world
on 25 Jul 2024 18:23
collapse
No itâs all of them because all the companies combined out side of the 500 wouldnât even have enough net worth large enough to move the needle. So technically they may not be included but would be covered by whatever amount they rounded up to make the even 5.4b
All the CrowdStrike companies on earth minus the 500 biggest (American) ones? I have a hard time believing itâs as insignificant as you assume. I guess weâll seeâŚ
flatlined@lemmy.dbzer0.com
on 26 Jul 2024 10:48
collapse
Itâs a variation on the old saw of âhow much is the difference between a million and a billion? About a billionâ.
Once numbers become so big, itâs hard to grasp the relative sizes.
That said, Iâm also interested in a more comprehensive breakdown. Seeing who are impacted, how much and where.
11111one11111@lemmy.world
on 27 Jul 2024 01:44
collapse
100% correct. I wasnât implying that I knew the figures just that the size of the Fortune 500 is used as an economic index for this reason.
0x0@programming.dev
on 24 Jul 2024 17:53
nextcollapse
On Wednesday, CrowdStrike released a report outlining the initial results of its investigation into the incident, which involved a file that helps CrowdStrikeâs security platform look for signs of malicious hacking on customer devices.
The company routinely tests its software updates before pushing them out to customers, CrowdStrike said in the report. But on July 19, a bug in CrowdStrikeâs cloud-based testing system â specifically, the part that runs validation checks on new updates prior to release â ended up allowing the software to be pushed out âdespite containing problematic content data.â
âŚ
When Windows devices using CrowdStrikeâs cybersecurity tools tried to access the flawed file, it caused an âout-of-bounds memory readâ that âcould not be gracefully handled, resulting in a Windows operating system crash,â CrowdStrike said.
Couldnât it, though? đ¤
And CrowdStrike said it also plans to move to a staggered approach to releasing content updates so that not everyone receives the same update at once, and to give customers more fine-grained control over when the updates are installed.
I thought they were already supposed to be doing this?
Plopp@lemmy.world
on 25 Jul 2024 06:30
nextcollapse
Couldnât it, though? đ¤
IANAD and AFAIU, not in kernel mode. Things like trying to read non existing memory in kernel mode are supposed to crash the system because continuing could be worse.
chaospatterns@lemmy.world
on 25 Jul 2024 18:26
collapse
They could and clearly they should have done that but hindsight is 20/20. Software is complex and thereâs a lot of places that invalid data could come in.
whatwhatwhatwhat@lemmy.world
on 25 Jul 2024 15:18
nextcollapse
The fact that they werenât already doing staggered releases is mind-boggling. I work for a company with a minuscule fraction of CrowdStrikeâs user base / value, and even we do staggered releases.
foggenbooty@lemmy.world
on 25 Jul 2024 19:26
collapse
They do have staggered releases, but itâs a bit more complicated. The client that you run does have versioning and you can choose to lag behind the current build, but this was a bad definition update. Most people want the latest definition to protect themselves from zero days. The whole thing is complicated and a but wonky, but the real issue here is cloudflareâs kernel driver not validating the content of the definition before loading it.
whatwhatwhatwhat@lemmy.world
on 26 Jul 2024 05:36
collapse
Makes sense that it was a definitions update that caused this, and I get why thatâs not something youâd want to lag behind on like you could with the agent. (Putting aside that one of the selling points of next-gen AV/EDR tools is that theyâre less reliant on definitions updates compared to traditional AV.) Itâs just a bit wild that there isnât more testing in place.
Itâs like weâre always walking this fine line between âsecurity at all costsâ vs âstability, convenience, etcâ. By pushing definitions as quickly as possible, you improve security, but youâre taking some level of risk too. In some alternate universe, CS didnât push definitions quickly enough, and a bunch of companies got hit with a zero-day. Iâd say itâs an impossible situation sometimes, but if I had to choose between outage or data breach, Iâm choosing outage every time.
cheddar@programming.dev
on 26 Jul 2024 06:15
nextcollapse
The company routinely tests its software updates before pushing them out to customers, CrowdStrike said in the report. But on July 19, a bug in CrowdStrikeâs cloud-based testing system â specifically, the part that runs validation checks on new updates prior to release â ended up allowing the software to be pushed out âdespite containing problematic content data.â
It is time to write tests for tests!
Passerby6497@lemmy.world
on 26 Jul 2024 11:48
collapse
My thoughts are to have a set of machines that have to run the update for a while, and if any single machine doesnât pass and all allow it to move forward, it halts any further rollout.
a bug in CrowdStrikeâs cloud-based testing system
Always blame the tests. There are so many dark patterns in this industry including blaming qa for being the last group to touch a release, that I never believe âitâs the testsâ.
Thereâs usually something more systemic going on where something like this is missed by project management and developers, or maybe they have a blind spot that it will never happen, or maybe thereâs a lack of communication or planning, or maybe they outsourced testing to the cheapest offshore providers, or maybe everyone has huge time pressure, but âitâs the testsâ
Ok, maybe Iâm not impartial, but when Iâm doing a root cause on how something like this got out, my employer expects a better answer than âitâs the testsâ
aStonedSanta@lemm.ee
on 26 Jul 2024 11:47
collapse
There was probably one dude at CrowdStrike going. Uh hey guys??? đ
Imgonnatrythis@sh.itjust.works
on 24 Jul 2024 20:59
nextcollapse
âCrowdStrike said it also plans to move to a staggered approach to releasing content updates so that not everyone receives the same update at once, and to give customers more fine-grained control over when the updates are installed.â
Hol up. So they like still get to exist? Microsoft and affected industries just gonna kinda move past this?
BakerBagel@midwest.social
on 24 Jul 2024 21:08
nextcollapse
Havenât seen anything from the affected major players. Obviously Crowdstrike isnât going to say they are fucked long term, they have to act like this is just a little hiccup and move on. Lawsuits are absolutely incoming
Ledivin@lemmy.world
on 24 Jul 2024 23:30
nextcollapse
Weâll see how fucked they are from SLA breaches/etc., and then weâll see how many companies jump ship to an alternative. We wonât have the real fallout from this event for months or years.
LodeMike@lemmy.today
on 25 Jul 2024 07:23
nextcollapse
Companies using CrowdStrike and Windows arenât really the type to be active about this sort of thing.
11111one11111@lemmy.world
on 25 Jul 2024 18:19
collapse
What do you mean by this?
LodeMike@lemmy.today
on 25 Jul 2024 18:57
collapse
The companies who use CrowdStrike (lazy fix) on Windows (garbage OS) arenât really the type to want to switch away from it (will take effort)
best_username_ever@sh.itjust.works
on 26 Jul 2024 10:49
collapse
I donât understand the downvotes. Youâre right on all points. If the task is too big, it can take years from testing another solution to using it for real.
Modern_medicine_isnt@lemmy.world
on 25 Jul 2024 17:01
nextcollapse
Newsflash, Solarwinds still exists too. Not sure I could name a company that screwed up so big and actually paid the price.
Imgonnatrythis@sh.itjust.works
on 25 Jul 2024 17:14
nextcollapse
Yeah, what was I thinking. United airlines was bankrupt and literally beating people up on their planes and still got taxpayer payouts and is around paying investors divends still today.
TheLimiter@lemmy.world
on 26 Jul 2024 11:21
collapse
Two days ago my company sent out an all hands email that weâre going company wide with Crowdstrike.
A CTO is at lunch when a call comes in. Thereâs been a huge outage, caused by a low level employee pressing the wrong button.
âDamn, you going to fire that guy?â
âHell no, do you know how much I just spent on training him to never do that again?â
(</Blah>)
essteeyou@lemmy.world
on 25 Jul 2024 00:14
nextcollapse
Oh, finally, I have been waiting for so long.
Semi_Hemi_Demigod@lemmy.world
on 25 Jul 2024 19:38
nextcollapse
For the rest of history this sort of thing will mention Crowdstrike, or it might even be called a âcrowdstrike.â
You canât buy that kind of marketing
Wispy2891@lemmy.world
on 26 Jul 2024 06:34
nextcollapse
This crowdstrike stuff seems an expensive subscription
I saw a lot of photos of crashed ad screens.
Why the hell are corps paying this much money for windows+cloudstrike for a glorified digital picture frame?? Wouldnât be 100x cheaper to do it with some embedded stuff instead of having a full desktop computer running a full desktop os???
sugar_in_your_tea@sh.itjust.works
on 26 Jul 2024 13:46
collapse
Yeah, an RPi or similar with a screen would be more than plenty for this, and the Pi Zero is really small. Connect that to a central Linux server with a hot backup or two (through local DNS) and youâll have a hard time crashing it.
unexpectedteapot@lemmy.ml
on 26 Jul 2024 10:07
nextcollapse
Do we actually know? We might know that Crowdstrike was the cause but we donât actually know what went wrong and how it happened. It is an unfree proprietary closed source software, we just have to take their word for it, which for all purposes is PR in line with the fact that it is coming from a profit-driven organisation.
lightsblinken@lemmy.world
on 26 Jul 2024 14:27
collapse
this is exactly the question that needs answering⌠the PIR is bullshit
riodoro1@lemmy.world
on 26 Jul 2024 10:22
nextcollapse
Ok. Can we get a solar storm next? I want linux servers out this time too.
sugar_in_your_tea@sh.itjust.works
on 26 Jul 2024 13:44
collapse
Best I can do is an xz vuln where half the Linux servers go down for maintenance.
JasonDJ@lemmy.zip
on 26 Jul 2024 11:37
nextcollapse
Pretty soon we are gonna have to start deciding if itâs safer for enterprise computers to run without AV or AMP.
threaded - newest
And the stockades?
Any word on the stockades?
George Kurtz has only crashed the world twice so he has one strike to go, I guess.
Wowowow! This is insane! đ¨đ¤Ż
You can only fail upwards at the executive level. He went from CTO to CEO on his last global crash. Whatâs next? Running for President?
No risk, All rewards.
Please no
$5.4 Bn so far, not including lost worker productivity or damage to brand reputations, so thatâs a very conservative estimate. And Cybersecurity insurance will supposedly only cover up to 20% of that (but good luck getting even that much). What a clusterf***
And that $5,400,000,000 loss estimate is only Fortune 500 companies!
No itâs all of them because all the companies combined out side of the 500 wouldnât even have enough net worth large enough to move the needle. So technically they may not be included but would be covered by whatever amount they rounded up to make the even 5.4b
All the CrowdStrike companies on earth minus the 500 biggest (American) ones? I have a hard time believing itâs as insignificant as you assume. I guess weâll seeâŚ
Itâs a variation on the old saw of âhow much is the difference between a million and a billion? About a billionâ. Once numbers become so big, itâs hard to grasp the relative sizes. That said, Iâm also interested in a more comprehensive breakdown. Seeing who are impacted, how much and where.
100% correct. I wasnât implying that I knew the figures just that the size of the Fortune 500 is used as an economic index for this reason.
.
âŚ
Couldnât it, though? đ¤
I thought they were already supposed to be doing this?
IANAD and AFAIU, not in kernel mode. Things like trying to read non existing memory in kernel mode are supposed to crash the system because continuing could be worse.
I.meant couldnât they test for a NULL pointer.
They could and clearly they should have done that but hindsight is 20/20. Software is complex and thereâs a lot of places that invalid data could come in.
The fact that they werenât already doing staggered releases is mind-boggling. I work for a company with a minuscule fraction of CrowdStrikeâs user base / value, and even we do staggered releases.
They do have staggered releases, but itâs a bit more complicated. The client that you run does have versioning and you can choose to lag behind the current build, but this was a bad definition update. Most people want the latest definition to protect themselves from zero days. The whole thing is complicated and a but wonky, but the real issue here is cloudflareâs kernel driver not validating the content of the definition before loading it.
Makes sense that it was a definitions update that caused this, and I get why thatâs not something youâd want to lag behind on like you could with the agent. (Putting aside that one of the selling points of next-gen AV/EDR tools is that theyâre less reliant on definitions updates compared to traditional AV.) Itâs just a bit wild that there isnât more testing in place.
Itâs like weâre always walking this fine line between âsecurity at all costsâ vs âstability, convenience, etcâ. By pushing definitions as quickly as possible, you improve security, but youâre taking some level of risk too. In some alternate universe, CS didnât push definitions quickly enough, and a bunch of companies got hit with a zero-day. Iâd say itâs an impossible situation sometimes, but if I had to choose between outage or data breach, Iâm choosing outage every time.
It is time to write tests for tests!
My thoughts are to have a set of machines that have to run the update for a while, and if any single machine doesnât pass and all allow it to move forward, it halts any further rollout.
Always blame the tests. There are so many dark patterns in this industry including blaming qa for being the last group to touch a release, that I never believe âitâs the testsâ.
Thereâs usually something more systemic going on where something like this is missed by project management and developers, or maybe they have a blind spot that it will never happen, or maybe thereâs a lack of communication or planning, or maybe they outsourced testing to the cheapest offshore providers, or maybe everyone has huge time pressure, but âitâs the testsâ
Ok, maybe Iâm not impartial, but when Iâm doing a root cause on how something like this got out, my employer expects a better answer than âitâs the testsâ
There was probably one dude at CrowdStrike going. Uh hey guys??? đ
âCrowdStrike said it also plans to move to a staggered approach to releasing content updates so that not everyone receives the same update at once, and to give customers more fine-grained control over when the updates are installed.â
Hol up. So they like still get to exist? Microsoft and affected industries just gonna kinda move past this?
Havenât seen anything from the affected major players. Obviously Crowdstrike isnât going to say they are fucked long term, they have to act like this is just a little hiccup and move on. Lawsuits are absolutely incoming
Weâll see how fucked they are from SLA breaches/etc., and then weâll see how many companies jump ship to an alternative. We wonât have the real fallout from this event for months or years.
Companies using CrowdStrike and Windows arenât really the type to be active about this sort of thing.
What do you mean by this?
The companies who use CrowdStrike (lazy fix) on Windows (garbage OS) arenât really the type to want to switch away from it (will take effort)
I donât understand the downvotes. Youâre right on all points. If the task is too big, it can take years from testing another solution to using it for real.
Newsflash, Solarwinds still exists too. Not sure I could name a company that screwed up so big and actually paid the price.
Yeah, what was I thinking. United airlines was bankrupt and literally beating people up on their planes and still got taxpayer payouts and is around paying investors divends still today.
Two days ago my company sent out an all hands email that weâre going company wide with Crowdstrike.
Nows the time to sign up. Theyâll slash prices and hopefully never fuck up this bad again.
Have we had a XaaS fuck up real, real bad, twice, yet?
I wasnât effected but I bet a lot of admins, as pissed as they were, were thinking âI could easily fuck up this bad or worseâ.
Yeah, whatâs the jokey parable thing?
(</Blah>)
Oh, finally, I have been waiting for so long.
For the rest of history this sort of thing will mention Crowdstrike, or it might even be called a âcrowdstrike.â
You canât buy that kind of marketing
This crowdstrike stuff seems an expensive subscription
I saw a lot of photos of crashed ad screens.
Why the hell are corps paying this much money for windows+cloudstrike for a glorified digital picture frame?? Wouldnât be 100x cheaper to do it with some embedded stuff instead of having a full desktop computer running a full desktop os???
Yeah, an RPi or similar with a screen would be more than plenty for this, and the Pi Zero is really small. Connect that to a central Linux server with a hot backup or two (through local DNS) and youâll have a hard time crashing it.
.
Do we actually know? We might know that Crowdstrike was the cause but we donât actually know what went wrong and how it happened. It is an unfree proprietary closed source software, we just have to take their word for it, which for all purposes is PR in line with the fact that it is coming from a profit-driven organisation.
this is exactly the question that needs answering⌠the PIR is bullshit
Ok. Can we get a solar storm next? I want linux servers out this time too.
Best I can do is an xz vuln where half the Linux servers go down for maintenance.
Pretty soon we are gonna have to start deciding if itâs safer for enterprise computers to run without AV or AMP.
Beautiful