CrowdStrike broke Debian and Rocky Linux months ago, but no one noticed (www.neowin.net)
from hal_5700X@sh.itjust.works to technology@lemmy.world on 22 Jul 2024 01:39
https://sh.itjust.works/post/22584531

#technology

threaded - newest

viking@infosec.pub on 22 Jul 2024 01:54 next collapse

To avoid such issues in the future, CrowdStrike should prioritize rigorous testing across all supported configurations.

Bold of them to assume there’s a future after a gazillion off incoming lawsuits.

finley@lemm.ee on 22 Jul 2024 02:02 next collapse

I was listening to a podcast earlier, and they mentioned the fact that their legal liability may, in fact, be limited because of specific wording in most of their contracts.

In other words, they may actually get away with this in the short term. In the long-term, however, a lot of organizations and governments that were hit by this will be reevaluating their reliance on such monolithic tech solutions as crowdstrike, and even Microsoft.

So you may be right, but not for the reasons you think.

rumschlumpel@feddit.org on 22 Jul 2024 02:07 next collapse

and even Microsoft

(x) doubt

They had decades to consider Microsoft a liability. Why start doing something about it now?

catloaf@lemm.ee on 22 Jul 2024 02:20 next collapse

Because cybersecurity is becoming more of a priority. The US government has really put their attention on it in the last few years.

Maeve@kbin.earth on 22 Jul 2024 02:38 next collapse

Hard to tell, sometimes.

Tinidril@midwest.social on 22 Jul 2024 03:01 collapse

I was in IT back in 2001 when the Code Red virus hit. It was a very similar situation where entire enterprises in totally unrelated fields were brought down. So many infected machines were still trying to replicate that corporate networks and Internet backbone routers were getting absolutely crushed.

Prior to that, trying to get real funding for securing networks was almost impossible. Suddenly security was the hottest topic in IT and corporations were throwing money at all the snake oil Silicon Valley could produce.

That lasted for a couple years, then things started going back to business as usual. Microsoft in particular was making all sorts of promises and boasts about how they made security their top priority, but that never really happened. Security remained something slapped on at the end of product development and was never allowed to interfere with producing products demanded by marketing with inherently insecure designs.

xyguy@startrek.website on 22 Jul 2024 04:25 collapse

You’re absolutely right. Everyone will be very worried and talk about the importance of security in the enterprise and yada yada yada until a cool new AI spreadsheet software comes out and everybody forgets to even check if their firewall is turned on.

But with that being said, if you have been looking for a good time to ask for cybersecuity funding at your org, see if you can’t lock down 5 years worth of budget while everyone is aware of the risk to their businesses.

Maeve@kbin.earth on 22 Jul 2024 02:37 collapse

Literally lol'd. Thanks for that!

Brkdncr@lemmy.world on 22 Jul 2024 02:49 next collapse

Contracts aren’t set in stone. Not only are those contracts modified before they are accepted by both parties, it’s difficult to limit liability when negligence is involved. CS is at worst going to be defending against those, at best defending against people dumping them ahead of schedule against their contracted term length.

TheBat@lemmy.world on 22 Jul 2024 12:05 collapse

Oh so you can fire QA department, get absolutely destructive update to millions of systems across the globe and this gross negligence doesn’t matter because of magic words in a contract? I don’t think so.

finley@lemm.ee on 22 Jul 2024 14:09 collapse

That’s not what I said

TheBat@lemmy.world on 22 Jul 2024 15:03 collapse

Then how else is their legal liability is limited?

They killed off their QA department to chase profits which resulted in a broken product that crippled hundreds of organizations across the globe.

They don’t get to just shrug, say oopsie, and point at the contract.

mipadaitu@lemmy.world on 22 Jul 2024 05:35 collapse

They mean after Crowdstrike gets sold, the new company promises a more rigorous QA, and quietly rebrands it.

captain_aggravated@sh.itjust.works on 22 Jul 2024 06:36 next collapse

Slorp is now Bonto!

Default_Defect@midwest.social on 22 Jul 2024 08:16 next collapse

Cloudstrike, wait no!

derpgon@programming.dev on 22 Jul 2024 14:12 collapse

What are you doing Counterstrike

bitchkat@lemmy.world on 22 Jul 2024 14:24 collapse

I think you mean after they sell their assets to a new company. Leave the lawsuits with the old company who will shut down.

quinkin@lemmy.world on 22 Jul 2024 02:47 next collapse

Additionally, organizations should approach CrowdStrike updates with caution

We would if we were able to control their “deployable content”.

AlecSadler@sh.itjust.works on 22 Jul 2024 03:11 next collapse

Serious question, can you not? There isn’t an option to…like…set a review system first?

EncryptKeeper@lemmy.world on 22 Jul 2024 03:26 collapse

For antivirus definitions? No, and you wouldn’t want to.

AlecSadler@sh.itjust.works on 22 Jul 2024 03:28 collapse

But it sounds like this added files / drivers or something, not just antivirus rules?

SeeJayEmm@lemmy.procrastinati.org on 22 Jul 2024 03:42 next collapse

Turns out it was a content update that caused the driver to crash but the update itself wasn’t a driver (as per their latest update.)

AlecSadler@sh.itjust.works on 22 Jul 2024 04:08 next collapse

Oh, wow.

b161@lemmy.blahaj.zone on 22 Jul 2024 05:49 next collapse

Do you know if the sensor update policy had been set to N-2 would this have avoided the issue?

quinkin@lemmy.world on 22 Jul 2024 06:54 next collapse

No it would not.

starneld@infosec.pub on 22 Jul 2024 07:05 collapse

Setting the update policy to N-2 (or any other configuration) would not have avoided the issue. The Falcon sensor itself wasn’t updated, which is what the update policy controls. As it turns out, you cannot control the content channel updates - you simply always get the updates.

b161@lemmy.blahaj.zone on 22 Jul 2024 07:56 collapse

💀 Fucking hell CrowdStrike.

wolfylow@lemmy.world on 22 Jul 2024 05:57 collapse

Found this post that explains what happened in detail: lemmy.ohaa.xyz/post/3522666

As an application developer (rather than someone who can/does code operating systems) I was just left open-mouthed …

Looks like they’re delivering “code as content” to get around the rigour of getting an updated driver authorised by MS. I realise they can’t wait too long for driver approval for antivirus releases but surely - surely - you have an ironclad QA process if you’re playing with fire like this.

mox@lemmy.sdf.org on 22 Jul 2024 03:50 collapse
corsicanguppy@lemmy.ca on 22 Jul 2024 04:00 next collapse

We would if we were able to control their “deployable content”.

Minimum safe distance.

ISOmorph@feddit.org on 22 Jul 2024 05:42 collapse

I read on another thread that an admin was emulating a testing environment by blocking CrowdStrike IPs on their firewall for the whole network before each update, with the exception of a couple machines. It’s stupid that he has to do this but hey, his network was unaffected

DasAlbatross@lemmy.world on 22 Jul 2024 04:04 next collapse

But I’ve read so many posts on here about how Linux is flawless!

ganymede@lemmy.ml on 22 Jul 2024 05:00 next collapse

not sure if you’re being sarcastic, but if anything this news paints linux deployment in an even better light.

breakingcups@lemmy.world on 22 Jul 2024 06:56 collapse

This is good for Bitcoin

FalseMyrmidon@kbin.run on 22 Jul 2024 05:06 next collapse

Are you shocked that bad software can crash multiple operating systems or something?

ChairmanMeow@programming.dev on 22 Jul 2024 05:38 next collapse

Nah, but there were some Linux evangelists claiming this couldn’t possibly happen to Linux and it only happened to Windows because Windows is bad. And it was your own fault for getting this BSOD if you’re still running Windows.

And sure, Windows bad and all, but this one wasn’t really Microsofts fault.

Petter1@lemm.ee on 22 Jul 2024 06:25 next collapse

Well, ever heard freeBSD?

vext01@lemmy.sdf.org on 22 Jul 2024 06:44 collapse

Yeah, it supports kernel modules, so is also vulnerable to bad third party kernel code.

Petter1@lemm.ee on 22 Jul 2024 07:31 collapse

🤔if nobody makes a third party kernel module, then there is still no risk

vext01@lemmy.sdf.org on 22 Jul 2024 07:45 next collapse

Also, even if they do, you can choose to not load it.

It amused me that so many people had this installed, but had no idea what it was for.

Passerby6497@lemmy.world on 22 Jul 2024 13:00 collapse

Security through apathy!

Ferris@infosec.pub on 22 Jul 2024 06:35 next collapse

if they dont know the boot sequence is a thing maybe their opinion on this doesnt really matter 🤷🏼

rottingleaf@lemmy.world on 22 Jul 2024 08:37 collapse

The sane ones of us know well that a faulty driver is a faulty driver, but! Linux culture is different. Which is why this happened so spectacularly with Windows. EDIT: and not with Linux

dan@upvote.au on 23 Jul 2024 07:53 collapse

I’ve had the proprietary Nvidia driver crash my whole system a few times. Hoping their new open-source driver (not nouveau, I mean the new out-of-tree open-source one) is better.

rottingleaf@lemmy.world on 23 Jul 2024 18:15 collapse

I had X crash due to Nvidia under FreeBSD a few times, and fewer kernel panics due to it. Never used Linux with Nvidia though.

DasAlbatross@lemmy.world on 22 Jul 2024 16:42 collapse

I’m not shocked at all, but there seems to be a very sizable number of people on Lemmy who think if people just used Linux there’d never be another problem or exploit again, which is ridiculous. Mac users used to feel the same way until the market share started to grow and all of the sudden you’re seeing news of serious exploits.

Alborlin@lemmy.world on 22 Jul 2024 19:08 next collapse

Haven’t you heard 4% market is captured by Linux , it’s the ONLY saviour os out there , windows users and macos users are idiots and all Lemmy Linux dudebros grandpa’s are using Linux without single problem. Despite the fact that each Linux had it’s own shell and there is no escape from terminal ( in 2024) if you even as try to use something more complicated. ;)

Realitaetsverlust@lemmy.zip on 23 Jul 2024 09:30 collapse

For almost every use case a normal user needs, there is a gui. You do not need the terminal.

Alborlin@lemmy.world on 23 Jul 2024 10:38 collapse

Tell me where to find executables for programs installed without using Terminal , a very very clickable task in windows

privatizetwiddle@lemmy.sdf.org on 23 Jul 2024 20:34 next collapse

/usr/bin

There, no clicking needed. 🙃

Alborlin@lemmy.world on 24 Jul 2024 09:39 collapse

Hah not true in many many many cases

kombos@discuss.tchncs.de on 25 Jul 2024 11:06 collapse

Did you ever use linux? There is a file explorer in most, if not all linux distributions.

Realitaetsverlust@lemmy.zip on 24 Jul 2024 23:00 collapse

Huh? if you install anything via a software manager which is included with most user-friendly distros like Ubuntu, popos, mint or zorin, it comes with a .desktop file which makes it discoverable by using the means of the desktop environment - usually something like the start menu. And that’s not something new. That has been the case for years now.

hal_5700X@sh.itjust.works on 23 Jul 2024 02:08 collapse
BurnSquirrel@lemmy.world on 22 Jul 2024 14:32 next collapse

Companies don’t really use Debian or Rocky in widescale production because they have no support.

Now red hat or ubuntu is a different matter.

Honestly though this does point out that this is a pattern of behavior on crowdstrikes part. This should have been the canary in the coalmine.

lud@lemm.ee on 22 Jul 2024 15:08 next collapse

We actually use rocky and I think Debian at work for servers. We are currently migrating away from EOL centos .

histic@lemmy.dbzer0.com on 22 Jul 2024 18:31 next collapse

A lot of companies use debian

TrumpetX@programming.dev on 23 Jul 2024 11:40 next collapse

We use Alma, which is basically Rocky. Before that, CentOS. Lots of people don’t need or want the expensive support contracts.

OSS support though donations and commits is the way to go unless you get value out of those contracts (we would not).

ninekeysdown@lemmy.world on 23 Jul 2024 12:48 collapse

I don’t know about that. In the HPC space we use a lot of EL distros. Mainly Centos & now Rocky. Most of the nodes run the os in ram too. Though almost all those kind of systems have no internet connection and don’t use things like crowdstrike. I’ve worked for a few places where the only part of the company that used windows was the office staff eg accounting, hr, etc. everything else is/was using an EL distro or upstream of one eg Fedora. Those type of places usually don’t mess things like crowdstrike for a lot of different reasons eg the kind of data they’re processing and security requirements on that data.

StaySquared@lemmy.world on 22 Jul 2024 14:53 next collapse

I recently learned that this is the same company that gave us the bs Russia Gate.

kevindqc@lemmy.world on 22 Jul 2024 15:24 next collapse

So who do you think hacked the DNC and got their emails, then? Is it the same people who hacked the RNC but didn’t leak the emails? What makes you more qualified than CrowdStrike on this?

StaySquared@lemmy.world on 22 Jul 2024 15:41 collapse

U.S. intelligence officials cannot make definitive conclusions about the hacking of the Democratic National Committee computer servers because they did not analyze those servers themselves. Instead, they relied on the forensics of CrowdStrike, a private contractor for the DNC that was not a neutral party, much as “Russian dossier” compiler Christopher Steele, also a DNC contractor, was not a neutral party. This puts two Democrat-hired contractors squarely behind underlying allegations in the affair – a key circumstance that Mueller ignores.

kevindqc@lemmy.world on 22 Jul 2024 16:45 next collapse

So no evidence it wasn’t Russia? No alternative? Ok.

btaf45@lemmy.world on 23 Jul 2024 04:03 collapse

theguardian.com/…/donald-trump-us-senate-report-r…

A report by the Senate intelligence committee… runs to nearly 1,000 pages and goes further than last year’s investigation into Russian election interference by special prosecutor Robert Mueller… identifies Konstantin Kilimnik as a Russian intelligence officer employed by the GRU, the military intelligence agency behind the 2018 poisoning of the Russian double agent Sergei Skripal. It cites evidence – some of it redacted – linking Kilimnik to the GRU’s hacking and dumping of Democratic party emails.

btaf45@lemmy.world on 23 Jul 2024 03:59 collapse

I recently learned that this is the same company that gave us the bs Russia Gate.

WTF you mean the US Senate?

theguardian.com/…/donald-trump-us-senate-report-r…

NutWrench@lemmy.world on 22 Jul 2024 15:23 next collapse

In April, a CrowdStrike update caused all Debian Linux servers in a civic tech lab to crash simultaneously and refuse to boot.

And then, you boot their servers from a Linux Live USB, run TimeShift to restore the last system snapshot, refuse the latest patch from Cloudstrike and they all lived happily ever after.

Evilcoleslaw@lemmy.world on 22 Jul 2024 16:06 next collapse

And it’s not much more difficult to fix on Windows, except for the scale of the problem.

avidamoeba@lemmy.ca on 22 Jul 2024 16:06 next collapse

None of these things are used in actual server operations.

RecluseRamble@lemmy.dbzer0.com on 23 Jul 2024 10:57 next collapse

Good luck doing that remotely. Which is the sole problem with this most recent CrowdStrike bug.

friend_of_satan@lemmy.world on 23 Jul 2024 12:17 collapse

Anybody who doesn’t already have ipmi serial console access set up needs to put that on their list of acceptance criteria for remediation of this incident.

friend_of_satan@lemmy.world on 23 Jul 2024 12:15 next collapse

boot their servers from a Linux live usb

If I ran a computer lab that wasn’t already net booted, I’d use this as the motivating factor to put that in place. Net booting to a repair image, or just reinstalling the whole OS either from scratch or a known good disk image, is where anybody who manages a fleet of computers should be.

There was a point in time where I had a pxe boot server vm set up on my laptop that I used to reload servers in our little row of racks at 365 main, because it let me quickly swap out the boot iso, and was faster than usb sticks were at the time.

kurap1ka@lemmy.world on 23 Jul 2024 18:42 collapse

And on Windows you booted in safe mode and removed one file. What’s the point of your post?

Vilian@lemmy.ca on 22 Jul 2024 15:28 next collapse

Because Linux sysadmins know to test a fucking update before applying to the whole company

suzune@ani.social on 23 Jul 2024 18:52 collapse

Linux admins know that you’re worsening security when installing 3rd party stuff into kernel, so most of them tend to avoid it. And that’s why no one noticed that Crowdstrike problem.

menas@lemmy.wtf on 22 Jul 2024 16:45 next collapse

So in the end, they is an internal contradiction in capitalism. It just append to be collapse due to lack of ressources and dumb management

rottingleaf@lemmy.world on 22 Jul 2024 18:27 next collapse

It just append to be collapse due to lack of ressources and dumb management

TIL reverting the direction of Siberian rivers and turning Kazakh steppe into agricultural land were capitalist projects.

This one is a contradiction of highly hierarchical and degenerate systems.

With capitalism the contradiction is old and well known - power bends rules. Bent rules cause degeneracy. Degeneracy causes degradation and collapse.

JoeBigelow@lemmy.ca on 22 Jul 2024 20:48 collapse

Got me interested enough to Google, maybe you should too

Research and planning work on the project started in the 1930s and was carried out on a large scale in the 1960s through the early 1980s. The controversial project was abandoned in 1986, primarily for environmental reasons, without much actual construction work ever done.

rottingleaf@lemmy.world on 23 Jul 2024 06:14 collapse

If you mean the rivers part, then yes.

If you mean the steppe part, then no, they’ve caused a few ecological catastrophes first before stopping.

Realitaetsverlust@lemmy.zip on 23 Jul 2024 09:28 next collapse

I feel like no matter what’s happening, some people will always blame capitalism

the_toast_is_gone@lemmy.world on 23 Jul 2024 12:09 collapse

What does an economic system have to do with bad IT decisions?

menas@lemmy.wtf on 23 Jul 2024 18:29 collapse

  • Short term interest: Yearly benefits make the corporation value. Work to enhance stability, such as investment in other open source project, documentation, formation, or code quality enhancement are less likely to qet time
  • Commercial focus: In a capitalist economy, we don’t have pure and perfect knowledge of product. Even if it’s supposed to work like this, commercials and adds are way more effective to sell products, than a top notch product
  • Antagonist interests: even if workers tend to like making good stuff, they’d rather eat and get housed. Sending a warning because the products are bad or dangerous can threat someone that made a bad decision, which is likely to be someone in charge. Keeping a low profile is (unfortunately) a reasonable behavior

I think that an economy lead by financial interest, open market, and a hierarchy in the production is a good definition of capitalism.

And yes, definitely the way that people get food, housing, and not being exclude will define a lot of thing in society.

the_toast_is_gone@lemmy.world on 23 Jul 2024 20:12 collapse

  • Short term interest: this is just human nature. All economic models work around human nature and desires. People desire short-term gains in pretty much any endeavor. If this was a communist society, they’d still rush to get this thing out as fast as possible so they could meet state quotas/meet whatever other incentive is being offered to finish the job. The problem comes not from the motivations, but how they respond to it. Rushing deadlines and ignoring the need for testing and quality code is a universal human constant.
  • Commercial focus: we have a much better idea of how much an endeavor, product, service, etc. will cost under capitalism because we have a decentralized and automatic way to calculate its value in the form of prices. Miscalculations - or simple human errors, like pushing bad code by accident - happen though, and hopefully this company has learned that prioritizing pushing something out can risk losing them money vs. testing it and coming out with a quality product.
  • Antagonist interests: this is another question of short-term vs. long-term interests. Say you have a factory. If you crank up the machines to double speed, you’re potentially doubling your production, right? It isn’t that simple, actually. You can end up with a lot more workplace accidents that way, which will destroy your productivity extremely quickly. Same deal here. This will, hopefully, be a lesson learned by the industry in not pushing garbage code. M$ can’t serve ads to people who can’t boot their PCs, and will instead lose boatloads of money suddenly having to fulfill tech support contracts because of their screw-up, for example. Crowdstrike is going to have its competitors look a lot more appealing from here on out because they’ve been exposed as fools. (If they have no competitors - IT people, this is your sign!) Mistakes will happen until the end of time, of course, but that doesn’t mean fat-fingering the keyboard is a fault of the Western economic system.

Capitalism is, in essence, the ability for people to exchange their goods freely. It isn’t dependent on corporations or some weird hierarchy of managers and workers. Those are facts of living in this system, but it isn’t a direct consequence of “capitalism.” If everyone worked only for themselves and produced something to bring to the exchange, that would still be capitalism.

ZILtoid1991@lemmy.world on 22 Jul 2024 17:06 collapse

Microsoft already has a very bad reputation, so they will be blamed for every issue on their OS.

Vista suffered from bad 3rd party drivers, then people proceeded to just dunk on M$ due to their already bad name. Despite Edge is nowadays just a different flavor of Chromium, people are still making “haha IE slow” memes, even those that still claim Google being the “savior of the internet”.