Some bad code just broke a billion Windows machines
(www.youtube.com)
from TimeSquirrel@kbin.melroy.org to technology@lemmy.world on 20 Jul 2024 01:42
https://kbin.melroy.org/m/technology@lemmy.world/t/364533
from TimeSquirrel@kbin.melroy.org to technology@lemmy.world on 20 Jul 2024 01:42
https://kbin.melroy.org/m/technology@lemmy.world/t/364533
Cybersecurity firm Crowdstrike pushed an update that caused millions of Windows computers to enter recovery mode, triggering the blue screen of death. Learn …
threaded - newest
<img alt="" src="https://ani.social/pictrs/image/511341e2-44ed-422e-a6d7-d1f6fa20e552.webp">
Whoda thunk automatic updates to critical infrastructure was a good idea? Just hope healthcare life support was not affected.
Many compliance frameworks require security utilities to receive automatic updates. It’s pretty essential for effective endpoint protection considering how fast new threats spread.
The problem is not the automated update, it’s why it wasn’t caught in testing and how the update managed to break the entire OS.
It is pretty easy to imagine separate streams of updates that affect each other negatively.
CrowdStrike does its own 0-day updates, Microsoft does its own 0-day updates. There is probably limited if any testing at that critical intersection.
If Microsoft 100% controlled the release stream, otoh, there’d be a much better chance to have caught it. The responsibility would probably lie with MS in such a case.
(edit: not saying that this is what happened, hence the conditionals)
I don’t think that is what happened here in this situation though, I think the issue was caused exclusively by a Crowdstrike update but I haven’t read anything official that really breaks this down.
Some comments yesterday were claiming the offending file was several kb of just 0s. All signs are pointing to a massive fuckup from an individual company.
Which makes me wonder, did the company even test it at all on their own machines first?
Nah EDR is pointless like all of cybersecurity. All these compliance frameworks are just a further grift to get a slice of B2B procurement budgets. The practice of cybersecurity has caused a more severe widespread outage than any malware ever could.
lol, ok
OP is not entirely wrong. At least in Linux land you can now implement EDR like functionality entirely with EBPF without installing a fucking rootkit. So traditional EDR products are a grift if you are on the bleeding edge.
Ok Russian comrade. Security in companies is terrible. You’re right. It’s just a giant grift.
Now, go buy some limited time offer fight fight fight shoes from agent orange.
Genuinely, what? What is “fight fight fight shoes” and “agent orange” like the chemical? What does me being Russian have to do with it? Is this some kind of twitter lingo I’ve touched grass too much to understand?
EDIT: Figured out it’s probably a trump reference. Idk I’m not a trump fan so idunno.
Hospital stuff was affected. Most engineers are smart enough to not connect critical equipment to the Internet, though.
I’m not in the US, but my other medical peers who are mentioned that EPIC (the software most hospitals use to manage patient records) was not affected, but Dragon (the software by Nuance that we doctors use for dictation so we don’t have to type notes) was down. Someone I know complained that they had to “type notes like a medieval peasant.” But I’m glad that the critical infrastructure was up and running. At my former hospital, we used to always maintain physical records simultaneously for all our current inpatients that only the medical team responsible for those specific patients had access to just to be on the safe side.
That’s actually a very smart idea, keeping physical records of every inpatient. Wonder why the ai companies don’t do transcription of medical notes, instead of trying to add ai features to my washer/ dryer combo. Just seems like a very practical use of the tech
They do, one of the things my hospital is working on implementing, much to my chagrin, is an AI thing where the doctor leaves their phone out during the visit. It listens to the patient and the doctor and generates a note. I think it’s a Nuance product, I’m not directly involved with the implementation.
For me, as soon as I see a doctor have his phone out I’m telling him to put that shit away and I don’t consent to some app listening to what I’ve got to say.
I doubt you can decline, any more than you can tell them not to use a notebook.
It is subject to actually serious HIPAA regulations though.
I’m aware of the regulations, I’m an Epic analyst and been in the world of healthcare since 2009 (used to be an MLS). Watch me decline right out the door and immediately contact patient relations. Put your fucking phone away. Healthcare has managed for quite a while without them.
Saying a personal cell phone is equivalent to a written notebook is wild. Can you compromise a notebook with malware?
The doctor will very likely have no choice but to send you to complain to someone else. Not recording will violate policy if that’s what their system is, and it’s an entirely reasonable policy to have. Medical interactions are heavily documented for a reason, and it’s because mistakes can literally kill people.
All your records are already electronic, and a notebook is a far bigger security risk than a cell phone is. It’s not encrypted. Anyone can walk away with it.
I disagree. I don’t consent to a doctor having his personal device out recording. Doesn’t seem like either of us will be changing our minds, so I’ll bid you adieu.
I meant to transcribe handwritten patient notes to dogital after the patient is released. That way op can retain the physical records for inpatients to mitigate another crowdstrike situation, and still have searchable records long term. Recording a patient consultation sounds like a bad idea all round. They’d have to figure out how to read doctors handwriting though,bbut I gather most of the difficulty is due to the cryptic names they have for things.
Huh. I thought medieval peasants were usually illiterate? Even less computer literate?
This is pretty much correct. I work in an Epic shop and we had about 150 servers to remediate and some number of workstations (I’m not sure how many). While Epic make not have been impacted, it is a highly integrated system and when things are failing around it then it can have an impact on care delivery. For example if a provider places a stat lab order in Epic, that lab order gets transmitted to an integration middleware which then routes it to the lab system. If the integration middleware or the lab system are down, then the provider has no idea the stat order went into a black hole.
Our lab was absolutely fucked from multiple integrations going down. I’m a Cupid analyst and we weren’t really affected. What app do you work on?
I’m an integration guy at my roots but I lead a variety of different teams at the moment. We use Corepoint as one of our interface engines and it shat the bed big time. We had to restore it from backup, which was nuts in my opinion. We had a variety of apps impacted.
That’s cool. I was going to move over to our integration team but I’m looking into Epic consulting instead. Our integration team was very busy on Friday along with our clinical apps team. We use Cloverleaf for our interface engine, I’ve got a bit of experience poking around in there. HL7 is interesting, but I’d like to learn FHIR. Do you have a Bridges cert?
I’m Bridges certified as well as in Cloverleaf, which we also use. FHIR is great but it doesn’t require much in the way of integration engineers.
I’m an Epic analyst - while Epic was fine, many of our third party integrations shit the bed. Cardiology (where I work) was mostly unaffected aside from Omnicell being down, but the laboratory was massively fucked due to all the integrations they have. Multiple teams were quite busy, I just got to talk to them about it eventually.
I work healthcare adjacent and some providers were affected as expected. Hoping as well that those critical systems were not, but that chance is non zero.
Are there really a billion systems in the world that run Crowdstrike? That seems implausible. Is it just hyperbole?
Yes
Probably includes a bunch of virtual machines.
Yeah, our VMs completely died at work. Has to set up temporary stuff on hardware we had laying around today. Was kinda fun, but stressful haha.
Could you just revert VMs to a snapshot before the update? Or do you not take periodic snapshots? You could probably also mount the VM’s drive on the host and delete the relevant file that way.
This is assuming you have those access. Some companies can sometimes be a bit … Stupid.
Yeah, like the other person said, corporate IT is responsible for that stuff. I guess they’re working through the weekend to try to get it fixed.
Yes you can just go into safe mode on an affected machine and delete the offending file. The problem is it took a couple hours before that resolution was found, and it has to be done by hand on every VM. I can’t just run an Ansible playbook against hundreds of non-booted VMs. Then you have to consider in the case of servers, there might be a specific start up order, certain things might have to be started before other things and further fixing might be required given that every VM hard crashed. At the minimum it took many companies 6-12 hours to get back up and running and on many more it could take days.
Makes sense - thanks for the details.
.
I doubt it’s too much of a stretch, since even here in australia, we’ve had multiple airlines, news stations, banks, supermarkets and many others, including the aluminium extrusion business my father works at, all go down, scale this do hundreds of countries with populations tenfold of ours, it puts it into perspective that there may even be more than a billion machines affected
.
Sounds pretty plausible to me. An organization doesn’t have to be very big to get into the hundreds or thousands of devices on a network when you account for servers and VM.
A company with 40 employees all accessing and RDS server using a company laptop is looking at 85+ devices already
Despite how it may seem on Lemmy, most people have not yet actually switched to Linux. This stat is legit.
I know that Windows is everywhere, I just don’t know the percentage of Windows computers that run Crowdstrike.
Keep in mind, it’s not just clients, but servers too. A friend of mine works for a decently sized company that has about 1600 (virtual) servers internationally. And yes, all of them were affected.
You do realize that linux is something like 80% of servers. Which also well out number personal machines. If you include android linux is easily the most used os on the planet.
It’s 80% of web servers but not 80% of ALL servers.
if the question is which os has the most distinct running copies then yeah its linux.
And for the 451855528th time: switch to Linux already. Why do people keep paying for this shit? Every time I get excuses. I switched to a Linux desktop 20 years ago. There were enout moments that I needed to tweak things to make it work but for the last decade, I haven’t had any issues.
If you’re dum enough to use windows for servers then you just deserve to burn, if you make that decision then its all on you.
This has nothing to do with Windows or Linux. Crowdstrike has in fact broken Linux installs in a fairly similar way before.
Don’t worry, if it had broken in Linux, these same posters would be railing on CrowdStrike directly, but since it broke on Windows, obviously Microsoft is to blame.
You make Linux users look bad
They wouldn’t if they were consistent and had also left degenerate social media (which Lemmy is part of, despite being much better than corporate alternatives). But then they also wouldn’t because we wouldn’t read it here.
Very interesting. What are some other things that are degenerate?
Production of computer hardware being centralized, the accepted amount of complexity and obscurity in that and customer software.
A desktop system should involve a lot of standardized coprocessors at least. Like in Amiga architecture.
It’s a bit sad that with RISC-V the seemingly accepted direction of development for desktops is replacing Intel\AMD with the same paradigm.
EDIT: I mean, a person asking this and apparently thinking that the word can only be used in fascist context, can be called degenerate in their education too =)
Woah buddy slow down there you’re gonna cut yourself on that edit. I fully agree, I’m just not ignorant of how the connotations of words evolves over time, otherwise I’d say your language development seems retarded to me :)
You didnt say he was wrong though.
Sure, but damaging the sentiment of the position that he is arguing for makes him stupider than simply being wrong.
“stupider than simply wrong”
What are you? 5?
My sentiment is that it’s a crazy situation where people are defending a multi billion dollar company that we all coninously pay, who spies and serves ads despite said payments, that time after time willfully neglects security, anything in the name of profits, over a free system that works better, more reliably, is open, and dependable.
Your response: you’re stupid
Yeah, if the only thing you can do to support your sentiment is to make it look unappealing to the majority of normal users, you would be pretty stupid. Or maybe you’re actually 5 and that was just a projection on your part, I wouldn’t know.
Yes, it’s really unappealing to say that windows is the shit that it is, makes Linux look so bad
It does when you say it, which is why it’s stupid. Saying something so simple, yet you can’t even do it without giving Linux a bad look. Talk about incompetent.
Also:
Crowd strike should be held responsible, and with that I don’t mean the developmers who were forced to do this shit, I mean the ceo, the CTO.
Jail them.
If you are so critical you better not fuck around and I can guarantee you, they were fucking around, pushing bad practices, etc. why do I say that? Because its lways like that
That comp ay should be dissolved, the C suite jailed.
Also, STOP USING WINDOWS FOR DESKTOP FOR FRACK SAKE. Switch to Linux already, I’m getting tired of having to read this shit.
If you’re using windows for servers then you deserve your place right next to those C suite guys and gals
Sure, throw people in jail who haven’t committed a crime, that’ll fix all kinds of systemic issues
As if management even knows what the engineers are doing. lol
If you make decisions (typically focussing on profit over anything else) that causes so much disruption, time, and money (not to mention the possibility of risking lives), then yeah, that is a crime.
As always, if I do something like that, I get jailed. If a CTO causes it,.it’s cost of business,.let’s hand slap the company, and act as if nothing happens. Fuck them, you get paid for this. You fucked it up, you get to he actually be held responsible.
How about holding an investigation first? You know, just to see where the wrongdoing happened and who actually perpetrated it. (It just might have been a bitter developer or something.)
Also, if people want to use windows, it’s their choice and their consequences. Government and corporate services might do well to consider Linux, but most people don’t even know what a command line is.
While this is true, Linux is not the only operating system which is not Windows.
Haiku doesn’t require you to do much with CLI.
OpenBSD is Unix with the accompanying culture, but it’s just more coherent to the degree that both CLI’s are very simple for administration (the way I use my non-work machine, I sometimes think that maybe I should switch ; lacking Wine and games would be an advantage, not a disadvantage) and GUI’s to do it have fewer problems than in Linux. NetBSD - a bit more messy, but same as compared to Linux, FreeBSD - even more, but same as compared to Linux. I’m talking about the base system, because X, desktop environments and such are the same.
This doesn’t solve the problem of Windows device drivers’ support, which is realistically the main thing you’d need for an OS to be popular. Applications are important, but I think if Altera would have a big buyer willing to run Altium on Linux workstations, they’d find in themselves the effort needed make it work in Wine.
But then there was time when ndiswrapper and ndisgen were a thing for Linux and FreeBSD users. Things may have gotten much more complex, but it’s a matter of demand.
If your (large scale) security system is designed properly a bitter developer can’t break it. It would take deliberate collusion from multiple people to do so.
.
Still with this? Jeez… that’s like saying people probably won’t adopt tv today because it’s still black and white.
Linux cli is great and many of us use it because of that, but it’s been at least 15 years that a regular user would not ever NEED to use it to do anything in linux
I'm gonna be honest, I had to use it last night, but that was because I was installing a non-Steam Windows game (Simcity 2013) and needed to use winetricks on the terminal to configure something before it would launch. If you are doing anything outside the predefined and preconfigured stuff, it can still get pretty hairy sometimes.
Would have been easily avoided by using Lutris for your non steam games
Just last night I needed to check what my network card model was and had to open the command line to do so on Ubuntu. Couldn’t seem to find an equivalent to Windows’ Device Manager.
Because they’ve done that countless times before and its always the same. A few motbsh ago there was a senate hearing on ehy Microsoft knowingly caused the Chinese government hacking the US government by deciding not to fix critical security bugs to avoid losing contracts and thus,.money.
What is the result, every damn time?
Weeeeewwweee sowweeeyyyy, but the CEO is on it this time! THIS time we won’t fuck you over! That was what, a month ago?
Meanwhile I say, fuck Microsoft, stop paying for that corrupt badly built spyware shit, switch to Linux, and then I’m the bad guy.
Edit: judging from the downvotrs here, it’s fair to say that a lotmof people are perfectly fine with paying to get screwed over
Woah chill dude. I never said you were a bad guy. In fact, I get the frustration. I even agree with it to some extent. I just don’t think a lot of people being temporarily inconvenienced is a justifiable excuse to go throwing people into jail willy nilly. Let professionals figure out what happened, let the judges decide if it was actually a criminal offense or just a fuck up of epic proportions, then let the consequences roll out accordingly.
If people are dying (I’m not sure), then I think we need to be acting in greater earnest.
You make Linux users look bad. Please cease.
To be fair this seems to be the sentiment on most Linux and linux-ancillary forums.
Which while wrong and ignorant on multiple levels, seems on brand none the less.
What is wrong with saying that a paid product that is far inferior to a free (both as in beer and freedom) product should be dropped in favor of said free product when said paid product has embarrassing failures, scandals and abusive behavior on pretty much a weekly basis?
How long ago was it that Microsoft testified in front of Congress after deciding they weren’t going to fix a security hole because the negative press would cost them contracts (and thus money) which resulted in China successfully hacking the US government? A month or so? What did the Microsoft dipshit say again? Oh yeah, THIS time (after all the previous times) the CEO would be on top of security, weally! And more importantly, weeewe sowwyyyy, we won’t do it again.
Yes yes yes, this is a different provider, who cares? Its a core product that is only there because their system is so badly Designed in the first place.
These are.paid products and millions if not billions of machines went down and I’m the bad guy for saying that this is not acceptable and that people should drop that paid spyware and anware shit for Linux, which is free and a hell of a lot more reliable than this windows crap? YOU ARE PAYING MONEY FOR THIS, YOU STILL GET ABUSED AND THEN ITS STILL NOT RELIABLE! WHY DO YOU A ACCEPT THIS BULLSHIT? what is wrong with you?
Edit: At this point it’s hard not to see these reactions as a beaten and abused wife saying “oh but my husband really isn’t such a bad guy once you get to know him!”. I don’t get these attitudes, why do you accept this treatment from a multi billion dollar company?
Why?
Seriously, why?
Linux is a free system with a fraction of the daily / weekly issues that Microsoft has. Its Been like this for literally decades now.
Microsoft Sella expensive paid systems that spy on us and still feed us advertising. This week, Microsoft and vendors caused likely billions of dollars in damage. Will there be any consequences? Nah. Even better; If I say stip using Microsoft software, I’m a bad guy!
Just a few months ago, it came out that Microsoft consciously decided not to fix critical security hugs resulting in hacks in the US government bu the Chinese government. There was a senate hearing in this where they weally weally promised this time they would behave. I said the same back then, install Linux already and got the same responses, I’m making Linux look bad!
So I ask you… HOW? How exactly am I the bad guy here, why isn’t everyone shitting on Microsoft and it’s providers for fucking this up so so wonderfully bad, AGAIN…?
And mind you, we all pay for this shite. Why hasn’t anyone there hone to jail for causing shit like this by making decisions that obviously knowingly would cause this shit?
But yeah, you’re riiiiight. I make Linux look bad, and we wouldn’t want that, now would we?
Are you 14?
Are you? Since you’re only able to feign an insult, I’m guessing you’re not the brightest
I wasn’t only able to feign an insult, I already responded to you that you’re making Linux look bad. You’re definitely a teenager with a small attention span and unable to recognize user names. I don’t actually hold it against you, I’m glad you’re interested in FOSS!
Your advocacy is simply not effective though, insulting people for their choices may make you feel superior but it sure doesn’t make them feel good. Instead - recognize that other people are equally intelligent, and they make informed choices. As an advocate for anything seeking to sway a neutral public your purpose is to inform them of better choices, and if you do so positively and with charisma, you will associate the subject of your advocacy with said positivity in the minds of the public.
So instead of all those rants, you could’ve simply wrote a quick and concise:
" Linux doesn’t have these problems :) (it has other problems) "
and been updooted to the top.
Yeah, been there, done that. I’ve been on a Linux desktop for over 25 years now and I’m done with the corporate Microsoft bullshit that I’ve dealt with for over 30 years of my life. If you’re new here, cool. I’m not, my patience is gone. If all Microsoft offices burn to the ground tomorrow I’ll be celebrating. I’ve had so much shit, so much time lost, so many frustrations, I’m done
"Also, STOP USING WINDOWS FOR DESKTOP FOR FRACK SAKE. Switch to Linux already, I'm getting tired of having to read this shit."
1000% agreed
Lol good fuckin luck.
In a corporate environment you just aren’t getting what you need out of Linux that you don’t of windows for many of the kinds of endpoints affected.
Yeah, youndont know what you’re talking about. Literally.
Yeah, good luck telling my IT department to deal with Wine for every piece of software we require that doesn’t have a Linux version or even some kind of replacement.
Keep dreaming. Maybe next year Linux on desktops will increase by another 0.2% and you can post hundreds of articles about how the “age of Linux” is coming… again.
Crowdstrike took Debian and Rocky down earlier this year due to a bad update… Linux is not immune.
And?
Debian is a FREE (as in beer) AND a free (not as in beer but as in freedom) system maintained mainly by volunteers which has an actual focus on us, the end users.
Microsoft, on the other hand, makes us pay through the nose for shit systems that all have focus on Microsoft, NOT on the end user. If you make me pay and spy on me and serve me ads, then at the very fracking least I expect you to take responsibility when you fuck up, and paye for my lost time and money. However, as windows fracks up just about every week, Microsoft would be bankrupt within a month if they’d have to do that.
Y’know, I’m pretty deep in the FLOSS brainrot, but as someone who: A. Daily drives Fedora and Debian B. Works for an MSP and deals with Windows daily
Most companies cannot afford the productivity, monetary, or labour hour investment that is involved with changing to a whole new OS and re-training all of the workforce. Thats even if you ignore that switching to Linux generally also involves changing some percentage of programs that are used for business critical processes.
I love Linux, but it’s not meant for every situation
And a lot of times it actually isn’t that hard. I’m currently the CTO of a medium medical company and we will transition to Linux over the next 2 years. All the work will be browser based, you don’t need windows for that. Hell, you don’t need windows for anything, except a few inhouse developed apps, which you can restructure to Linux yourself.
A lot of times it’s more lack of will than lack of ability, even though the wins are right there for the taking
It’s because Windows is crap software. Just stop using anything Microsoft makes.
This was very much not caused by windows
I don’t hear about billions of Linux or Mac computers going down all at the same time. I’m hearing that windows allows a simple text file change to bring down all of them at the same time.
Weird, cus it literally happened a few months ago:
neowin.net/…/crowdstrike-broke-debian-and-rocky-l…
This was in no way a Windows problem, this was entirely a CrowdStrike problem. Claiming otherwise makes you look like an uninformed moron.
They are most likely uninformed with a very strong opinion based on how they feel…while I’d like to call them a moron I’ve done this exact thing and like to think I’m not a moron.
Calling a kernel mode driver a “simple text file” sure is interesting
Even if you write assembly code straight out like a total hacker, it’s still a text file. Literally jump 0x12345 is text. And if it’s just a few kilobits long, then it’s a simple text file yes. Got anything else to ad? Specially if the file actually doesn’t work and the system made to run it “windows” is such shit that every copy of it got halted.
Yes and at the end of the day it’s all just binary getting dumped into a cache and processed by the CPU. The point is that the intent of the file matters and while they do both hold text, the intent, purpose, and handling of the kernel mode/ring 0 driver is much different than a “simple text file”
So different in fact, that as another user pointed out, it has happened to Linux too
In a way, it was. If Windows was not as crappy as it is, external solutions would not be needed.
Linux machines also require Crowdstrike because of business requirements. That does mean Linux is just as crap as Windows then?
Do they really require it, or is this just the usual security theatre?
Probably the latter. Though, I’m not familiar enough with cybersecurity to make a comment on that.
Not to jump at you in another comment thread, but any OS that is deployed in a business environment should have some form of endpoint protection installed unless it is fully airgapped + isolated.
Despite the myth that “Linux doesn’t get malware”, it absolutely does and should have protection installed. Even if the OS itself was immune to infection, any possible update can introduce a vulnerability to that.
Additionally, again, even if the OS (or kernel in the case of linux) couldn’t be infected or attacked, the packages or services installed can be attacked, infected, or otherwise messed with and should be protected.
Is your point “Linux and Mac dont get viruses or targeted for cyberattacks”?
Or is it “This wouldn’t have broken on a different operating system”?
No to both. Windows is so broken, it needs kernel-level external software to protect it from attacks that should not be possible in the first place. It is a joke of history that this software was even worse than windows itself.
I see you’re operating on a plane of reality where windows is the only bad software, so it’s kinda pointless for me to continue here. I hope you have a wonderful day.
Windows is definitely not the only bad software, but for the amount of resources they could spend on quality, it is a rather shitty product. They could do better, but they don’t want to.
This happened because a file that CrowdStrike pushed out, which by their own processes is not one that is signed, was immediately pushed out with one of their updates. This update was pushed directly through CrowdStrike’s own method, not via Windows Update. CrowdStrike maintains this capability in order to quickly respond to and prevent security threats. The fact that they have .sys files that aren’t signed is crazy on its own, and a huge screwup by CrowdStrike. So many companies relied upon and trusted this company because up until now, everybody considered it a great product, so it was extremely popular and prevalent. It’s been a huge wake up call for everybody in I.T.
I’m not sure if you intended to reply to me, but I am aware of this. Thanks for checking my understanding though :)
lol you are correct. I meant to reply to the other guy. Low on sleep like many of us here
Combing over it's Wikipedia article, this company already had a series of other issues.
Sucks to anyone who ever relied on them. Oh look at that, they've been acquiring other security startups and companies. Perhaps that should also be looked into as well?
There is no learning, companies just move to different antivirus. The new hotness, the cycle repeats over and over until the new antivirus does this same shit. Look at McAfee in 2010, in fact the CEO of Crowdstrike was the CTO of McAfee then. That easily took down millions of windows XP machines.
The hero of Linux adoption then. All hail - what’s the name of that guy?
This isn’t the Windows L you think it is. This can and has happened on Linux. It’s a Crowdstrike/Bad corp IT issue.
I know, but the whole culture of using such things is Windows-centered.
There is learning here.
As companies, we put faith in an external entity with goals not identical to our own: a lot of faith, and a lot of control.
That company had the power to destroy our businesses, cripple travel and medicine and our courts, and delay daily work that could include some timely and critical tasks.
This is not crowdstrike’s fault; for the bad code yes, but for the indirect effects of that no. We knew - please tell me we had the brains god gave a gnat and we knew - that putting so much control in the hands of outsiders not concerned or aware of our detailed needs and priorities, was a negligent and foolish thing to do.
The lesson is to do our jobs: we need to ensure we have the ability to make the decisions to which were entrusted, and the power that authority gives us that our decisions when accepted are not threatened by a negligent mistake so boneheaded it’s all but the whim of a simpleton. We cannot choose to manage our part of our organization effectively, no matter how (un)important that organization or part is, and then share control with a force that we’ve seen can run roughshod over it.
It’s exactly like the leopards eating our face, except people didn’t see they were leopards. No one blames the leopards, as they’re just conforming to their nature, eventually.
And no one should blame this company for a small mistake, just because we let the jaws get so close to our faces that we became complacent.
Have you never worked in corporate IT or something? Of course we should blame Crowdstrike, that way we don’t get a sev 1 on our scorecard.
https://youtu.be/v0mwT3DkG4w?t=438
It’s funny that corporate IT will be one of the groups getting the blame in this case, despite it being in most cases not their decision that a company lacks a separate test and production environment. The executives that decided that usually gets off scot free.
Hahah, no doubt, while popping in and out of the outage call repeating the phrases “can I get an update?”, " Is there an ETA on recovery?" and “We need to get this back online”
Unless you have the ability and capacity to develop your own ISA/CPU architecture, firmware, OS, and every tool you use from the ground up, you will always be, at some point, “relying on others stuff” which can break on you at a moments notice.
That could be Intel, or Microsoft, or OpenSSH, or CrowdStrike^0. Very, very, very few organizations can exist in the modern computing world without relying on others code/hardware (with the main two that could that come to mind outside smaller embedded systems being IBM and Apple).
I do wish that consumers had held Microsoft more to account over the last few decades to properly use the Intel Protection Rings (if the CrowdStrike driver were able to run in Ring 1, then it’s possible the OS could have isolated it and prevented a BSOD, but instead it runs in Ring 0 with the kernel and has access to damage anything and everything) — but that horse appears to be long out of the gate (enough so that X86S proposes only having Ring 0 and Ring 3 for future processors).
But back to my basic thesis: saying “it’s your fault for relying on other peoples code” is unhelpful and overly reductive, as in the modern day it’s virtually impossible to do so. Even fully auditing your stacks is prohibitive. There is a good argument to be made about not living in a compute monoculture^1; and lots of good arguments against ever using Windows^2 (especially in the cloud) — but those aren’t the arguments you’re making. Saying “this is your fault for relying on other peoples stuff” is unhelpful — and I somehow doubt you designed your own ISA, CPU architecture, firmware, OS, network stack, and application code to post your comment.
——- ^0 — Indeed, all four of these organizations/projects have let us down like this; Intel with Spectre/Meltdown, Microsoft with the 28 day 32-bit Windows reboot bug, and OpenSSH just announced regreSSHion.
^1 — My organization was hit by the Falcon Sensor outage — our app tier layers running on Linux and developer machines running on macOS were unaffected, but our DBMS is still a legacy MS SQL box, so the outage hammered our stack pretty badly. We’ve fortunately been well funded to remove our dependency on MS SQL (and Windows in general), but that’s a multi-year effort that won’t pay off for some time yet.
^2 — my Windows hate is well documented elsewhere.