Bishma@discuss.tchncs.de
on 20 Oct 20:27
nextcollapse
So Roblox is the only of those companies that can handle proper region failover if us-east-1 shits the bed? You young ops have gotten soft! Learn to live by ChaosMonkey or die by the Gorilla.
NaibofTabr@infosec.pub
on 20 Oct 20:50
nextcollapse
Yeah, but proper failover and recovery requires additional infrastructure, and that costs money.
Hopefully a bunch of risk management people are writing I-told-you-so emails to C-suites right now.
homesweethomeMrL@lemmy.world
on 20 Oct 21:02
nextcollapse
Yeah but not to worry, C-suites have pretty good filtering rules in place to never read them. Saves time, really.
Yup, and some things which can be moved cannot be done automatically, quickly or easily…even if you are prepared. AWS is a huge suite of products and services, and there’s a lot of old legacy shit running on it. I wouldn’t punch down on the ops for this one. Cybersecurity and disaster recovery are not directly profitable, so they are almost always neglected in your average shop.
Roblox doesn’t really lock down regions unless they are China. The game client can connect to any server they own. It naturally falls back to the server with the least issues without prompts and intervention. If for an even bigger example all US servers were to go offline, the client automatically redirects people to either Europe or Asia based on ping.
mesamunefire@piefed.social
on 20 Oct 20:48
nextcollapse
Amazon sneezes, the whole internet catches a cold.
Im still seeing services up/down the entire day at work. Services that are not even AWS like Azure are slow for some reason (probably businesses failing over to other infa). Its crazy.
None of our in office infa is having issues. Managers are talking about fail-overs all day lol.
There are some downstream / knock on effects going on which can be explained…but I can’t help but wonder if today’s story is bigger than just AWS. AWS saying it was an outage of a “few hours” for DynamoDB and DNS…and that doesn’t line up all that great with what people are reporting in the wild . I’m not trying to start a conspiracy theory, just wondering what the post mortems will tell us, if anything. Obviously the suits want to keep embarrassing fuckups downplayed as much as possible.
mesamunefire@piefed.social
on 20 Oct 21:21
nextcollapse
Its crazy, we are seeing unrelated services stop sending emails, issues with DNS, all sorts of strange stuff.
Once upon a time they insisted that Amazon had independent decision making on their providers when they were needing new infrastructure and they “always decided that AWS suited them best.”
Shockingly (/s), they stopped making that claim right about the same time they started admitting that their biggest users are all under Private Pricing Agreements.
tomiant@piefed.social
on 20 Oct 21:16
nextcollapse
Not my ancestors!
thepompe@ttrpg.network
on 20 Oct 21:18
nextcollapse
The cloud was a scam for people with more money than sense.
Most people hosting things on the cloud, like Fediverse services, would be better off selfhosting and buying the hardware themselves.
I got started self-hosting last week when I got a hold of a smal Lenovo ThinkCentre. I installed Proxmox on it and if I want to self-host something I just spin up a container or virtual machine on the Proxmox system. It’s so much easier than installing self-hosted projects on bare metal. And if you want to change things around then just disable or delete the container/vm and let Proxmox stay clean. If one container breaks the rest of the system will still function.
I could easily host Lemmy from home with Proxmox and a reverse proxy with my current setup. I am not going to because I am not interested in moderating a platform and all the responsibility that comes with it, but it’s very possible to do.
Edit: the Proxmox community helper scripts makes installing most things a breeze! I use them every opportunity I get.
Edit: ohh noooo I just saw PhobosAnomaly’s post, I am so slow
peoplebeproblems@midwest.social
on 21 Oct 00:24
nextcollapse
It’s kind of telling when Amazon services like Ring, Prime Video, and Alexa failed over pretty quickly, but everything else just didnt.
There’s no conspiracy here. It’s just highlighting that Amazon could prepare for AWS outages in a region, but since everything auth related was on us-east-1, everything else went down.
Shortcuts are costly.
Bluefalcon@discuss.tchncs.de
on 21 Oct 00:37
nextcollapse
threaded - newest
So Roblox is the only of those companies that can handle proper region failover if us-east-1 shits the bed? You young ops have gotten soft! Learn to live by ChaosMonkey or die by the Gorilla.
Yeah, but proper failover and recovery requires additional infrastructure, and that costs money.
Hopefully a bunch of risk management people are writing I-told-you-so emails to C-suites right now.
Yeah but not to worry, C-suites have pretty good filtering rules in place to never read them. Saves time, really.
And plausible deniability.
Yup, and some things which can be moved cannot be done automatically, quickly or easily…even if you are prepared. AWS is a huge suite of products and services, and there’s a lot of old legacy shit running on it. I wouldn’t punch down on the ops for this one. Cybersecurity and disaster recovery are not directly profitable, so they are almost always neglected in your average shop.
Yup, AWS is legacy cloud. It was only recently that they set encryption by default on S3 buckets, before that they were just in the clear by default.
It’s never important until suddenly it’s the most important thing in the world.
Either that, or the reports stopped coming in since it's a school day.
I think they’re the only ones doing multi-cloud. If us-east-1 shits the bed, it’s a bad day for AWS in general.
Roblox doesn’t really lock down regions unless they are China. The game client can connect to any server they own. It naturally falls back to the server with the least issues without prompts and intervention. If for an even bigger example all US servers were to go offline, the client automatically redirects people to either Europe or Asia based on ping.
Amazon sneezes, the whole internet catches a cold.
Im still seeing services up/down the entire day at work. Services that are not even AWS like Azure are slow for some reason (probably businesses failing over to other infa). Its crazy.
None of our in office infa is having issues. Managers are talking about fail-overs all day lol.
There are some downstream / knock on effects going on which can be explained…but I can’t help but wonder if today’s story is bigger than just AWS. AWS saying it was an outage of a “few hours” for DynamoDB and DNS…and that doesn’t line up all that great with what people are reporting in the wild . I’m not trying to start a conspiracy theory, just wondering what the post mortems will tell us, if anything. Obviously the suits want to keep embarrassing fuckups downplayed as much as possible.
Its crazy, we are seeing unrelated services stop sending emails, issues with DNS, all sorts of strange stuff.
Same with us. Had to reboot/restart a number of things, and resynch clocks.
They want to keep the news of the rally over the weekend as quiet as possible.
and the epstein files. i heard it dint affect international that much, so its rather covenient.
When a handful of people own all the companies in the world, the whole world becomes a single point of failure.
Soon: “Welcome to Amazon, I love you.”
games were affected online, plus apps, and then anyone in retail who does inventory, order writing.
.
Why does Amazon use AWS?
I thought they had an in-house solution.
A lot of AWS are wrappers over said in house solutions
And there’s a push to move off of those to proper AWS
Figma balls wheeeyyyy
Alright…
crushes your balls with ginger root which makes them burn
of your imagination?
It seems that there is not a single original thought in that head of mine.
doesn’t matter bro, they’re your thoughts and that’s what makes them awesome 😊
Jesus Amazon is a big enough company they should be doing their own hosting not using AWS.
Amazon is using their own hosting. AWS stands for Amazon Web Services. Or did I miss the joke?
Miss
I was confused too, but I’m still glad you didn’t add the /s because that always kills the joke.
I for one don’t think you missed anything
They’re big enough to be a utility and be nationalized!
And should be
Once upon a time they insisted that Amazon had independent decision making on their providers when they were needing new infrastructure and they “always decided that AWS suited them best.”
Shockingly (/s), they stopped making that claim right about the same time they started admitting that their biggest users are all under Private Pricing Agreements.
Not my ancestors!
The cloud was a scam for people with more money than sense.
Most people hosting things on the cloud, like Fediverse services, would be better off selfhosting and buying the hardware themselves.
Self hosting requires an immense amount of specialized knowledge and time…
It really doesn’t, and certainly not more than running something on AWS.
In fact, hosting on cloud infrastructure adds another layer of complexity.
I stand by my original assertion.
You’re just wrong…
I got started self-hosting last week when I got a hold of a smal Lenovo ThinkCentre. I installed Proxmox on it and if I want to self-host something I just spin up a container or virtual machine on the Proxmox system. It’s so much easier than installing self-hosted projects on bare metal. And if you want to change things around then just disable or delete the container/vm and let Proxmox stay clean. If one container breaks the rest of the system will still function.
I could easily host Lemmy from home with Proxmox and a reverse proxy with my current setup. I am not going to because I am not interested in moderating a platform and all the responsibility that comes with it, but it’s very possible to do.
Edit: the Proxmox community helper scripts makes installing most things a breeze! I use them every opportunity I get.
community-scripts.github.io/ProxmoxVE/
This is the first script to start with on a fresh install
community-scripts.github.io/ProxmoxVE/scripts?id=…
Diversity is a strength when things go bad.
It’s almost as if centralizing everything is fucking stupid.
But the bean counters said it was the best idea!
I’m okay to blame bozos.
Bottom left
figma ballz hahaha gotem
Edit: ohh noooo I just saw PhobosAnomaly’s post, I am so slow
It’s kind of telling when Amazon services like Ring, Prime Video, and Alexa failed over pretty quickly, but everything else just didnt.
There’s no conspiracy here. It’s just highlighting that Amazon could prepare for AWS outages in a region, but since everything auth related was on us-east-1, everything else went down.
Shortcuts are costly.
Shit, none of my websites are slow. Fuck Amazon
Let’s take a decentralized network and centalize it! Nothing will surely go wrong.
Heh. Say what you will about Roblox (plenty of stuff), but seems like their infrastructure is on lock.
OOWHUUH
Maybe it was a bad idea for society to put 90% of the internet on one company’s infrastructure.
…and I did not even notice it, aside from the news here.