from Kris@feddit.org to fediverse@lemmy.world on 04 Jun 16:03
https://feddit.org/post/13613230
Edit: unexpectatly we are back online. Will share how we found a solution later. There might be still some federation issues though.
Alt account of poVoq@slrpnk.net here.
Our instance is currently down and I can’t get remote access to the servers. It appears that there might have been a hardware failure of the main firewall, which is the one thing I can’t work around remotely.
I am still trying a few things, but I am not very optimistic that I can get access.
The really unfortunate part is that just now I am on one of my rare work deployments abroad, so I also can’t access it physically during the next few weeks and my usual back up that could restart it is not available either.
As something like that never happened in 3 years operating the servers, I thought I can risk it, but murphy’s law seems inescapable 😓
I will try to keep you posted here on any updates, but probably there will not be much I can do for a while. Really bad timing 😥
Edit: we might use this “opportunity” to migrate the instance to Piefed, which has been an idea for quite some time now. I will keep you posted on that.
threaded - newest
Very very bad timing. I just moved to that server and now it is just gone. I have moved to another server. I guess I am glad that it happened now so I know how poorly Slrpnk.net is managed.
Yeah, I know. Complete Murphy’s law situation.
If you want a professionally managed instance, you are welcome to pay for a hosted one.
Which is exactly what I did 2 years ago. Turns out the uptime is about the same and the price is higher than 0. But on the upside, if there’s downtime I can (usually) immediately reboot the server.
The server got a little more accounts on there than I initially expected though. But I’m fine with that.
<img alt="" src="https://feddit.org/pictrs/image/b7856806-775c-4b2d-a382-18bdf89b9e32.jpeg">
One issue in three years is poor management now? Wat
Anything less than perfection is unacceptable! 😤 /s
I signed up with them yesterday and then they go down for I don’t know how long. This is my only experience with them.
Ok, and?
That doesn’t change the fact this is a one off issue and theyve never had that issue in three years.
This is a very big issue for a server no mater when it happens. A sever you can’t reset and you go on a trip ridiculous.
You are acting like this is a life or death situation. I think you may need to take a break from the internet.
It usually can be reset, it’s entirely unexpected that a component that enables that functionality would randomly break. You’re essentially implying that any admin who self hosts cannot ever leave their house for any length of time without being irresponsible because they cannot ensure with 100% certainly that a computer component won’t fail while their gone, or a lightning storm won’t fry something, or a flood won’t harm the servers.
Like, dude, c’mon. We’re volunteers, and poVoq is running this stuff for free. We’re not an enterprise business with multiple staff on hand to swap out failed components 24/7 so that our ‘customers’ don’t lose millions of dollars.
If anything, this only proves the strength of decentralized media, as we’re able to swap to a functioning node and continue on our merry way while the other is being repaired. Like what an incredible and resilient system considering none of this is making anyone any money.
You’re going to want to try “my little Internet” by Fisher-Price before using this adult Internet we’re on.
Out of 3 years of reliable service, this is the one time the server has been down for any length of time, and only due to the admin not being able to be physically present for a wildly unlikely and unexpected problem.
I signed up with them yesterday and then they go down for I don’t know how long. This is my only experience with them.
I know, I’m the one who approved your application.
You happened to sign up right before we’ve had our first particularly bad issue in 3 years. Unlucky, yes, but it’s a bit out of order (IMO) to say we’re a poorly managed instance based on that, especially seeing as you lost nothing besides the time it took to write your application. :\
My loss of time is small true. I will just move on. But keep in mind new users may not be as knowledgeable as myself.
Knowledgeable? Lmao
Does Lemmy let you retroactively deny applications?
As a matter of fact it does, actually 😅
Thanks for letting us know
Thank you for letting us know, good luck!
Ouch. Best of luck getting it back up and running ASAP.
people hate the big cloud solutions, but this is the kind of thing their HA infrastructure prevents against... hardware failures
i dont enjoy using (or paying for) aws, but i will never have a firewall or disk failure.
Their HA infrastructure is all built on open source projects. The thing they have that we don’t is teams of SREs on-call 24/7.
Work trip and server problems. Where have I seen this before?
Yeah, but at least I have not disappeared yet 😅
I see you understand the allusion.
I did not, can you give me a hint ?
The feddit.de admin disappeared on a work trip to Japan. At least that is the official story.
Evil tongues claim he is in North Korea.
I’ve had the opposite–personal trip and work server problems.
Solarpunks can have a little downtime, as a treat
Seriously, I think a big part of solarpunk ethos is combating the notion that everything has to always be available 24/7. Society pays a lot to deliver every convenience like fruit out of season from the other side of the world.
And on the punk side of things, anybody familiar with volunteer community action, cooperative events, whatever, knows that sometimes a person doing shit for the community fails to do the shit they were supposed to do because some other shit happened in their life. It happens. We just move on to the backup shit-doer-person or find a workaround for the lack of shit.
And this is the cool thing about a distributed network like Lemmy. If a giant centralized database like FB goes down, its users can’t do anything. If our favorite Lemmy node goes down, we can log into other nodes until our home base is back up.
I will no longer read that domain as “slurpink” thank you
Oh you will. I know the name for a few months now and I still read it slurpnik
Sounds like your ‘bus factor’ is too high and you need a few more backup admins.
We have backup admins, but this seems to be an unexpected hardware issue, and physical access to the servers is more restricted.
It’s a hobby forum, not a hospital dispatch line.
It’s a service used by the public.
If it were just a single-member instance that hosted no communities it’d be one thing, but slrpnk has some decently-important stuff on it. That creates some amount of responsibility on the admin’s part even if he isn’t getting paid for it.
Are you familiar with the concept of volunteer burnout, and are you aware that your current behavior is exactly what causes it? Right here in the fediverse we’re losing lem.ee to burnout, for example.
What "my behavior?* Which “behavior?” Gently recommending that poVoq/Kris get some more help? 'Cause that’s the only thing I did! My other comment was directed at your stupid hot take, not him.
I don’t know what you think you’re bitching about, but whatever it is you misinterpreted it out of thin air.
Nobody is remotely at risk of losing life or limb because slrpnk is down for a little while.
Instance providers and admins don’t owe you anything.
All I’m saying is, if you don’t want to have a responsibility to the community, don’t set yourself up for people to rely on you. That’s your choice to take on that commitment – but once you’ve made that choice that’s what it is, a commitment.
Just to be clear, I’m not mad at poVoq/Kris, who has a very reasonable attitude about the situation. My initial comment giving feedback to him was very mild.
But no, this attitude of people like @ayyy@sh.itjust.works and you (@tfm@europe.pub), that instance admins can just let a bunch of people sign up and a bunch of communities to be created and then just shut it off without warning if they want, and that’s perfectly A-OK and 100% ethical to do, is bullshit and absolutely deserves to be pushed back on!
You need a break from the internet dude. Your behavior and outlook are troubling.
You came after me first. I’m just defending myself from your abusive attitude.
“If you don’t want responsibility, don’t take on responsibility” is not an unreasonable position. Cut your condescending crap.
So basically you think that I’m obligated to spend my money and time for the rest of my life to provide you a free service once I’ve started one. People like you are the reason why instances like lemm.ee shut down. Expecting the best service for nothing and be the loudest when something you don’t like happens.
Where the fuck are you getting that idea? I’m just pushing back on the idea that “they owe you nothing” – i.e. that they’re perfectly entitled to shut it off with no notice or warning.
Lemm.ee is an example of doing the right thing by giving advance notice and honoring their responsibility.
Quit being a strawmanning asshole!
Clearly someone who has never seen the dedication of the flight sim community. Those guys would put all of us, hobby service providers, to shame.
.
It happens, it’ll be fine. Just remember not to stress out too hard about it and it’ll be fine. The solar punk community knows how to survive a server outage in style.
It looks like there is no enough solar power for solarpunk!
~no need to show me where doors are, I will find them by myself~
anyway, good luck with that!
Thanks for the heads up, try not to stress too much.
Alt of ProdigalFrog@slrpnk.net chiming in, thanks for giving us the sitrep, poVoq!
This is an unusually unlucky set of circumstances, but for the Solarpunks who find their way here, let’s try to make the best of it. If you don’t already have an alternative account somewhere, you may want to create one to use until we’re back up and running. lemmy.dbzer0.com, sopuli.xyz, or lemmy.cafe are some solid choices to consider. After that, you may want to favorite poVoq’s post here so you can check back on any news.
Even without our home base, we can still share the solarpunky stuff we find and create around the threadiverse (though as Tofu mentioned, maybe this could be a time to take a breather to be refreshed when we come back 😄).
Besides, if the past is any indication, time will fly by and we’ll be back before you know it!
Well, on the plus side lots of lessons learned and I think I might move at least the xmpp server to an external vps to have a backup communication channel.
Not sure about your area, but a wireguard accessible OOB connection is a great piece of kit to keep handy. I use a cheap 768kbps SIM in an Ethernet connected switch into my personal systems. It’s saved my skin numerous times.
I’m sure this is obvious, especially in hindsight, but just mentioning because the existence of IoT LTE data plans for a minimal fee ( $100/year for me in Midwest US) was NOT obvious to me until 2 years ago.
Yeah I had plans to set up something like that, but always other priorities and in this specific case I could maybe access other internal servers but i would need KVM access to reboot the firewall or some other way to cut physical power. And exfiltrating hundreds of GBs of lemmy database wouldn’t work over such a small pipe either.
This is something that even larger corps struggle with. My old company would buy some other company, lay off a bunch of people, others would quit, and then it was shocked Pikachu faces all around when That One Thing stopped working in an office they turned into a ghost town and, well, no, nobody was going to be there until Monday morning to power cycle etc. True lights out/OOB reachability is WORK. And there’s always going to be a SPOF somewhere that requires hands on-site.
Do what you can, when you can. And thank you for all that y’all have done so far.
For that, I’d highly recommend a power conditioner with outlet controls (and preferably sequencing).
Most are pretty expensive, but you can get something like a surgex squid or a gude expert power control for a few hundred here and there, which will give you great data on power usage as well. You can also grab an older model with serial and/or relay controls, and use a pi or whatever is handy to trigger it. Actually have a setup like that in a few racks for clients (mostly orange pis and a couple of Asus tinker boards).
Hope your trip goes well!
Might it also be possible/feasible to send an “email blast” to everyone with a slrpnk.net account? It would probably be quite complicated to send that many emails at once without getting blocked by the major email providers, but hypothetically we’ve all registered with emails…
Just spitballing ideas for the future. Everything’s fine now. Gives me an excuse to spend a little more time on the Canadian end of the fediverse. :)
As email in profile is an optional feature on our instance about 50% of the userbase opted to not share one so we would only reach some people. But yes, getting spam filtered would be another problem of mass-mailing.
But I found a way to put up a simple html error page and will add this later to inform our members and link to this thread.
Lol love the new username, fitting! :)
Yarr, thank ye kindly, Lazy :p
Shit happens. Thanks for the update 🙂
Sending positive vibes, and passing offer of technical assistance once you get access again.
Hopefully it’s an easy fix once you get into things again.
That’s unlucky, but is it hosted at someone’s home?
Sort of yes. It is a homelab a few people have access to, but currently they are all either on holiday or on extended work related travel. It seemed like an acceptable risk as everything worked smoothly for years.
That makes sense, given the ethos of your instance, I hope you can fix it a soon as reason allows!
Good luck and all the best. Time to touch grass, everyone 😔
You mean create an alt, right?
That sucks. Thank you for your time and running the instance. Gonna miss you all like hell while you’re gone. 😞
Thanks for the info, a healthy amount of real life awaits then. :)
I knew it was running on solar energy and old hardware so I guessed something like this had happened.
If you need fail-over to awaken a backup system when the primary fails, things can be designed. :)
Sorry to hear that! My main account is on slrpnk.net. But don’t let it stress you out, “solarpunk” says “punk” which implies “DIY” which means this sort of thing happens. Much better to have an instance run by an admin who truly cares, rather than some soulless corporation.
Ty for the update! slrpnk is my favorite instance that isn’t my home instance
This is making me want to create an alt on slrpnk.net , since you’re so chill and communicative about issues.
Aren’t most admins this way?
*glances at your instance name*
Ah…
Lol. Yeah, sdf.org feels pretty hands-off. They have longevity going for them–I’ve had an sdf account of some sort since 2009 at the latest. The system started life as a BBS in the 80s. The name comes from the anime “Super Dimension Fortress”, which I’ve never seen.
Given that the single admin @SDF@lemmy.sdf.org hasn’t been active in two years, yeah, I’d say that counts as “pretty hands-off”. The SDF organization as a whole will probably be around for decades to come, but they don’t seem to care that much about their Lemmy instance.
They have stepped in when a lot of people complained, it took their whole instance complaining to do so while a Nazi troll spammed hate across the local feed.
I miss BBS.
you can still have it with NodeBB!
Are there modem sounds to play air guitar to?
@blazeknave@lemmy.world
beep beep boop boop boop boop beep... EEEEEEEEEEeeeeeeee awwwwwww ka-dong ka-dong da kshhhhhhhhhhhhhhHHHHHHHHHHHHHHHHHHHHHHHHHH
eerie silence
You’re very welcome! Kris@feddit.org / @poVoq@slrpnk.net is an incredible admin, and up until now has declined donations, funding the entire instance several years now from his own pocket.
This current downtime is the exception that proves the rule. I hope he’s not stressing out too much. We’ll all be ready to jump right back into it when he gets back.
Already thinking about how to make the best of it 🤷 Maybe we can use this opportunity to try and migrate to Piefed? I had this in the back of my mind for some time already and Rimu seems optimistic that it is possible.
That’d actually be pretty rad if it’s doable, especially now that the API for piefed was released and mobile apps are already beginning to support it.
What do you like more about Piefed compared to Lemmy?
Lemmy has a lot of individual parts that don’t interact very well with each other, especially the image host part. Futhermore the main UI is quite a mess and we were thinking of switching to an alternative already, but this would further increase the “too many moving parts” issue. Piefed on the otherhand has an integrated and very lightweight UI, which also has some nice additional filtering and moderation features Lemmy currently lacks.
And I personally feel more at home with the Python codebase, as it allows better troubleshooting and more standartized (Flask) tooling. The Rust codebase of Lemmy has a lot of obscure custom stuff and the error messages are extremely obstruse from a sysadmin perspective.
And looking at the performance metrics of Lemmy, the main limiting factor seems to be the Postgres database anyways, so the theoretically slower Python codebase of Piefed should not have much impact.
That would be so nice if we could move to Piefed.
Wouldn’t that mean the instance would completely restart? I mean I don’t know much about piefed but logically it seems like the DB structure would be incompatible. Thus “Migrating” to piefed means a complete reboot, all posts, all users, all content just bye bye.
Not necessarily, no. We aim to preserve users, communities and posts/comments. Image uploads might get lost though.
Such an in-place migration will need extensive database operations and likely some support by the Piefed developer (to add support for bcrypt hashed passwords), but we are hopeful to make it happen and maybe this will result in a database migration script other Lemmy instances could also use.
If this turns out to be infeasible, we will stay with Lemmy rather than reset everything.
Oh awesome, I mean I’ve heard of other platforms like sublinks offering database migration but I didn’t think it would be feasible with piefed since unlike sublinks it’s very different than lemmy.
Yeah! It’s definitely worth adding to my list of migration contenders now that lemm.ee is closing and I have to jump ship
Yeah me too, I’d wanted to for a while but never got around to it. Now I probably never will (unless they get the servers back up at some point in the future).
it seems they will, just not too soon
He’s pretty awesome. He has a monthly thread with updates and actually responds to you too.
Hey fellow slrpnk.net refugees… if you’re looking to recreate your list of subscribed communities, here’s a good place to start:
click here for a list of communities that are NOT politics, tech, or meme -related.
Most are currently active (except for the ones with a * which were less active last I checked) Sometimes politics, tech or memes sneak in but they’re not the focus. GENERAL DISCUSSION / QUESTIONS - !ask@lemm.ee - !asklemmy@lemmy.world - !askmeanything@lemmy.ca - !casualconversation@lemm.ee - !casualuk@feddit.uk - !Dullsters@dullsters.net - !dull_mens_club@lemmy.world - !forumlibre@jlai.lu - !general@lemmy.world - !nostupidquestions@lemmy.world - !goodoffmychest@lemmy.world - !preguntalealemmy@feddit.cl - !rant@lemmy.sdf.org ART / PHOTOS - !artporn@lemm.ee - !astrophotography@lemmy.world - !accidentalrenaissance@lemmy.blahaj.zone - !badrealestate@feddit.uk * - !collage@lemmy.ml * - !desire_paths@sh.itjust.works - !fakehistoryporn@lemmy.world * - !fediclicks@viewfinder.pro * - !filmsframes@lemmy.world - !foodphotos@sh.itjust.works * - !gameart@sopuli.xyz - !grasweeti@slrpnk.net - !hmmm@lemmy.world - !imadethis@lemm.ee - !imaginary@reddthat.com - !inhabitedbeauty@lemmy.world - !liminalspace@lemmy.world
- !mobile_wallpaper@lemmy.dbzer0.com - !nocontextpics@lemmy.world - !pics@lemmy.world - !pictures@lemm.ee - !pixelart@retrolemmy.com - !pixelart@lemmy.ml * - !sillydrawingrequests@sopuli.xyz - !traditional_art@lemmy.world - !trams_trolleys_streetcars@lemmy.blahaj.zone - !wallpapers@lemmy.ca - !52weeksofart@lemmy.world ANIMALS - !bats@lemmy.world - !catposting@lemmy.blahaj.zone - !cat@lemmy.world - !catsstandingup@lemmy.world * - !opossums@lemmy.world - !scrungycats@lemmy.world * - !superbowl@lemmy.world COMICS / GRAPHIC NOVELS - !alanmoore@feddit.uk * - !comics@lemmy.blahaj.zone - !comicstrips@lemmy.world - !eurographicnovels@lemm.ee - !truecomics@midwest.social * - !garfield@lemmy.world - !outofcontextcomics@lemmy.world - !unix_surrealism@lemmy.sdf.org ENTERTAINMENT - !80smusic@lemmy.world - !90smusic@lemmy.world
- !90s_tv@lemmy.sdf.org - !badmovies@lemmy.world
- !bmoviebonanza@lemmy.world - !Bside@fedia.io - !boardgames@sopuli.xyz - !cultfilms@lemux.minnix.dev * - !dwarffortress@lemmy.ml - !edm@reddthat.com - !fullmoviesonyoutube@lemm.ee - !iasip@sh.itjust.works
- !martialartsmovies@lemm.ee * - !metal@lemmy.world - !movies@lemm.ee - !music@lemmy.world - !pokemon@lemm.ee - !pokemon@lemmy.world - !pokemongo@lemmy.world * - !ska@lemmy.blahaj.zone - !thrashmetal@lemmy.world - !trailers@lemmy.blahaj.zone GENRES / STYLES - !cassettefuturism@lemm.ee - !cosmichorror@lemm.ee - !cyberpunk@lemmy.zip - !fuckyournelsonlamp@lemmy.world * - !goblincore@lemmy.blahaj.zone - !gothindustrial@lemmy.world
- !lovecraft_mythos@lemmy.world - !mullets@lemmy.ca * - !punk@lemmy.world - !punk_rock@lemmy.ca - !raygungothic@lemm.ee - !streetwear@lemmy.world * - !vampires@lemmy.zip - !weirdcore@sopuli.xyz - !witchcraft@lemmy.ml * - !witchesvspatriarchy@lemmy.ca
HISTORY - !ancientcoins@lemmy.world - !ancienthistory@slrpnk.net - !artp@sh.itjust.works * - !askhistorians@lemmy.world * - !historyartifacts@lemmy.world - !historicalpropaganda@lemmy.blahaj.zone - !history@lemmy.world - !historyporn@lemmy.world - !historyruins@lemmy.world - !historyillustrations@lemmy.world - !vintageads@sh.itjust.works * INFORMATION / KNOWLEDGE - !buyitforlife@slrpnk.net - !catastrophicfailure@lemmy.world * - !creepywikipedia@lemmy.world - !dataisbeautiful@lemmy.ml - !documentaries@lemmy.cafe - !esa@feddit.nl - !explainlikeimfive@lemmy.world - !goodnewseveryone@sh.itjust.works - !humanities@beehaw.org - !interestingasfuck@lemmy.world - !math@lemmy.world - !mealtimevideos@lemmy.cafe - !recommendations@lemmy.world - !solarpunk@slrpnk.net - !thingscutinhalf@lemmy.world - !til@lemmy.world - !vexillology@lemmy.world
- !whatisthisthing@lemmy.world - !wikipedia@lemmy.world - !youshouldknow@lemmy.world OTHER - !andfinally@feddit.uk - !academiagonewild@lemmy.world * - !internetisbeautiful@lemm.ee - !mildlyinteresting@lemmy.world - !nowhereelsetoshare@sh.itjust.works - !pbsod@lemmy.ohaa.xyz - !publictransport@slrpnk.net - !stick@sh.itjust.works * - !sumo@lemmy.world * - !shortstories@literature.cafe - !treehuggers@slrpnk.net - !the_heinlein_society@lemmy.world - !woahdude@lemmy.world * - !youcantparktheremate@feddit.uk * FEDIVERSE - !fedibridge@lemmy.dbzer0.com - !fedigrow@lemm.ee - !fediverse@lemmy.world - !fediverse@piefed.social - !fediverselore@lemmy.ca - !lemmyapps@lemmy.world - !comm_revival@lemmy.dbzer0.com * - !yepowertrippinbastards@lemmy.dbzer0.com FINDING NEW/GOOD COMMUNITIES ON LEMMY - !communities@ponder.cat - !communitypromo@lemmy.ca - !findacommunity@lemmy.ml - !index@lemmy.world * - !lemmy411@lemmy.ca - !newcommunities@discuss.online - !newcommunities@lemmy.world - !trendingcommunities@feddit.nl - !wowthislemmyexists@lemmy.ca *
Great list, as usual!
Good stuff, Sergio! :D
You will unfortunately have to get rid of the lemmy.ee links since they are closing down very soon.
lemm.ee/post/65824884
Many of the .ee communities are in the process of migrating, so hopefully the list can be updated with their new homes.
This is actually great timing, i’d say. Just now most of the northern hemisphere is starting into summer. At least i deeply feel it on a personal level that summer is arriving. I take this as a gift from the universe that directs me towards touching grass, and going swimming. Enjoy your summer, you all! Connect with your local community, if you can.
thirty minutes later
Okay I touched grass and now I’m bored and itchy
Glad that at least someone sees a positive side of it, because I sure don’t… well at least it gives me a bunch of ideas how to avoid such situations in the future 🤷
Try not to stress too much about it while you’re away! If nothing can be done, nothing can be done. Solarpunks need to be resilient to things like this anyway. Community over convenience.
Thanks, I am trying, but it really is quite annoying to lose access like that.
Thanks for letting us know! Try not to worry about it too much, it’s all good
Thanks for the update, good luck with your trip.
Thanks for the heads up! My Main account is in slrpnk, so I’ll set a reminder in an month to switch back. :)
Opengear cellular backup appliance with serial/console connectivity to hardware.
I this it’s a law that servers run 100% perfect until the literal day one leaves town with zero way to return home. One of the many reasons I got all my services off of unraid.
Very cool to learn you’re running your own machines. Do you go into detail about this anywhere?
We have a small write up about the hardware on our wiki, but it is also down right now.
I think we will share a post-mortem write up of the actual improvements we will do to avoid this in the future.
One thing I will definitly do is to add a KVM remote management console to one of our server boards and move the main firewall into a VM with hardware passthrough of the NICs (this was anyways planned for a 10gbit network upgrade for the second half of 2025). This way I should be able to reboot and even reinstall the main ingress point remotely, so that only the fiber gateway remains as a failure point that requires physical access.
Thanks for all that you do for us and this community! A little downtime won’t hurt anyone. I would also be in gavour of PieFed migration 💙
I’d probably make an account there if it moves to piefed.
I totally agree with everything you said :)
I just created an account in PieFed and it looks very interesting because there is the option to create your own feeds! I had the wrong impression that you can only choose from the existing ones. This is so exciting!!!
That sounds pretty cool. I don’t know much about Piefed — would we still be able to access posts from Lemmy instances?
Yes, they’re 100% cross compatible. As an example, you can access !piefed_meta@piefed.social from your Lemmy instance seamlessly.
I appreciate the update was thoroughly confused
Thanks for letting us know what is going on.
Sorry that it took so long to get the word out. I guess we relied a bit too much on the xmpp server for internal communication.
Even the big operations break in unexpected ways. Facebook accidentally broke the authentication for their door locks when they took the site down a few years back and had to fly people to a datacenter and break down a door to get to servers and restart things.
We managed to resolve the issue and I have many ideas how to avoid this in the future 🤓
I am a slrpnk user. Oh well, shit happens. What makes me a bit sad was that a few months (?) ago something similar happened. Slrpnk.net was down for aa day or two? Don’t remember the exact amount of time. What I do remember is asking if there was intention of adding a second admin and making the point that it would be good to create that sort of redundancy. I was ignored.
Mid of july is basically 1 and a half months from now. Hopefully a new solarpunk instance emerges, one that is safer from such scenarios.
Enjoy your vacation and don’t stress too much! Shit happens.
I am genuinely confused. What makes you think you were ignored? We do have additional admins, the problem is physical access to the servers to replace hardware that seems to have broken.
We plan to add additional safeguards to such scenarios, but in the end it is a trade-off between reliability and costs. Running a Lemmy instance of our size on a fully managed rented server would easily triple the monthly costs. This would be beyond what I am willing to cover out of my own private pocket, thus making the instance dependent on donations. And that IMHO makes it more likely to be shut down in the future as donations are very variable.
I am not on vacations, but on a work deployment abroad, which further limits the time I can spend on this. Not that it would have made much of a difference, but at least the domain redirection and placeholder text was only possible to do on the weekend.
Well, I got no reply and I didn’t see anyone else in the “mods:” bit in the bottom of the sidebar on slrpnk.net. I understand that having more admins wouldn’t help in this specific case, but it would help in others.
Regardless, having a public lemmy instance only ran by 1 person is, in my opinion, a bad practice. And please know that I am not personally attacking you. I am just stating my thoughts. This time you were out on a work deployment. Next time you could be sick. The next time maybe a family member had an emergency and you had to help. And the next time maybe your house caught fire or whatever.
Resilience and redundancy is extremely important in public lemmy instances. And SolarPunk is, in my opinion, all about community. There should be more (trustworthy) people with the same level of access as you do.
Regarding the price and all that, maybe that is an interesting discussion to have with the slrpnk.net community? And it would also be good to let slrpnk.net users know how the instance is being ran, what hardware, where, etc, so users can be an educated decision on whether they want to see an instance or not? Maybe providing that info on the sidebar would be nice?
But again, shit happens. Hopefully this serves as a learning event. If not, so much shit is going through the world, losing a lemmy instance is not something about which one should lose sleep.
EDIT: I may be missremembering about the whole “something similar happened a few months ago and my question about new admins was ignored”… I think that might have been on mander.xyz and not slrpnk.net. I really can’t fully remember, but I think it was mander.xyz after-all. Sorry! The rest of the post stands though!
We have been very transparent about that and documented that in the wiki which is linked in the sidebar. But it seems few people ever read the Wiki and having it integrated like it is with Piefed would probably help.
I agree that more admins with remote access would be good to have, and we have been working towards adding more (there are already more than one, who are not directly involved with slrpnk, but rather the mother project f-hub.org), but physical access will always be limited by the location (Azores) unless we move to an entirely different data-center (which in turn would make it hard for me to access the physical servers).
Oh ya. I knew about that, and I am fine with that. Though I think having it behind a link is bad UI. People will not read it haha… It should be presented on account creation, in my opinion. Just 1 sentence “This instance is ran on private hardware and may suffer some down time here and there” would suffice.
Sure, that seems reasonable. We will add that when the instance is back online.
If I forget, please remind me. For sure we will not just ignore you 😊
I see in your edit that you confused us for a different instance, but I want to mention regardless that Slrpnk.net has 3 active admins, it’s not just poVoq (Kris).
Found this post after a couple of days of hopeful reloading of the page! Took another couple of days to log in an old alt just to say I’m glad you communicated this, and let us know how we can help to make it more resilient (if at all necessary).
To the kind soul that was helping me with my greenhouse infested ants (in the unlikely event they are reading here - but iirc they were not from slrpnk.net but just passing by) I’m glad we spoke, it seems like they are harmlessly using my panels for hatching, and then moving on! Related, who has a nice gardening/greenhouse community to suggest? :)
I have heard of the migration to Piefed, but will it remove the (very cool) aesthetics of the lemmy ? I loved it :'(
I think we will try to come up with a theme for Piefed with similar colors.
yayyy :D