Slrpnk instance is down till mid July; they might relaunch their server on piefed.
from Pro@programming.dev to technology@lemmy.world on 07 Jun 08:46
https://programming.dev/post/31774896

Source. Thread.

#technology

threaded - newest

A_norny_mousse@feddit.org on 07 Jun 09:18 next collapse

Assuming slrpnk stands for solar punk - that produces funny little theories in my mind about the server location.

Zombie@feddit.uk on 07 Jun 09:50 next collapse

Their servers are located in Portugal I believe and do indeed run on solar power! They gave details of their set up on their wiki page but… that was on the server that’s gone down so… have a read in July I guess?

technohippie@piefed.social on 07 Jun 09:54 next collapse

I think they're located in the Azores, and yes, running their own servers on solar (of course) and other renewable energy. Coolest instance out there.

madjo@feddit.nl on 07 Jun 10:04 collapse

I hope they come back :)

PirateFrog@lemmy.dbzer0.com on 07 Jun 10:17 collapse

We will, promise! :D

SkaveRat@discuss.tchncs.de on 07 Jun 09:58 collapse

it’s on archive.org

web.archive.org/web/…/server-hosting

Server hosting

This is a work in progress
Slrpnk.net is hosted by F-hub.org, a volunteer driven and non-commercial effort to host federated community services in a resource efficient and ethical way. F-hub.org grew out of a community of open-source game developers (freegamedev.net) that has existed since the early 2000s.

All the servers are operated and maintained (as a hobby) by one of the founding members and are currently located on the Azores in Portugal. Connectivity is provided through a dedicated high-speed fibreglass connection.
Hardware

The F-hub.org servers are based on second-hand consumer PC and data-centre hardware, but optimized for low energy consumption. Battery backup power is provided and data is stored with triple redundancy (off-site backups are still a work in progress). Electricity is currently provided by the utility grid (about 60% green-energy, mostly from a geothermal power-plant) but it is a work in progress to upgrade to a on-site solar PV system for near 100% renewable power.

Slrpnk.net itself currently runs on a dedicated 6th gen Intel CPU server with 8 threads, 16GB RAM and SATA SSD storage in raid configuration. Image uploads are stored on a large HDD raid array. All data is snapshotted and transferred for backup to a second shared server on daily basis.
External services

Given how difficult it is to have outgoing email accepted by the large email providers, all emails are currently routed through an external SMTP server hosted by OVH in France (same as the domain registration and DNS routing).

There are backup plans to move the servers to a co-location data-center owned by Altice, should this be required for scaling. However, monthly rent of rack-space is quite costly (starts at around 300€/month), thus this will require a substantial regular donation base to be possible.

pixeltree@lemmy.blahaj.zone on 07 Jun 12:54 collapse

Lol in my head it stands for slurp nick

CmdrShepard49@sh.itjust.works on 07 Jun 14:01 next collapse

It’s clearly “slurp nook,” that nook where you can go to slurp soup as loudly as you want without being judged.

huppakee@lemm.ee on 07 Jun 20:48 collapse

Makes most sense

ayyy@sh.itjust.works on 07 Jun 17:45 collapse

I’m glad I’m not the only one lol

Whelks_chance@lemmy.world on 07 Jun 09:58 next collapse

Why is this a jpg?

14th_cylon@lemm.ee on 07 Jun 10:19 next collapse

Just be happy that the jpg is not embedded in docx

Valmond@lemmy.world on 07 Jun 10:29 collapse

Itself embedded in an auto extractable zip executable.

kurcatovium@lemm.ee on 07 Jun 15:25 collapse

Archived once more in RAR, just for sure.

dubyakay@lemmy.ca on 07 Jun 15:50 collapse

And then tar.bz2 on top so that no basic users of sidewalks can ever open it.

Zachariah@lemmy.world on 07 Jun 11:42 next collapse

ikr

svg, lossless webp, or png are way better options for an image of text

exu@feditown.com on 07 Jun 12:34 collapse

JPEG XL would be the true future format, unfortunately Google killed it

Zachariah@lemmy.world on 07 Jun 13:00 next collapse
drkt@scribe.disroot.org on 07 Jun 13:27 collapse

Not yet they haven’t

exu@feditown.com on 08 Jun 06:52 collapse

They’re doing their best though

SpicyToaster420@sopuli.xyz on 07 Jun 16:38 next collapse

It wouldn’t be meme-worthy were it lossless.

SpicyToaster420@sopuli.xyz on 07 Jun 20:10 collapse

Is this loss dot jaypeg

Damage@feddit.it on 07 Jun 18:18 collapse

Because fuck blind people, that’s why

drspod@lemmy.ml on 07 Jun 10:44 next collapse

Alt text:

Slrpnk.net is currently offline due to an unforseen hardware failure in combination with the main system-administrators having no physical access to the server location until mid July due to work or summer-holiday related travels.

We are very sorry for this unforseen down-time, but slrpnk.net will return for sure and we already have some plans for a nice relaunch, so stay tuned!

souperk@reddthat.com on 07 Jun 10:53 next collapse

Wondering if a volunteer could go to the physical location necessary to restore service. If it’s in Athens Greece, I can make the trip.

Edit: The server is located in Azores, the flight would cost 415eur, I cannot afford that, maybe someone in Portugal could help?

A_norny_mousse@feddit.org on 07 Jun 11:39 next collapse

Off topic, but in Portugal was the first time me, a Central European, ever ate a ripe Papaya. Yum! Thanks to the Azores. The shit you get in the supermarkets round here is useless. Same for mangoes, usually.

pdqcp@lemmy.dbzer0.com on 07 Jun 14:23 collapse

You are in luck then, because both are climacteric fruits, meaning they continue to ripe after being harvested, due to producing ethylene gas that triggers the ripening

Throw them in a paper bag and let them rest at room temp. If you want to speed up the process, add a ripe banana or apple (or other ripe climacteric fruits)

A_norny_mousse@feddit.org on 07 Jun 14:29 collapse

The taste just isn’t the same.

pdqcp@lemmy.dbzer0.com on 07 Jun 14:39 collapse

You might have eaten a different cultivar than those exported. In that case, I agree, there are so many different varieties that we never get to taste, supermarkets usually only have 1 or 2 different types

AnarchistArtificer@lemmy.world on 07 Jun 19:15 collapse

It’s baffling to me how customisable some fruit features can be. I know someone who works in plant genomics and cultivation and it’s really cool how they can use genomic analysis alongside selective breeding to select for traits. I imagine that imported papayas are cultivars that are easier to transport internationally.

IcyToes@sh.itjust.works on 07 Jun 12:36 collapse

Is it in a data centre or someone’s house? If the latter, would they let a stranger in?

Surely they would need a backup and replicate db to so in case of hardware failure they switch over.

Sounds like they could improve their setup.

Too much of a single point of failure.

Kris@feddit.org on 07 Jun 13:55 next collapse

Slrpnk.net admin here.

The failure seems to have been in the main firewall, if it had been the server itself we could have easily restored it on another server from the backups on another machine. But as it stands, remote access is entirely cut off.

There usually is another person with hardware access, but they are on summer holidays. This seemed like an acceptable risk at the time…

An off-site backup would have been nice of course, but due to the costs involved in running an Lemmy instance of that size on a rented server, it would have not been a great option either.

I have plans to add a KVM to the main firewall via a secondary connection, but even that might have not helped in this case. I’ll know more when I have physical access again.

nickwitha_k@lemmy.sdf.org on 07 Jun 20:37 next collapse

I’ve done a lot of SysAdmin and DCOps stuff in the past so, thought I’d give you some plausible suggestions (haven’t dug deep into Lemmy DB stuff and DNS/Federation of the stack, so not sure all is practical).

Scenario 1 - Preserve and merge when access is restored

Setup

  • Spin up two VMs/VPS (or one that has enough grunt for two Lemmy servers). Call them robak.slrpnk.net and slrpnk.net and point DNS appropriately.
  • Pull federated content from other instances and place it on robak, set as read-only.
  • Sync important comms to (new) slrpnk.net without content.
  • Allow users to sign up, vetting as possible (all mods). Keep a list of those that are vetted (call it vetted.list). Inform all users that any non-vetted users will have their content dropped when access is restored.

Merge!

  • Once access is restored, ensure that (old) slrpnk.net is set to read-only.
  • Schedule a maintenance window (announce more time than you are likely to need).
  • During the maintenance window, put (new) slrpnk.net into R/O, or just block external access.
  • Query the db on (old) slrpnk.net for all users.
  • Subtract the vetted users from vetted.list from the list.
  • Drop all records from the resulting list of non-vetted users from (new) slrpnk.net.
  • Insert the records from vetted and new users (those without conflicts) into the DB on (old) slrpnk.net.
  • Validate that everything is working
  • Cut over DNS and spin down the new VMs/VPS.

Scenario 2 - Server is in DC or Admin able to facilitate access

  • Get a db dump/backup.
  • Spin up temporary slrpnk.net on a VM/VPS.
  • Use backup of temporary server to restore data to original, when possible.
IcyToes@sh.itjust.works on 07 Jun 21:58 next collapse

Appreciate the answer and the detail. Good luck getting it all resolved.

sparky@lemmy.federate.cc on 08 Jun 06:45 collapse

Is it run out of a private residence? How could it happen if it’s in a real data center…?

Kris@feddit.org on 08 Jun 07:59 collapse

It is run from a private residence in the DIY punk spririt (and this also allows us to run of a local solar PV system), but more or less the same would happen if you rent rack-space in a “real” data-center. Only if you rent a managed server or VPS someone else will be responsible to fix such issue and this comes at a significantly higher cost at the scale we operate at (slrpnk is part of a bigger project that also hosts other services).

db0@lemmy.dbzer0.com on 08 Jun 09:13 collapse

If you rent rack space, you can at least call the DC to send someone to reboot it or smt.

Kris@feddit.org on 08 Jun 09:40 collapse

Sure, but they will not replace hardware and set up remote access for you, which seems like this would have been required in this case.

MangoPenguin@lemmy.blahaj.zone on 07 Jun 17:14 collapse

Probably quite expensive, and when doing something as a hobby it’s often hard to get the funds.

roofuskit@lemmy.world on 07 Jun 13:30 next collapse

After a month and a half downtime all the users will have moved on to other instances. This is essentially a death sentence for the instance and its communities.

henfredemars@infosec.pub on 07 Jun 13:35 next collapse

You may underestimate user dedication. I love my instance and totally would come back after a few months away temporarily.

You got me on the communities though.

Fediverse is designed to handle servers with less than stellar reliability.

TrackShovel@lemmy.today on 07 Jun 13:45 next collapse

You underestimate the userbase. I made a temp account in the mean time, but we are a hyper tight knit community. We will probably lose accounts - no question - but the core userbase will return

akrenion@lemmy.cafe on 07 Jun 17:51 next collapse

The local communities are such a great mix of uplifting and informative posts that I am sure to return.

AnarchistArtificer@lemmy.world on 07 Jun 19:10 next collapse

I agree. I replied to Kris elsewhere saying this, but I am super glad to have been a part of this instance because it feels like a nice balance of being large enough to be robust and diverse, but small enough to have a distinctive culture. I don’t often interact with the communities that were on the instance, but I always enjoy seeing my peers crop up in the comments of various posts — it’s one of my favourite parts of Lemmy being federated (db0 is another example of an instance that has such a distinctive vibe that seeing it as someone’s instance is often useful metadata that affects how I parse their comment)

pjusk@lemmy.dbzer0.com on 07 Jun 22:32 next collapse

Count me in! Slrpnks all the way! 💚

Kris@feddit.org on 11 Jun 20:10 collapse

Time to come back, it is working again. Lucky break 😅

Coolkat@jlai.lu on 12 Jun 12:18 collapse

Thanks for the speed and the work !

Kris@feddit.org on 07 Jun 14:01 next collapse

Lets see. I think a relaunch on Piefed might interest some people to come back, and most slrpnk communities are rather niche and will probably stay. /c/climate might move though.

fmstrat@lemmy.nowsci.com on 07 Jun 14:21 next collapse

I think the migration to Piefed would be worse. Piefed is great, but needs better App support for Lemmy users to migrate. A merge of the Thunder fork into main Thunder would get the ball rolling.

Tag @Kris@feddit.org

Kris@feddit.org on 07 Jun 14:28 next collapse

I think now that Piefed has an API for apps, we will see some of them adding support soon. Overall I think the benefits of a Piefed migration outweight the disadvantages, but it remains to be seen if it is doable.

GreenKnight23@lemmy.world on 07 Jun 18:06 collapse

you mean an API for bots…

tfm@europe.pub on 07 Jun 19:14 collapse

Mobile Apps need an API too.

GreenKnight23@lemmy.world on 07 Jun 19:34 collapse

why does everything need a mobile app?

tfm@europe.pub on 07 Jun 19:42 collapse

It’s not just native Apps. Alternative web UIs like Thunder, Photon and Voyager need them too.

GreenKnight23@lemmy.world on 07 Jun 19:58 collapse

yes, but those frontends are typically tied closer to the backend than a public API.

things like CSRF can help block abuse of the back end.

tfm@europe.pub on 07 Jun 20:04 collapse

Nope they all use the public API. Even the default Lemmy web client.

GreenKnight23@lemmy.world on 07 Jun 21:04 collapse

well that’s poor planning and why bots are such a problem.

I know CSRF tokens aren’t a silver bullet, but doing nothing to stop them does nothing to stop them.

tfm@europe.pub on 08 Jun 05:14 collapse

CSRF protection is a security feature not bot prevention. A bot would just need to get a token first.

xnx@piefed.social on 07 Jun 19:52 collapse

On Android you can use the Interstellar app https://interstellar.jwr.one/ with piefed.social! Also iOS: https://testflight.apple.com/join/JXRtHye2

ayyy@sh.itjust.works on 07 Jun 17:38 next collapse

They didn’t need you anyway.

AccountMaker@lemmy.dbzer0.com on 07 Jun 19:33 next collapse

Idk, quite a few (me included) made alt accounts for the time being, and we’ll return when it becomes possible, especially if we really do migrate to piefed.

hazeebabee@lemmy.dbzer0.com on 07 Jun 20:11 next collapse

I dont think it will be. I’ve had two accounts for a while to deal with unexpected issues & will happily return to my slrpnk account once it’s back up and running :)

Plus for the communities, people who were subscribed to them before will likely stay subscribed & once the instance is back up the posts will be in their feeds without an issue.

Being able to accommodate issues like this is one of the major upsides to a decentralized platform.

lautre@jlai.lu on 08 Jun 05:58 collapse

I don’t think so, I will definitely go back and do not wish to move permanently in another instance.

As others have said I think this instance has a strong base of dedicated users, it’s a “specialized” instance that has no equivalent.

TrackShovel@lemmy.today on 07 Jun 13:43 next collapse

Thank you so much for the update - I hadn’t had access for about a week and was getting worried. Couldn’t reach them on XMPP either.

Kris@feddit.org on 07 Jun 14:14 collapse

Yeah, the xmpp server is down too. That is something that bugs me quite a bit and I will probably move that one to an external small VPS to retain a more secure backup communication channel.

AnarchistArtificer@lemmy.world on 07 Jun 19:02 collapse

(speaking as a slrpnk user): Another backup communication strategy (once things are backup) might be to designate somewhere on a non-slrpnk instance as a place where people can check for updates if things go down; when I first discovered the outage, I wasn’t sure where to go to check for info/updates.

Unrelatedly, I hope that this unexpected outage isn’t causing you or other admins too much stress. Whilst the extended nature of this outage is unfortunate, I respect that you’re using this as an opportunity to migrate to a more robust solution. This kind of resilience focussed response is a key part of the solarpunk ethos, in my view.

Some people have said that such a long outage seems likely to kill an instance, but for my part, this community is worth waiting for — I have enjoyed having an account on this instance because it feels like the perfect blend of small enough to have a distinct culture and ethos, but is large enough to be robust and diverse.

Kris@feddit.org on 07 Jun 19:50 collapse

Thanks.

Yeah at the very least this will result in some changes in the hardware setup to avoid similar situations. But of course it is impossible to forsee all possible problems.

ClockworkOtter@lemmy.world on 07 Jun 14:02 next collapse

I’ll be back when they get back up.

underline960@sh.itjust.works on 07 Jun 14:43 next collapse

<img alt="" src="https://sh.itjust.works/pictrs/image/fdc41b98-c2de-4f93-9a5b-25ce7a37ee1c.gif">

Honytawk@lemmy.zip on 07 Jun 17:36 next collapse

Did they run out of sun?

Tangent5280@lemmy.world on 07 Jun 18:04 next collapse

Maybe is vacationing in United Kingdom

JackbyDev@programming.dev on 08 Jun 05:55 collapse

No, just punk.

philpo@feddit.org on 07 Jun 17:48 next collapse

Don’t pull a feddit.de on people, alright?

squaresinger@lemmy.world on 07 Jun 20:50 collapse

What exactly happened there? It was the big thing, then I didn’t use it for a month or so and then it was gone.

philpo@feddit.org on 07 Jun 21:44 collapse

The admin basically ran it as a one man show with only one other admin who had very limited privileges. He then went on a “business trip” or workaction or longterm vacation - there were different stories. Anyway, the database went belly up, the other admin couldn’t do a thing and none could contact the admin. There are some rumours that he wasn’t who he claimed he was and actually was a Chinese national who simply returned home, but who knows that. As a matter of fact none had any meaningful contact with him for months then and it appears he did not return. (But is alive)

A Austrian NGO who amongst others does host some mastodon instances,etc. took over and now feddit.org is on a very productive, professional and transparent level.

squaresinger@lemmy.world on 07 Jun 21:58 collapse

Thanks for the summary! That sounds freaky!

Well, the trade-off between trusting a huge corporation or a single dude on the internet.

JuxtaposedJaguar@lemmy.ml on 08 Jun 01:28 collapse

XKCD #2347

JackbyDev@programming.dev on 08 Jun 05:54 collapse

Ya gotta give a line from it or something, I don’t just see numbers and know what they are.

ernest314@lemm.ee on 08 Jun 06:10 collapse

from context, it’s probably the “single dude in nebraska holding up the entire internet” one

e: holy shit I got the state right

nickwitha_k@lemmy.sdf.org on 07 Jun 20:05 next collapse

That sucks.

Gsus4@mander.xyz on 07 Jun 20:34 next collapse

What is so special about piefed? I see a few communities moving there. The interface looks different from the original lemmy interface.

Gsus4@piefed.social on 07 Jun 21:30 next collapse

Well, you can add flairs :)

PirateFrog@lemmy.dbzer0.com on 07 Jun 22:28 next collapse

Our sysadmin explained some technical advantages here: feddit.org/post/13613230/7063696

qweertz@programming.dev on 08 Jun 01:42 collapse

Postgres slowing factor

I’m pretty sure one of the best optimised free/libre DBMS’s is faster than Python 💀

EDIT: skimmed it wrongly, see corrections below

Also no one knows how Piefed scales, since it only has like 350 MAU

PirateFrog@lemmy.dbzer0.com on 08 Jun 04:56 next collapse

Piefed also uses PostgreSQL. He was mentioning that the limiting factor on either platform is the DB, meaning that the parts written in Python will likely not be a limiting factor.

Piefed also has quite impressive optimizations in other areas as well compared to Lemmy or even Mbin.

JackbyDev@programming.dev on 08 Jun 05:51 collapse

Reread it. They’re saying “The slowest part is postgres, so even if python is slower than rust it won’t make a difference.”

SorteKanin@feddit.dk on 08 Jun 09:24 collapse

That is, in theory. In practice, it can make a big difference. Source: Worked professionally with large services built in Python and Rust.

Kris@feddit.org on 08 Jun 10:05 collapse

Yes, I also suspect that at a certain scale it would make a difference, but with a single Postgres database and no advanced clustering or so, the real-life performance metrics of a medium sized Lemmy instance strongly point to Postgres being the bottleneck and not the Python or Rust codebase of the rest of the software.

Kris@feddit.org on 08 Jun 10:01 collapse

For regular users there is not a huge difference, but the web-interface is significantly faster and you can subscribe to topics that combine multiple communities. The disadvantage is less mobile app support. Right now only Interstellar supports Piefed.

The real advantage is for moderators and admins, as the Piefed developers actually listened to community feedback and implemented a lot of nice moderation features that require bots or annoying work-arounds on Lemmy.

And there are a few technical differences that make Piefed easier to administrate and troubleshoot from a sysadmin perspective.

GreenKnight23@lemmy.world on 08 Jun 11:16 next collapse

“unforseen hardware failure”

somebody didn’t check their backups!

Kris@feddit.org on 09 Jun 14:54 collapse

The backups are most likely fine 😅

GreenKnight23@lemmy.world on 09 Jun 16:27 collapse

yes, that’s why it’s been down for this long…

Kris@feddit.org on 11 Jun 20:09 collapse

Slrpnk.net is unexpectatly back online 🥳