What's a reliable filesystem for long term data archival that also supports transparent file compression?
from HiddenLayer555@lemmy.ml to linux@lemmy.ml on 15 Aug 06:00
https://lemmy.ml/post/34700384
from HiddenLayer555@lemmy.ml to linux@lemmy.ml on 15 Aug 06:00
https://lemmy.ml/post/34700384
I currently use btrfs on almost all my server and offline backup drives mainly because of the ability to transparently compress files to save space. But apparently btrfs can get corrupted if it unexpectedly loses power? Is there a more robust and corruption resistant filesystem that you’d recommend that also has the ability to compress files?
threaded - newest
ZFS
I believe ZFS fits your needs. Using ZFS with TrueNAS I can verify it supports transparent compression. It’s supposed to be very stable against data corruption. The only thing I can say about data corruption is in my 10+ years of using it, I haven’t had any issues.
It never had data corruption issues in 30+ years on EXT4 either… It doesn’t mean that you had none.
ZFS has built in error checking and correction if you have parity data. Its less if they have had corruption and more that it doesnt matter, because their system is designed to correct it automatically anyway.
With EXT4 over 30 years, you've likely been lucky or unaware of any corruption. Guess thats the same thing in the end.
For the OP, in case you go for zfs, don’t forget to create a cron job for scrubbing, truenas has support for this built in. It repairs things like bit rot, should they occur.
Good looking out.
Corruption on power loss is something that journaling should prevent and modern filesystems do that. So btrfs should be as safe as other modern filesystems.
Edit: Btrfs does no journaling, it implements resilience with another mechanism, namely Copy on write.
It should, but apparently it isn’t, at least not reliably.
I’ve lost power with it many times, none of my BTRFS partitions have been corrupted so far.
It’s perfectly reliable. You only need caution when using RAID5 and RAID6 as these features are still in the experimental phase.
If you’re using consumer HDD’s or SSD’s then they themselves can get corrupted in power failures, so the software almost doesn’t matter. Better use a UPS.
What do you mean by that?
Afaik this is only a problem if you use RAID 5 or 6. Otherwise it’s as solid as other CoW filesystems like zfs.
btrfs.readthedocs.io/en/latest/btrfs-man5.html#ra…
Are you using RAID at all? If so ZFS is probably the way to go. If not I think it matters less whether you use either btrfs or ZFS.
Regarding btrfs and power loss:
unix.stackexchange.com/…/does-btrfs-guarantee-dat…
…btrfs is designed to only experience data loss not corruption, assuming well behaving hardware in power outage scenario. In practice ZFS has more maturity overall (definately) so may be better (my speculation).
Beyond direct comparisons if you already have on and offline backups then you are protected from power corruption and only have to worry about data loss anyway?
I think for home archival use is btrfs is a terrible idea. I burnt myself with it so many times.
Yes it should be safe and resistant to unclean shutdowns, but my experience with reliability is not great.