31 May
2019
31 May
'19
5:41 p.m.
With the discussion about RAID 10, that got me back to thinking about
better/alternative system to my current RAID1+LVM+EXT4 setup on our
Linux home server and I'm looking for advice from other members.
Currently we have 13.6 TB of storage (a lot of which are photos by my
semi-professional girlfriend, and videos from our wildlife cam which
produces about 15 - 20GB of videos a day [email me off-list if you want
to want the URLs of my fledgling Youtube hedgehog and bird channels]).
There is some amount of file duplication, for instance where I have
stuck old backups (copied files and folders, not tar/compressed
archives) on there or photos/videos have been copied to different
folders (e.g. to categorise), so filesystems with built-in deduplication
(like I believe BTRFS has) would be nice. However my main priorities
are: maintaining data integrity, ease of administration, and really a
sub-category of that: ease to expand or shrink and reallocate storage as
required (if necessary - quotas are not required, but crashing due to a
full disk is to be avoided).
For years I have been looking at BTRFS, but it's never sounded 100%
production ready to me (although I remember that at least one distro
made it their default fs). Andy's mention of ceph and stratis were
something new to me, but I'm not sure if they are a bit much for a
single server, and I've no experience with ZFS, but I think I read about
some disadvantages that put me off a few years back, but I forget what
they were now.
Anyway, what do/would you use for this sort of scenario/requirement or
what are your experiences with suitable filesystems for my
requirements? Just to be clear - I want to ensure that a single disk
failure is very unlikely to result in data loss. Also, all disks are
currently the spinning disk type, so any features that takes advantage
of SSDs would be wasted.
Thanks
Gavin
31 May
31 May
6:07 p.m.
On Fri, May 31, 2019 at 06:41:55PM +0100, Gavin Westwood wrote:
With the discussion about RAID 10, that got me back to
thinking about
better/alternative system to my current RAID1+LVM+EXT4 setup on our
Linux home server and I'm looking for advice from other members.
Currently we have 13.6 TB of storage (a lot of which are photos by my
semi-professional girlfriend, and videos from our wildlife cam which
produces about 15 - 20GB of videos a day [email me off-list if you want
to want the URLs of my fledgling Youtube hedgehog and bird channels]).
There is some amount of file duplication, for instance where I have
stuck old backups (copied files and folders, not tar/compressed
archives) on there or photos/videos have been copied to different
folders (e.g. to categorise), so filesystems with built-in deduplication
(like I believe BTRFS has) would be nice. However my main priorities
are: maintaining data integrity, ease of administration, and really a
sub-category of that: ease to expand or shrink and reallocate storage as
required (if necessary - quotas are not required, but crashing due to a
full disk is to be avoided).
For years I have been looking at BTRFS, but it's never sounded 100%
production ready to me (although I remember that at least one distro
made it their default fs). Andy's mention of ceph and stratis were
something new to me, but I'm not sure if they are a bit much for a
single server, and I've no experience with ZFS, but I think I read about
some disadvantages that put me off a few years back, but I forget what
they were now.
Anyway, what do/would you use for this sort of scenario/requirement or
what are your experiences with suitable filesystems for my
requirements? Just to be clear - I want to ensure that a single disk
failure is very unlikely to result in data loss. Also, all disks are
currently the spinning disk type, so any features that takes advantage
of SSDs would be wasted.
I'd go for btrfs. (Disclaimer: I'm a regular contributor to the
btrfs community, and I've written quite a bit of their documentation).
I've been running a RAID-1 btrfs data store for about 10 years. In
that time, I've hit one bug that required me to restore from
backup. That was 9 years ago, when the FS was about as stable as an MP
in a three-way marginal constituency.
I'm now up to 11TB of data on a 13TB RAID-1, and it's been fine for
years, even over power failures and several disk failures.
The primary things you need to ensure is that your hardware is
reliable (no USB storage, and disks and controllers which honour flush
instructions), and that you have backups (because your hardware *will*
fail, eventually).
Scrub regularly, avoid compression and qgroups and parity RAID for
metadata, and you should be fine.
Note that RAID-1 is *not* a backup. You still need backups. RAID-1
is an uptime mechanism. RAID-1 will not save you from an accidental
"rm -rf /mnt/my-preciousss/". RAID-1 will not save you from a fire, or
a double disk failure, or a faulty TRIM implementation, or something
writing zeroes to the device, or running mkfs on the wrong device,
or... (yes, we've seen all of these in #btrfs).
If you can't afford backup *and* RAID-1, use half the disks in the
live system with "single" profile, and half the disks in the backup
machine with "single" profile. It's going to be safer than everything
in RAID-1 all attached to one machine.
For backups of a mostly-append data store such as yours, the
cheapest option in the ~4 TB -- ~150 TB range is BD-R. Over 150 TB or
so, it's cheaper to use LTO tapes. I'll need to update my spreadsheet
if you want more precise figures on that.
Hugo.
--
Hugo Mills | Great oxymorons of the world, no. 2:
hugo@... carfax.org.uk | Common Sense
http://carfax.org.uk/ |
PGP: E2AB1DE4 |
9:49 p.m.
Hello,
On Fri, May 31, 2019 at 06:07:45PM +0000, Hugo Mills wrote:
I'm now up to 11TB of data on a 13TB RAID-1,
and it's been fine for
years, even over power failures and several disk failures.
My experiences with btrfs and online disk replacement have not been
great. No lost data (partly thanks to you talking me through one
recovery a long time ago - thanks!), but have not yet managed a
single instance without having to reboot at least once.
The best thing I have to say about btrfs is that it's enabled me to
recycle the large number of variously-sized HDDs that BitFolk has
gone through as I can just slot them in without worrying about array
topologies and such.
For backups of a mostly-append data store such as
yours, the
cheapest option in the ~4 TB -- ~150 TB range is BD-R. Over 150 TB or
so, it's cheaper to use LTO tapes. I'll need to update my spreadsheet
if you want more precise figures on that.
I didn't know anything about BD-R until just now but it looks like
they max out at 100GB. So do you need to sit there changing discs
for the initial backup? I guess an incremental will fit on one disc
and then it's just a matter of how long you go before doing a full
checkpoint.
I am currently backing up about 530GiB to a rented Hetzner server
that costs me about €40/month so not cheap, but fairly convenient.
Would be much cheaper to build my own backup host out of cast off
hardware and only ongoing cost being the electricity, but I need it
outside my home for disaster recovery purposes.
I also periodically send a full copy to Amazon Glacier for emergency
purposes. This costs about $3/month to keep there, but if I were to
ever need to restore it, it would cost about $50 one off to download
it all. So I hope to never have to do that.
Since the end of the HP Microserver cashback offer I wonder what the
next iteration of my home fileserver would look like. Right now it
is one of the Microservers plus an 8 bay eSATA disk shelf.
I think those disk shelves are too expensive for what should be
quite a dumb device. Maybe it would be better to just buy one tower
case to fit the drives into. The nice thing about the Microservers
though is that you have a known good level of engineering and
design, whereas some cheap thing you buy and put together yourself I
feel you'd risk encountering all manner of niggling problems.
My requirements for such a machine would be:
- amd64 architecture
- Not immensely power hungry. Doesn't have to be crippled either,
but a CPU that doesn't need active cooling would probably be good.
- At least 10 hot swap 3.5" drive bays connected by SATA, or 8 of
them if I could install a pair of internal flash devices to hold
the OS.
So if anyone has built something like that recently on the cheap I'd
be interested to hear what you went for.
If the cashback offer was still there I'd be tempted to buy 2 or 3
Microservers and play with Ceph…
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
11:56 p.m.
On Fri, May 31, 2019 at 09:49:40PM +0000, Andy Smith wrote:
Hello,
On Fri, May 31, 2019 at 06:07:45PM +0000, Hugo Mills wrote:
Don't use anything other than the 25 GB disks. The higher capacity
ones are up to four times the cost per GB, and have been for some
years. It's not cost-effective, and I don't see that changing any time
soon.
And yes, you'd have to keep feeding disks in for the initial
backup. After that, I have a processing pipeline where all the
incoming files end up (eventually) in one directory, and a cron script
which runs a bin-packing algorithm on it, filling a temp directory
with hardlinks if it's found 25 GB ± 10 MB of data to write to
BD-R. It works very well for my append-only workload. It would work
much less well for a mostly-modify workload.
I started out writing CD-Rs about 15 years ago, migrated to DVD+R,
and then to BD-R. I'm considering LTO-6 as the next migration, despite
the poor economics w.r.t. BR-R at my kind of scale.
I'm now up to 11TB of data on a 13TB
RAID-1, and it's been fine for
years, even over power failures and several disk failures.
My experiences with btrfs and online disk replacement have not been
great. No lost data (partly thanks to you talking me through one
recovery a long time ago - thanks!), but have not yet managed a
single instance without having to reboot at least once.
The best thing I have to say about btrfs is that it's enabled me to
recycle the large number of variously-sized HDDs that BitFolk has
gone through as I can just slot them in without worrying about array
topologies and such.
For backups of a mostly-append data store such
as yours, the
cheapest option in the ~4 TB -- ~150 TB range is BD-R. Over 150 TB or
so, it's cheaper to use LTO tapes. I'll need to update my spreadsheet
if you want more precise figures on that.
I didn't know anything about BD-R until just now but it looks like
they max out at 100GB. So do you need to sit there changing discs
for the initial backup? I guess an incremental will fit on one disc
and then it's just a matter of how long you go before doing a full
checkpoint. I am currently backing up about 530GiB to a rented
Hetzner server
that costs me about €40/month so not cheap, but fairly convenient.
Would be much cheaper to build my own backup host out of cast off
hardware and only ongoing cost being the electricity, but I need it
outside my home for disaster recovery purposes.
I'm storing more than an order of magnitude more than that. It gets
really expensive.
I also periodically send a full copy to Amazon Glacier
for emergency
purposes. This costs about $3/month to keep there, but if I were to
ever need to restore it, it would cost about $50 one off to download
it all. So I hope to never have to do that.
Since the end of the HP Microserver cashback offer I wonder what the
next iteration of my home fileserver would look like. Right now it
is one of the Microservers plus an 8 bay eSATA disk shelf.
I think those disk shelves are too expensive for what should be
quite a dumb device. Maybe it would be better to just buy one tower
case to fit the drives into. The nice thing about the Microservers
though is that you have a known good level of engineering and
design, whereas some cheap thing you buy and put together yourself I
feel you'd risk encountering all manner of niggling problems.
My requirements for such a machine would be:
- amd64 architecture
- Not immensely power hungry. Doesn't have to be crippled either,
but a CPU that doesn't need active cooling would probably be good.
- At least 10 hot swap 3.5" drive bays connected by SATA, or 8 of
them if I could install a pair of internal flash devices to hold
the OS.
These latter two requirements seem to be impossible to achieve in
the current market, unless you want to go for external port
multipliers. It seems that nobody builds machines with small low-power
chips and lots of SATA ports.
You can get lots of SATA ports on a socketed motherboard, but then
it's really hard to get hold of a low-power CPU that will fit. Or you
can get a low-power CPU soldered onto a board with a couple of SATA
sockets. It's really irritating.
So if anyone has built something like that recently on
the cheap I'd
be interested to hear what you went for.
If the cashback offer was still there I'd be tempted to buy 2 or 3
Microservers and play with Ceph…
Ceph needs lots of IOPS, and when I adminned a Ceph setup a few
years ago, it sucked *massively* for repeated small file random
access. It's probably OK for streaming big files with sequential
reads.
We were running Ceph and CephFS on three fairly hefty Xeons, with
six HDDs in each, plus another data machine with six large SSDs, and
another two for the metadata servers, with SSDs in those. Oh, and I
think we had a master for the cluster, too. The machines had two
networks -- one GbE network going to the users, and a 40 Gb Infiniband
interconnect reserved for the cluster. That was something of a minimal
configuration in terms of the number of machines.
Total data holdings were something like 30 TB. The metadata servers
were pretty much always running CPU-hot. Using it for home directories
on desktop machines was basically unusable -- it would take 15 minutes
to log in on a desktop session, and anything up to half an hour to
open a Thunar window on someone's home directory. We ended up
migrating the home dirs to a NetApp and the big data store for HPC to
a GlusterFS. It took over a month to copy the 30 TB of data over a GbE
network.
Hugo.
--
Hugo Mills | I am but mad north-north-west: when the wind is
hugo@... carfax.org.uk | southerly, I know a hawk from a handsaw.
http://carfax.org.uk/ |
PGP: E2AB1DE4 | Hamlet, Prince of Denmark
1 Jun
1 Jun
3:26 a.m.
Hello,
On Fri, May 31, 2019 at 11:56:54PM +0000, Hugo Mills wrote:
Well, there's plenty of Intel Atom-based boards with 8 to 12 SATA3
ports, e.g.:
https://www.supermicro.com/products/motherboard/atom/A2SDi-8C-HLN4F.cfm
This is a 2 year old CPU though and after all the various
Spectre/Meltdown mitigations it may have its performance cut
significantly. I also don't know the cost - could be £400+ just for
the board/S-o-C.
AMD EPYC 3000-based boards have a better performance story but I
haven't seen one with more than 4 SATA ports.
There are some Xeon D-15xx boards that can have up to 20 SATA3:
https://www.supermicro.com/products/motherboard/Xeon3000/#1667
The FlexATX form factor has limited suppliers though, but apparently
Fractal Design do some. Here's a blog post about a system built with
one of these boards and a Fractal Design case that has 10 drive
bays:
https://www.bussink.ch/?p=1766
2016 though. And they still put a CPU fan on it. And the chassis
bays aren't hot swap. And yikes at the cost!
Perhaps it would be best to compromise on the noise front if
possible and just use a regular low end Xeon or AMD CPU with a CPU
fan, so that a non-specialist board could be used.
I found some more recent ideas here:
https://forums.servethehome.com/index.php?threads/looking-for-8-bay-hot-swa…
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
My
requirements for such a machine would be:
- amd64 architecture
- Not immensely power hungry. Doesn't have to be crippled either,
but a CPU that doesn't need active cooling would probably be good.
- At least 10 hot swap 3.5" drive bays connected by SATA, or 8 of
them if I could install a pair of internal flash devices to hold
the OS.
These latter two requirements seem to be impossible to achieve in
the current market, unless you want to go for external port
multipliers. It seems that nobody builds machines with small low-power
chips and lots of SATA ports. 
7:29 a.m.
On 2019-05-31 23:56+0000, Hugo Mills wrote:
I started out writing CD-Rs about 15 years ago,
migrated to DVD+R,
and then to BD-R. I'm considering LTO-6 as the next migration, despite
the poor economics w.r.t. BR-R at my kind of scale.
What is the data integrity on optical media over ten or more years? I
have some CDs from 1990's era (silver) and starting to go mouldy. Many
of my writable CDs have lost their film, so useless for long term data
retention.
LTO on the other hand should span multiple decades without issue. If I
remember correctly, the tape should be forward compatible with drives
(not sure if there's a maximum number of iterations here, but I'd hope
not).
You could regularly export ZFS mirrors if you work spinny disks out to
be lower cost than LTO.
Remember to off-site one half of the mirror.
--
Best regards,
Ed http://www.s5h.net/
7:44 a.m.
On Sat, Jun 01, 2019 at 08:29:02AM +0100, Ed wrote:
On 2019-05-31 23:56+0000, Hugo Mills wrote:
It depends on the quality of the media. I've seen some that didn't
last more than the time it took to put the new-burned disk into its
case. (Seriously -- they were practically see-through). I've gone 5-8
years on mine without obvious data corruption. I believe it's possible
to get media which are intended for long-term storage, and they're
reputed to go up to 100 years, but those are going to be spendy.
Personally, I'd refresh the media at least every 10 years or so for
"bog standard" (but reasonably reputable) disks. I used to use TDK,
but I haven't seen a TDK optical medium for a few years -- I think
they dropped out of the market. I'm now on Verbatims.
I started out writing CD-Rs about 15 years
ago, migrated to DVD+R,
and then to BD-R. I'm considering LTO-6 as the next migration, despite
the poor economics w.r.t. BR-R at my kind of scale.
What is the data integrity on optical media over ten or more years? I
have some CDs from 1990's era (silver) and starting to go mouldy. Many
of my writable CDs have lost their film, so useless for long term data
retention. LTO on the other hand should span multiple decades
without issue. If I
remember correctly, the tape should be forward compatible with drives
(not sure if there's a maximum number of iterations here, but I'd hope
not).
With a drive of generation N <= 7, you can use tapes of no earlier
than generation N-2 for reading, and N-1 for writing. For N=8 (the
latest), you can only use tapes no earlier than gen N-1, for reading
or writing.
You could regularly export ZFS mirrors if you work
spinny disks out to
be lower cost than LTO.
Remember to off-site one half of the mirror.
Yes, this is an important point.
Hugo.
--
Hugo Mills | I went to a fight once, and an ice hockey match
hugo@... carfax.org.uk | broke out.
http://carfax.org.uk/ |
PGP: E2AB1DE4 |
8:54 a.m.
On 2019-06-01 07:44+0000, Hugo Mills wrote:
It depends on the quality of the media. I've
seen some that didn't
last more than the time it took to put the new-burned disk into its
case. (Seriously -- they were practically see-through). I've gone 5-8
years on mine without obvious data corruption. I believe it's possible
to get media which are intended for long-term storage, and they're
reputed to go up to 100 years, but those are going to be spendy.
This is not something you want to find out about when you need to go to
your backup for something. Sadly my college work is now only available
on neurons.
With a drive of generation N <= 7, you can use
tapes of no earlier
than generation N-2 for reading, and N-1 for writing. For N=8 (the
latest), you can only use tapes no earlier than gen N-1, for reading
or writing.
This makes me sad. You need to keep a drive and working interface
around. Then again, if you're doing that then it isn't unreasonable to
refresh by SHA validating stream to new media, then don't throw the old
copy away.
--
Best regards,
Ed http://www.s5h.net/
1:22 p.m.
On 2019-05-31 21:49+0000, Andy Smith wrote:
I am currently backing up about 530GiB to a rented
Hetzner server
that costs me about €40/month so not cheap, but fairly convenient.
Would be much cheaper to build my own backup host out of cast off
hardware and only ongoing cost being the electricity, but I need it
outside my home for disaster recovery purposes.
I also periodically send a full copy to Amazon Glacier for emergency
purposes. This costs about $3/month to keep there, but if I were to
ever need to restore it, it would cost about $50 one off to download
it all. So I hope to never have to do that.
Not sure of the retention cost, but have you looked at Azure blob
storage and fuse mounts to do it? On the surface it looks like $20.80
plus some more for I/O. I've yet to work out correct IO costs with Azure
and wouldn't be my first choice.
First choice, as always is with a good chap with equipment in THDN, has
a good referral scheme and goes by the name 'grifferz' :)
--
Best regards,
Ed http://www.s5h.net/
12 Jun
12 Jun
1:37 p.m.
On 31/05/2019 19:07, Hugo Mills wrote:
I'd go for btrfs. (Disclaimer: I'm a
regular contributor to the
btrfs community, and I've written quite a bit of their documentation).
I've been running a RAID-1 btrfs data store for about 10 years. In
that time, I've hit one bug that required me to restore from
backup. That was 9 years ago, when the FS was about as stable as an MP
in a three-way marginal constituency.
I'm now up to 11TB of data on a 13TB RAID-1, and it's been fine for
years, even over power failures and several disk failures.
That sounds good, I'll get it set up and migrate the data across. :-)
The primary things you need to ensure is that your
hardware is
reliable (no USB storage, and disks and controllers which honour flush
instructions), <snip>
Thanks for the tips.
Scrub regularly, avoid compression and qgroups and
parity RAID for
metadata, and you should be fine.
Note that RAID-1 is *not* a backup. You still need backups.
<snip>
I'm very good about preaching about backups, not so great at actually
doing them myself. My longer-term intention is to set up an off-site
daily-updated backup server, probably at my parents rather than in the
cloud, but I don't have the money for that currently.
Thanks
Gavin
1773
days inactive
1785
days old
9 comments
4 participants
participants (4)
-
Andy Smith -
Ed -
Gavin Westwood -
Hugo Mills