30 May
2019
30 May
'19
2:11 a.m.
Hello,
A new BitFolk server that I will put into service soon has 1x SSD
and 1x NVMe instead of 2x SSD. I tried this because the NVMe,
despite being vastly more performant than the SATA SSD, is actually
a fair bit cheaper. On the downside it only has a 3 year warranty
(vs 5) and 26% of the write endurance (5466TBW vs 21024TBW)¹.
So anyway, a pair of very imbalanced devices. I decided to take some
time to play around with RAID configurations to see how Linux MD
handled that. The results surprised me, and I still have many open
questions.
As a background, for a long time it's generally been advised that
Linux RAID-10 gives the highest random IO performance. This is
because it can stripe read IO across multiple devices, whereas with
RAID-1, a single process will do IO to a single device.
Linux's non-standard implementation of the RAID-10 algorithm can
also generalise to any amount of devices: conventional RAID-10
requires an even number of devices with a minimum of 4, but Linux
RAID-10 can work with 2 or even an odd number.
More info about that:
https://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10
As a result I have rarely felt the need to use RAID-1 for 10+ years.
But, I ran these benchmarks and what I found is that RAID-1 is THREE
TIMES FASTER than RAID-10 on a random read workload with these
imbalanced devices.
Here is a full write up:
http://strugglers.net/~andy/blog/2019/05/29/linux-raid-10-may-not-always-be…
I can see and replicate the results, and I can tell that it's
because RAID-1 is able to direct the vast majority of reads to the
NVMe, but I don't know why that is or if it is by design.
I also have some other open questions, for example one of my tests
against HDD is clearly wrong as it achieves 256 IOPS, which is
impossible for a 5,400RPM rotational drive.
So if you have any comments, explanations, ideas how my testing
methodology might be wrong, I would be interested in hearing.
Cheers,
Andy
¹ I do however monitor the write capacity of BitFolk's SSDs and they
all show 100+ years of expected life, so I am not really bothered
if that drops to 25 years.
--
https://bitfolk.com/ -- No-nonsense VPS hosting
30 May
30 May
10:22 p.m.
Without doing it myself, if I have understood what you have done
correctly, I would guess it is the imbalance and software raid that is
doing it.
With raid 1 (mirror set, I think) being done in software the total write
time form the writees point of view is the time taken to write the
blocks to the first device (primary) in the mirror pair) the software is
then mirroring this to the second and the whole thing is decoupled a touch.
Where as once you start to stripe, the total write time from the writees
point of view is the time taken to write serially across both devices
not just one.
So in the first case you get the speed of the primary per write, where
as in the second it is the speed of serially writing to both devices.
Of course this is a generalization and ignores that fact that under
uni*s/linuxs of all types the writes are buffered by the OS. But if
doing metrics how you do and take your measures will be affected by this.
Striping can be faster but only where the writes/reads are queued
optimally across disks with synchronized spindles. Software raid across
UN-synchronized disks will never achieve the same performance. This is
the province of hardware raid and clever drive electronics.
There again I may be completely wrong, I suspect there may be factors at
play I have not accounted for in my idealized view.
Cheers
Kirbs
On 30/05/2019 03:11, Andy Smith wrote:
Hello,
A new BitFolk server that I will put into service soon has 1x SSD
and 1x NVMe instead of 2x SSD. I tried this because the NVMe,
despite being vastly more performant than the SATA SSD, is actually
a fair bit cheaper. On the downside it only has a 3 year warranty
(vs 5) and 26% of the write endurance (5466TBW vs 21024TBW)¹.
So anyway, a pair of very imbalanced devices. I decided to take some
time to play around with RAID configurations to see how Linux MD
handled that. The results surprised me, and I still have many open
questions.
As a background, for a long time it's generally been advised that
Linux RAID-10 gives the highest random IO performance. This is
because it can stripe read IO across multiple devices, whereas with
RAID-1, a single process will do IO to a single device.
Linux's non-standard implementation of the RAID-10 algorithm can
also generalise to any amount of devices: conventional RAID-10
requires an even number of devices with a minimum of 4, but Linux
RAID-10 can work with 2 or even an odd number.
More info about that:
https://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10
As a result I have rarely felt the need to use RAID-1 for 10+ years.
But, I ran these benchmarks and what I found is that RAID-1 is THREE
TIMES FASTER than RAID-10 on a random read workload with these
imbalanced devices.
Here is a full write up:
http://strugglers.net/~andy/blog/2019/05/29/linux-raid-10-may-not-always-be…
I can see and replicate the results, and I can tell that it's
because RAID-1 is able to direct the vast majority of reads to the
NVMe, but I don't know why that is or if it is by design.
I also have some other open questions, for example one of my tests
against HDD is clearly wrong as it achieves 256 IOPS, which is
impossible for a 5,400RPM rotational drive.
So if you have any comments, explanations, ideas how my testing
methodology might be wrong, I would be interested in hearing.
Cheers,
Andy
¹ I do however monitor the write capacity of BitFolk's SSDs and they
all show 100+ years of expected life, so I am not really bothered
if that drops to 25 years.
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
--
admins(a)sheffieldhackspace.org.uk
www.sheffieldhackspace.org.uk
11:47 p.m.
Hello,
On Thu, May 30, 2019 at 11:22:01PM +0100, admins wrote:
Without doing it myself, if I have understood what you
have done
correctly, I would guess it is the imbalance and software raid that is
doing it.
Clearly it is the imbalance (of device performances) because it
won't happen if they both perform the same. The question is why can't
RAID-10 cope with the unbalanced devices? Is it by design? Is it
generally known? Can it be fixed? etc.
With raid 1 (mirror set, I think) being done in
software the total write
time form the writees point of view…
The anomalous result is for random reads. There wasn't an observable
difference between RAID-1 and RAID-10 for writes.
The issue is that the RAID-1 driver is able to parallelise reads
to any member device, but the RAID-10 driver seems to just issue a
fraction to each device.
The best answer so far seems to be just that the RAID-10 driver
wasn't designed to handle devices with different performance
characteristics. A different OS's driver or a hardware RAID
implementation might work (and perform) differently.
I don't know whether it is possible to fix or if anyone is
interested in doing so. Probably RAID in all its forms is on its way
out in favour of zfs, btrfs, ceph, stratis and other forms of
software-defined storage.
Purists would probably be telling me to centralise storage with
something like ceph, using cache tiering to make best use of the
NVMe devices. That is a lot of infrastructure however. Would love to
try it on someone else's dime!
Striping can be faster but only where the writes/reads
are queued
optimally across disks with synchronized spindles. Software raid across
UN-synchronized disks will never achieve the same performance.
I'm unconvinced that this is an issue because if you look at how
many IOs went to each device, it's much closer to 50/50 in the
RAID-10 case, so the driver is just blindly splitting the IOs by the
number of devices present and would be doing that regardless of what
the devices actually were.
This is the province of hardware raid and clever drive
electronics.
Linux software RAID generally outperforms all hardware RAID except
for the case of writes in a situation where the RAID controller has
a big persistent write cache. As pure HBAs (i.e. no RAID
functionality) with persistent write cache are not generally
available, this setup has to be replicated by using a write journal
on a smaller flash device. But as all my devices in this case are
flash, that isn't a factor.
Hardware RAID is so 10 years ago! :)
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
31 May
31 May
2:36 a.m.
On Thu, May 30, 2019 at 11:47:01PM +0000, Andy Smith wrote:
Clearly it is the imbalance (of device performances)
because it
won't happen if they both perform the same. The question is why can't
RAID-10 cope with the unbalanced devices? Is it by design? Is it
generally known? Can it be fixed? etc.
Well here's some of the answer!
https://marc.info/?l=linux-raid&m=155926702609282&w=2
This allows RAID-1 to pick most idle device to do a read from:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/d…
So that explains how Linux RAID-1 is able to prefer fastest device
to read from. Same improvement was never done for Linux RAID-10.
Don't know if it could be.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
2064
days inactive
2065
days old
3 comments
2 participants
participants (2)
-
admins -
Andy Smith