[bitfolk] Interesting (?) Linux RAID-10 performance caveat

Hello, A new BitFolk server that I will put into service soon has 1x SSD and 1x NVMe instead of 2x SSD. I tried this because the NVMe, despite being vastly more performant than the SATA SSD, is actually a fair bit cheaper. On the downside it only has a 3 year warranty (vs 5) and 26% of the write endurance (5466TBW vs 21024TBW)¹. So anyway, a pair of very imbalanced devices. I decided to take some time to play around with RAID configurations to see how Linux MD handled that. The results surprised me, and I still have many open questions. As a background, for a long time it's generally been advised that Linux RAID-10 gives the highest random IO performance. This is because it can stripe read IO across multiple devices, whereas with RAID-1, a single process will do IO to a single device. Linux's non-standard implementation of the RAID-10 algorithm can also generalise to any amount of devices: conventional RAID-10 requires an even number of devices with a minimum of 4, but Linux RAID-10 can work with 2 or even an odd number. More info about that: https://en.wikipedia.org/wiki/Non-standard_RAID_levels#Linux_MD_RAID_10 As a result I have rarely felt the need to use RAID-1 for 10+ years. But, I ran these benchmarks and what I found is that RAID-1 is THREE TIMES FASTER than RAID-10 on a random read workload with these imbalanced devices. Here is a full write up: http://strugglers.net/~andy/blog/2019/05/29/linux-raid-10-may-not-always-be… I can see and replicate the results, and I can tell that it's because RAID-1 is able to direct the vast majority of reads to the NVMe, but I don't know why that is or if it is by design. I also have some other open questions, for example one of my tests against HDD is clearly wrong as it achieves 256 IOPS, which is impossible for a 5,400RPM rotational drive. So if you have any comments, explanations, ideas how my testing methodology might be wrong, I would be interested in hearing. Cheers, Andy ¹ I do however monitor the write capacity of BitFolk's SSDs and they all show 100+ years of expected life, so I am not really bothered if that drops to 25 years. -- https://bitfolk.com/ -- No-nonsense VPS hosting