Re: [bitfolk] Archival Storage

Hi Andy, On Mon, Oct 28, 2019 at 10:41:47PM +0000, Andy Bennett wrote: I hope you will get some responses from customers, but I'll add some of my experiences too, since of course I do use the archive storage in a few bits of BitFolk infrastructure. It is hard to compare. You're talking about pre 2015-Q4 so more than 4 years ago, and while those SATA disks back then are individually similar in performance to the SATA disks used for archive now, back then we typically had: - 8x SATA HDDs - Hardware RAID with battery/supercap backed write cache - 30–50 VMs using that for everything Whereas now for hosts with archive storage it's more like: - 2 or 4 SATA HDDs - Software RAID (no write cache) - 20–30 VMs using that but not as their main storage I can show you IOPS and throughput graphs but it is hard to tell from that how loaded the devices are. (You can isolate only the archive storage in these examples by clicking on "md6_Read" in the legend and then ctrl-clicking on "md6_Write" to select only those. md4 and md5 are SATA SSDs for comparison.) https://tools.bitfolk.com/grafana/dashboard/snapshot/eaiaSqz5sZ2awYsmxu67RV… https://tools.bitfolk.com/grafana/dashboard/snapshot/XOwe36vgYYznDXzz55OjQ2… What would be better for judging storage loading I think is something like 95th or 99th percentile IO latency, but it seems like that is very hard to collect long term in Prometheus. It can be done as a short test with something like fio. If there are other metrics that people feel are relevant then let me know and I'll see if I can get them. I'm guessing that the total data is not going to fit in 5GiB. If it was going to, well, 5GiB of SSD costs the same as 50GiB of archive so for me it would be a non-question and would just go with SSD to make life simpler. At a larger scale, this was one of the driving factors for going all-SSD at BitFolk - I was spending a lot of time tracking down IO starvation problems, constantly changing loads, customers affecting each other despite my best efforts. Yes, SSD is vastly more expensive but since late 2015 I don't have to do any of that any more. Even the SATA SSDs will do 50k+ IOPS so other resources run out first. So, moving on, with the assumption that you absolutely do require more than 5GiB of storage. My gut instinct is that an IMAP server is going to be too much random IO to work well with this kind of storage. I don't run any big IMAP servers so I don't know first hand, but that's just my guess. If it was something like a POP server I'd say, it's email: are users really going to be sitting waiting for it to complete? It's only got to keep up with the rate of mail delivery to not completely back up your system, right? But with IMAP I'm thinking you're going to have multiple mail clients connected, continually requesting this message or that or searching over headers and whatnot, so it's going to be all over the place and there are users sitting waiting for the mail client to update so they can read the individual mail. I'm honestly not sure I would risk it. A while back I did investigate two block device caching solutions, lvmcache and bcache: http://strugglers.net/~andy/blog/2017/07/19/bcache-and-lvmcache/ http://strugglers.net/~andy/blog/2017/08/03/a-slightly-more-realistic-look-… I liked lvmcache over bcache because it seemed a lot more mature and safer, although I understand that as it mandates LVM and has to have cache removed before you can snapshot, it is rather inflexible compared to bcache. These experiments were done on actual BitFolk hardware, though it's been more than 2 years now so I should really repeat them. The completion latency stats in the second article are of most interest and seem promising, but the setup does assume that 90% of requests are for 10% of the data. I'm not convinced that is very representative of an IMAP server. You could just try it though, as it's not too big of a deal to attach both SSD and HDD storage to your VM, let you experiment, and then detach the one you don't end up using… If you wanted to experiment with block device caching I suppose you could create a loopback device from a file on your SSD, for the fast device. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting