Hi, In addition to Lester's email below, I have received one email off-list expressing concern about the amount of Xen security advisories recently, and asking whether I was considering changing to a different hypervisor. I thought I'd respond on-list [also Bcc'd to the person who contacted me off-list]. I am of course not happy with the amount of serious Xen security flaws that have been uncovered recently. BitFolk turned 10 years old this year and in the first few years it was unusual to see one serious XSA per year. This latest set will have been the third set in three months. Most of the problems are being discovered by the Xen project themselves as part of their ongoing development efforts, or by developers at the handful of very large companies that use it, e.g. Amazon and Oracle. I do not believe that code quality has gone down recently; I think it is more the case that they are getting better at spotting bugs. Most of these bugs affect all versions of Xen, so have been present for years. When considering another hypervisor, what that effectively means is KVM. There are probably more companies using KVM and it is certainly a better known brand name in the virtualisation world, though I suspect in terms of number of bare metal hosts running it, AWS's use of Xen would put it ahead there. If I started BitFolk over again then KVM would probably be the first thing I would look at, but I'm still not entirely sure that I would go with it. Although Xen has seen a lot more security advisories than I would like, especially in the last 2 years, I do appreciate its security disclosure process. It's what enables BitFolk to be notified of security bugs at the same time that huge companies like Amazon and Oracle find out about them (unless they discovered them, obviously!), 2 weeks before the rest of the Internet. Another thing to consider is that Xen are disclosing every bug that could ever possibly have an effect, not just bugs that are exploitable in Linux. When comparing the situation against KVM it's hard because the KVM project doesn't have a security bug disclosure process at all. They don't send advisories. The only place they show up is in the changelog of the Linux kernel, and not all security issues make it there. I'm sure that any that are known to be exploitable in Linux do, of course. So basically, I really appreciate there being a comprehensive list of advisories for Xen¹ and BitFolk being included in a 2 week pre-disclosure, neither of which we would get with KVM. It's true though that I haven't had a look at KVM in a long time and haven't ever had a proper look at it. I will do that in the next few months just to get a better handle on things. As regards making Xen patching a less disruptive process, as you're probably aware I'm already pushing suspend-and-restore (almost all of BitFolk's infrastructure VMs do it). I am next going to put concerted effort into investigating live patching: https://wiki.xenproject.org/wiki/LivePatch That's basically similar to the various kernel live patching efforts, but for Xen (which is booted as a kernel, too). We're currently running 4.8.x and live patching was only a technology preview in 4.7. I still don't think it is quite ready yet and don't expect this to become really usable in production until I've upgraded all hosts to Debian stretch and moved to Xen 4.9. I would hope by then that most security bugs can be live patched so as to not require a reboot. I am sorry for the disruptions these security patchings are creating — it's certainly no fun for me either! Not only do I have to do the work but also the vast majority of my personal and professional hosting runs at BitFolk too. Cheers, Andy ¹ http://xenbits.xen.org/xsa/ On Fri, Sep 29, 2017 at 09:39:04PM +0100, Lester Hawksby wrote: > Blimey, Xen's giving you some right trouble at the moment! > > Thanks very much for all the clear explanation as to what's up. Much > appreciated. (Doesn't affect me at the moment as my VM is idle after I > changed my original plans and have been very short of time to start again, > but the clarity is still making me glad I chose you guys). > > Best > > Lester

On 29/09/17 21:39, Lester Hawksby wrote: Andy's transparency and honesty are why I have stopped scooting around the internet looking for a better deal. Don't tell him, but I would happily pay (a bit) more.