[bitfolk] Update on the progress of the RAM upgrade

Hi, I've been fielding a number of questions on this subject so I thought it was best to give an update. After the hardware upgrades earlier in the month it became apparent that not all machines were seeing all the RAM. In fact, it was looking pretty bleak: Host Previous RAM New RAM Recognised config. config. RAM -------------------------------------------------- corona 4x2G 4x2G 8G curacao 4x2G 2x4G, 4x2G 16G kahlua 2x4G, 4x2G 4x4G, 4x2G 16G kwak 4x2G, 4x1G 4x4G, 4x2G 16G obstler 6x2G 2x4G, 6x2G 16G That's 20G missing out of 40G purchased. At the time it was fairly obviously a software issue and in any case we did end up with more RAM than I started with, so I left the RAM installed and carried on with the work. Further research reveals that 32bit Xen is limited to 16G RAM even in PAE[1] mode. This was a surprise to me - my understanding of PAE was that it would allow up to 64G RAM to be used, and indeed that is how it works in Linux. But Xen is not Linux (the Xen hypervisor boots Linux, but it's not Linux). There is no possibility of making 32bit Xen see more than 16G RAM unless someone in the community writes that functionality. Why does Bitfolk even use 32bit software when all the servers are 64bit-capable? Well, it is actually only since islay was removed at the last datacentre visit that all hosts became 64bit-capable, but the main reason was that for a long time 32bit seemed the most stable and offered the most stable choice of guest operating systems. While all the supported Xen guests do have 64bit releases, not all of these were of the same quality as their 32bit counterparts. And running 32bit guests on a 64bit Xen host has only been possible relatively recently. Finally, there is no real performance gain going 64bit on a server as small as the typical VPS. It will actually be worse, as processes use more memory under 64bit. These decisions were due for a rethink soon though, and for the next set of servers I had already decided to go 64bit Debian Lenny or Ubuntu Hardy, while keeping 32bit guests. So, in order to use the full RAM that has been installed, a 64bit hypervisor is necessary. As I say, all the servers are 64bit-capable, and I've been advised that the hypervisor and Xen kernel from Debian Lenny are stable when running 32bit guests on 64bit hypervisor. My plan at the moment, therefore, is to go to that configuration on the new server that I installed at the last visit: faustino. Over the next couple of days I will be moving some of my own and Bitfolk infrastructure VPSes onto this server as a means of testing its stability. Provided that goes well, then next week I would like to step up the testing by putting customer VPSes onto it. I'm looking for a few volunteers for that. It will involve a shutdown, 10-15 minutes of downtime then being booted on the new server. As reward for volunteering I'm offering a week of free service and the promised 120M RAM upgrade when booted on faustino. Please drop an email to support(a)bitfolk.com if you'd like to volunteer for this. I want to be very careful about doing this fairly radical configuration change on existing stable servers each of which is holding 30-50 customers, but until I do it I can't give the RAM upgrade. Most of the existing servers are full in terms of IO capacity so I do not need to do this for any reason other than to give you the extra RAM. Therefore, my judgement call is on the side of caution and I am not going to change configuration for at least a month, in order to do sufficient testing on the new server. If you want the RAM upgrade before this then you are going to have to be moved to the new server. Provided I get enough volunteers and of course depending on things being stable, then probably towards the end of next week I will alter the web site to reflect the RAM upgrade and new customers will also go onto the new server. Finally, when the time comes that I'm satisfied that the configuration on faustino is stable, I will install the same software on all existing servers. On each server in turn I will: - Install the new hypervisor and kernel - Update VPS configurations to have the additional RAM - Shut down every VPS on that server - Reboot the server into the new hypervisor/kernel and confirm it sees the full amount of RAM - Start up every VPS on that server I will most likely do that one server per night at 1AM GMT or something. So, that's the plan for all this. I'm sorry that it hasn't been as simple as originally planned, but the PAE thing was rather unexpected. I hope that by including volunteers in the testing process and offering the RAM upgrade with it that it will be a suitable compromise for both adventurous people who need the extra RAM, and cautious people who prefer to stay with the stability known prior to this month. Not that I would even be contemplating this if I thought it wasn't going to be stable, but there is always risk! If you have any further questions, feel free to ask on- or off-list. Cheers, Andy [1] http://en.wikipedia.org/wiki/Physical_Address_Extension -- http://bitfolk.com/ -- No-nonsense VPS hosting Encrypted mail welcome - keyid 0x604DE5DB