There's some major and unavoidable upcoming maintenance work being undertaken by our colo provider. The effect on BitFolk is that 7 of our servers need to be physically removed from one rack and moved to a different rack in a different room in Telehouse. This is part of the colo provider's need to rebuild the entire rack, so it's not in our control, is unavoidable and we have limited say over the schedule as it also affects all their other customers in that rack as well. It's a big piece of work, though not complicated (for us). I'm writing to you because I'm not sure of the best way to notify customers about this work and would like you to give me some feedback on how you'd prefer the communication to work. We haven't yet agreed firm dates/times but it's going to be no sooner than a month from now and probably no later than six weeks from now. We're going to be upgrading the servers to 10GE networking so we're going to move one of them first, wait a week to be sure the hardware is stable in that configuration and then do the remaining six the following week. The first one may happen evening UK time, like 9pm or something, but the rest will likely happen a bit later, around midnight into early hours. So if affected, assume you're in that latter group, and expect half an hour or so of being powered off. An additional complication is that this comes while we are right in the middle of doing rolling upgrades of our fleet of servers. That work was started before I was made aware that the server move would be necessary, otherwise I might have postponed it. Some of you will have already been through that rolling upgrade process or be going through it now. As that's completely within our control we've been moving customers between servers one by one, at times convenient to you and individual to you, and then upgrading the server once it's empty. The consequence of that is, we don't know which customers will be on which servers come the date of the move. We could send out personalised notices as soon as we know the date, but some of those people won't be on the affected servers when the time comes, and some people who didn't get the notice WILL be affected when the time comes. As part of the rolling upgrades I'm trying to avoid moving anyone from a server that's not going to be moved onto one that will be moved. It is however unavoidable that some people will be moved between two servers that are going to be moved, so some will get one very short outage for the move of their VM and then the later longer outage when the whole server is relocated. So is there any value in sending out personalised advanced notice of maintenance more than a month ahead? Would it be better just to send notice to the announce@ address giving the list of affected servers and the time it's going to happen and then a refresher a week ahead and again a day ahead or something? If the prospect of being powered off for half an hour or so a month from now is not acceptable to anyone, then we can most likely move their VM to another server ahead of time - one that we know won't be involved in the move. By semi-live migration if necessary. That's the only per-customer thing we can do and it's extra work so we will only do it if people ask for it. There isn't enough spare capacity to do it for everyone so doing migration as default for everyone is not an option. But happy to do it on a case by case basis. Your thoughts? I just want to reiterate that these servers moves—part of larger work involving our colo provider and their other customers—cannot be negotiated individually with the several hundred BitFolk customers that will be affected. I'm asking you only about broad arrangements for notice that can be applied to the whole customer base. Once the date/time is decided the only per-customer action that can take place is whether you require us to do extra work to migrate your VM ahead of time, and you don't need to tell me that now as there will be an announcement when the date is known (soon, possibly even today). Cheers, Andy

So this is an example of the difficulties here. 😀 I'm not picking on you specifically because a lot of people have said and will say the same, but I need to work out in each case what it actually means.

So does this mean that you think we SHOULD send out a personalised notification to each customer on all the affected servers, even though they may not be affected at the actual time because they got moved before it happened? Your later thing about being told if no longer affected being the way to handle that?

The other option I offered was to send out a notice to announce@ just listing the servers that would be affected. That leaves people to work out themselves if their service would be affected. Do you mean that this option is less preferred than the personalised notifications that may later not be relevant?

Do you mention this in the context of the rolling upgrades? In case I wasn't clear, this maintenance work involving moving hardware isn't going to move any customer VMs¹ except for people who can't take the downtime and ask to be moved ahead of time.

You can just look up what host your VM is on any time² but if you're saying that you think that between now and the rerack people should just be told if their status with regard to the rerack changes, then yes we can do that.

¹ If you're reading this, you are subscribed, because anything posted to announce@ goes here too.

Hi Andy, On Thu, Mar 18, 2021 at 01:40:04PM +0000, Andy Bennett wrote: Thanks, this is helpful - it's certain there would be things that needed clarifying, but not easy without knowing what! existing Indeed no, we're literally just going to be: - turning some servers off - removing them from the rack - opening them to install 10GE NICs - closing them again - installing them in another rack in another room - switching them on again …at which point we expect all the VMs on them to boot up as they were before with no visible change from the customer's point of view apart from having been shutdown for however long it took to do that. This would be relatively simple (if unwelcome) if it wasn't for the issue of the rolling upgrades that are ongoing and causing customer VMs to move about. Okay. So at the moment I'm leaning towards sending out a non-personalised public email to announce@ a month before the work, with all the details, which will include a list of host machines. That will also include the offer of moving customer VMs ahead of the work if necessary. I do have some concerns that there may be customers who aren't subscribed to announce@, even though they are supposed to be¹, so I think we will still send out a direct email to affected customers a week before the work. If it's a surprise to anyone at that point it will be unwelcome but at least they still have time to prepare for it, and we do have the justification that they should have been on the announce list. Cheers, Andy ¹ If you're reading this, you are subscribed, because anything posted to announce@ goes here too. -- https://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ users mailing list users(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/users

Hi, There's some major and unavoidable upcoming maintenance work being undertaken by our colo provider. The effect on BitFolk is that 7 of our servers need to be physically removed from one rack and moved to a different rack in a different room in Telehouse. This is part of the colo provider's need to rebuild the entire rack, so it's not in our control, is unavoidable and we have limited say over the schedule as it also affects all their other customers in that rack as well. It's a big piece of work, though not complicated (for us). I'm writing to you because I'm not sure of the best way to notify customers about this work and would like you to give me some feedback on how you'd prefer the communication to work. We haven't yet agreed firm dates/times but it's going to be no sooner than a month from now and probably no later than six weeks from now. We're going to be upgrading the servers to 10GE networking so we're going to move one of them first, wait a week to be sure the hardware is stable in that configuration and then do the remaining six the following week. The first one may happen evening UK time, like 9pm or something, but the rest will likely happen a bit later, around midnight into early hours. So if affected, assume you're in that latter group, and expect half an hour or so of being powered off. An additional complication is that this comes while we are right in the middle of doing rolling upgrades of our fleet of servers. That work was started before I was made aware that the server move would be necessary, otherwise I might have postponed it. Some of you will have already been through that rolling upgrade process or be going through it now. As that's completely within our control we've been moving customers between servers one by one, at times convenient to you and individual to you, and then upgrading the server once it's empty. The consequence of that is, we don't know which customers will be on which servers come the date of the move. We could send out personalised notices as soon as we know the date, but some of those people won't be on the affected servers when the time comes, and some people who didn't get the notice WILL be affected when the time comes. As part of the rolling upgrades I'm trying to avoid moving anyone from a server that's not going to be moved onto one that will be moved. It is however unavoidable that some people will be moved between two servers that are going to be moved, so some will get one very short outage for the move of their VM and then the later longer outage when the whole server is relocated. So is there any value in sending out personalised advanced notice of maintenance more than a month ahead? Would it be better just to send notice to the announce@ address giving the list of affected servers and the time it's going to happen and then a refresher a week ahead and again a day ahead or something? If the prospect of being powered off for half an hour or so a month from now is not acceptable to anyone, then we can most likely move their VM to another server ahead of time - one that we know won't be involved in the move. By semi-live migration if necessary. That's the only per-customer thing we can do and it's extra work so we will only do it if people ask for it. There isn't enough spare capacity to do it for everyone so doing migration as default for everyone is not an option. But happy to do it on a case by case basis. Your thoughts? I just want to reiterate that these servers moves—part of larger work involving our colo provider and their other customers—cannot be negotiated individually with the several hundred BitFolk customers that will be affected. I'm asking you only about broad arrangements for notice that can be applied to the whole customer base. Once the date/time is decided the only per-customer action that can take place is whether you require us to do extra work to migrate your VM ahead of time, and you don't need to tell me that now as there will be an announcement when the date is known (soon, possibly even today). Cheers, Andy

Nothing I run will worry about a half hour of downtime but I would want to know exactly when it is so I can plan.

rolling upgrades may move it between physical kit at some point before then.