29 Oct
2011
29 Oct
'11
9:43 a.m.
Hello,
We're now very close to the point when we need to begin getting you
all to renumber your IP addresses. We are not there yet, but here is
some more info:
https://tools.bitfolk.com/wiki/Renumbering_out_of_212.13.194.0/23
Before finalising the procedure and sending out instructions I could
use some feedback from you on a couple of points. These involve
situations where BitFolk is contacting you by IP address.
- BitFolk Backups
Our backup servers are doing rsync-over-SSH by IP address.
If we change the IP addresses our side then your backups stop
working until you make your change.
If we don't change the IP addresses our side then your backups
will break when you make your change, and then you would need to
contact support to get the change done.
Which would be preferred?
- Nagios checks
Many of you have Nagios checks. These are done by IP address.
If we immediately make the change on our side then alerts will
start to fire for you, and will not right themselves until you
complete the IP address change on your side.
If we don't make the change on our side then as soon as you make
your change, alerts will start to fire and then you'd have to
contact support to ask for the monitoring to be fixed.
I have a strong preference for us making the change as soon as you
have the info to make the change your side, so that the monitoring
fixes itself as you do the work.
Any other ideas how to do it?
- Old rsync-style zone transfers
Some of you still have DNS secondary services driven by zone files
that we are rsyncing from you. This is happening by IP address.
This service was deprecated years ago, since everyone got enough
RAM to run a proper DNS server for this purpose.
I would like to finally retire this service. How much time would
people like to stop relying on it and switch to running a real DNS
server?
- DNS slaves
Those of you who have secondary DNS services will be running your
own DNS server on your VPS and we'll be doing AXFR to that IP
address.
The DNS secondary service comes with Nagios monitoring, so as soon
as we switch the IP configured on our side then monitoring as
mentioned above will begin to alert.
There can be multiple masters, so what we could do is set both old
and new IP addresses so that zone transfers can continue to take
place, but our monitoring can only have one IP address.
With that in mind I think I would prefer to set both old and new
IPs as masters, let monitoring alert for your TCP/53 and that will
fix itself as your sort out your new IP address.
Any other ideas how to do it?
As far as possible in all cases above I would like to avoid the
situation where we have to chase individual customers one-on-one
about things that suddenly stopped working.
I can imagine this might not be avoidable—particularly in the case
of the backups—so if we have to do it, we have to do it.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
29 Oct
29 Oct
10:07 a.m.
The backup scenarios described below seem to be the main ones that affect me
directly (apart from the obvious ones like changing firewalls, DNS records,
SSH logins etc) It seems to me that the obvious answer seems to be you
should make changes first
Keith
On 29 October 2011 10:43, Andy Smith <andy(a)bitfolk.com> wrote:
Hello,
We're now very close to the point when we need to begin getting you
all to renumber your IP addresses. We are not there yet, but here is
some more info:
https://tools.bitfolk.com/wiki/Renumbering_out_of_212.13.194.0/23
Before finalising the procedure and sending out instructions I could
use some feedback from you on a couple of points. These involve
situations where BitFolk is contacting you by IP address.
- BitFolk Backups
Our backup servers are doing rsync-over-SSH by IP address.
If we change the IP addresses our side then your backups stop
working until you make your change.
If we don't change the IP addresses our side then your backups
will break when you make your change, and then you would need to
contact support to get the change done.
Which would be preferred?
- Nagios checks
Many of you have Nagios checks. These are done by IP address.
If we immediately make the change on our side then alerts will
start to fire for you, and will not right themselves until you
complete the IP address change on your side.
If we don't make the change on our side then as soon as you make
your change, alerts will start to fire and then you'd have to
contact support to ask for the monitoring to be fixed.
I have a strong preference for us making the change as soon as you
have the info to make the change your side, so that the monitoring
fixes itself as you do the work.
Any other ideas how to do it?
- Old rsync-style zone transfers
Some of you still have DNS secondary services driven by zone files
that we are rsyncing from you. This is happening by IP address.
This service was deprecated years ago, since everyone got enough
RAM to run a proper DNS server for this purpose.
I would like to finally retire this service. How much time would
people like to stop relying on it and switch to running a real DNS
server?
- DNS slaves
Those of you who have secondary DNS services will be running your
own DNS server on your VPS and we'll be doing AXFR to that IP
address.
The DNS secondary service comes with Nagios monitoring, so as soon
as we switch the IP configured on our side then monitoring as
mentioned above will begin to alert.
There can be multiple masters, so what we could do is set both old
and new IP addresses so that zone transfers can continue to take
place, but our monitoring can only have one IP address.
With that in mind I think I would prefer to set both old and new
IPs as masters, let monitoring alert for your TCP/53 and that will
fix itself as your sort out your new IP address.
Any other ideas how to do it?
As far as possible in all cases above I would like to avoid the
situation where we have to chase individual customers one-on-one
about things that suddenly stopped working.
I can imagine this might not be avoidable--particularly in the case
of the backups--so if we have to do it, we have to do it.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
--
Keith Williams
If the only tool you have is a hammer, you tend to see every problem as a
nail.
- Abraham Maslow
ทำดีได้ดี ทำชั่วไดชั่ว
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
iEYEAREDAAYFAk6ryrYACgkQIJm2TL8VSQv2+ACgmHPWSk7fzBZJ8MEP0kFzigfX
m3UAnij1d8w1e+qbuVYQclCuWST6nQVS
=p30y
-----END PGP SIGNATURE-----
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
11:18 a.m.
Hi Andy,
Could VMs run with both sets of IPs?
ie:
Step 1: give new ips to customers
Step 2: customer adds new ip(s) as a extra virtual interfaces
Step 3: customer checks service works okay on new ips, updates their
dns, requests bitfolk backup/nagios change to new ips
Step 4: sometime later customer remove the old ips, and hopefully
nothing breaks (but if it does they can temporarily add old ips back
whilst they fix it)
Step 5: sometime later bitfolk remove routing to the old ips
Cheers,
Joseph
11:39 a.m.
Hi Joseph,
On Sat, Oct 29, 2011 at 12:18:29PM +0100, Joseph Heenan wrote:
Could VMs run with both sets of IPs?
Yes, this is the intention. There will be a period of several months
where both sets of IP addresses will work.
Step 3: customer checks service works okay on new ips,
updates their
dns, requests bitfolk backup/nagios change to new ips
Well, this would involve opening a support ticket with every
customer (or them with us) and I'm trying to avoid that where
possible.
Perhaps since backups are so important, we can leave them running on
the customer's old IP addresses. When they eventually remove those
IPs the backups will begin to fail and we can fix them
individually.
For everything else I am inclined to let monitoring send out alerts
and see them fixed as people bring up the new IPs..
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
"I'd be happy to buy all variations of sex to ensure I got what I wanted."
-- Gary Coates (talking about cabling)
12:13 p.m.
On 29/10/2011 12:39, Andy Smith wrote:
Perhaps since backups are so important, we can leave
them running on
the customer's old IP addresses. When they eventually remove those IPs
the backups will begin to fail and we can fix them individually.
Would it be possible to have a cron job automatically migrate the backup
IPs in some fashion? One way it could work is have a weekly job which
checks if customers are using their new IP address: the first time it
gets a positive hit for a customer it then emails/alerts them to let
them know their backups will migrate in (say) two week's time, and
providing it gets positive hits on sebsequent runs it then does the
migration at the specified time. If a customer needs more time they
could raise a ticket to opt-out of the automatic migration (or opt back
in at some later point, perhaps). Thoughts?
--
Phil Stewart
1:17 p.m.
Hi Phil,
On Sat, Oct 29, 2011 at 01:13:34PM +0100, Phil Stewart wrote:
Would it be possible to have a cron job automatically
migrate the backup
IPs in some fashion? One way it could work is have a weekly job which
checks if customers are using their new IP address: the first time it
gets a positive hit for a customer it then emails/alerts them to let them
know their backups will migrate in (say) two week's time, and providing it
gets positive hits on sebsequent runs it then does the migration at the
specified time. If a customer needs more time they could raise a ticket to
opt-out of the automatic migration (or opt back in at some later point,
perhaps). Thoughts?
That sounds a lot more complicated than just waiting until a backup
run fails due to being unable to contact the host and then retrying
it again with their new IP address to see if that works.
My proposal was to continue using old IP addresses for backups and
only take an interest when told to or when a backup stops working. I
think you have hit upon a good idea about checking whether someone
has enabled their new IP addresses, and we can do this without
customer interaction, so it should be possible to pro-actively
reconfigure people's backups with fairly high confidence that they
will work.
I am less convinced with the bit about sending automated emails with
two week deadlines, etc. etc. I tend to think that most of you will
just want this to happen without the added complication and without
the interaction.
So..
Worst case: The deadline for renumbering is reached, all old IP
addresses cease being routed, a bunch of people's VPSes stop
working because they have somehow managed to miss all notifications
of this taking place.
Fix for that: They'll need to do the renumbering that they'll have
been nagged about for months, and then their VPS will work again and
their backups too. Anyone affected by this will have already missed
many emails and be caught completely unaware, so no other scheme of
email notifications would have saved them.
Common case: Someone with backups completes their renumbering and
disables their old IP address(es) before we notice that they have
renumbered. Their backups start failing.
Fix for that: We will notice within a day that their backups stopped
working, and will just try the backups again with the new IP
address. Only if they do not work with neither old nor new addresses
would we have cause to open a support ticket, and that would be some
weird failure mode.
People who care about their backups: Can just enable their new IP
addresses and leave their old IP addresses going for a short while,
long enough for us to notice and reconfigure.
People who *really* care about not missing a backup run: Can simply
open a support ticket *after* they have enabled their new IP
address(es) but *before* they have disabled their old ones. Thus not
missing a single backup run.
Now, when I send out the notification and instructions for
renumbering your VPSes I will be sending them direct to the email
address on record for everyone, as well as a copy here, just to do
our best to see that everyone gets it.
If I included a note about all this for those with backups, even
assuming that it was TL;DR and no one read it, I think we could get
pretty good coverage and reconfiguration without much need for
interaction and without many people missing more than a day of their
backups.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
1:10 p.m.
On 29/10/2011 12:39, Andy Smith wrote:
I see - perhaps a slight modification:
time t: new ips become available
time t+4 weeks: customers told in advance that backups & nagios will
change to new IPs on this date, and they should ensure they have both
IPs running if they don't want failures.
Joseph
Hi Joseph,
On Sat, Oct 29, 2011 at 12:18:29PM +0100, Joseph Heenan wrote:
Excellent.
Could VMs run with both sets of IPs?
Yes,
this is the intention. There will be a period of several months
where both sets of IP addresses will work. Step 3:
customer checks service works okay on new ips, updates their
dns, requests bitfolk backup/nagios change to new ips
Well, this would involve
opening a support ticket with every
customer (or them with us) and I'm trying to avoid that where
possible.
Perhaps since backups are so important, we can leave them running on
the customer's old IP addresses. When they eventually remove those
IPs the backups will begin to fail and we can fix them
individually.
1:41 p.m.
Hi Joseph,
On Sat, Oct 29, 2011 at 02:10:26PM +0100, Joseph Heenan wrote:
time t: new ips become available
time t+4 weeks: customers told in advance that backups & nagios will
change to new IPs on this date, and they should ensure they have both IPs
running if they don't want failures.
That is a good idea. I think you're right that there is some value
in pre-announcing the date at which the monitoring will flip to only
using people's new IPs, and one month sounds like a decent time.
So something like:
time t:
Renumbering is announced, plus instructions.
DNS secondaries set to check both old and new IPs as masters.
time t+(at least 1 month):
Nagios monitoring switches to using new IPs only. Lots of alerts
start being sent out, but renumbering your VPS makes them go
away.
It is also possible for customers to log in to Nagios web
interface and "acknowledge" an alert, which makes it stop firing
until the state changes, e.g. until you fix it. Instructions
will be provided.
Those running a DNS server on their VPS and having BitFolk do
secondary DNS would now start getting alerts for tcp/53,
although it will actually still work due to having both sets of
IP addresses as masters.
time t+(at least 3 months):
Old IP addresses are no longer routed.
Regarding backups:
Throughout this period we would be checking who has enabled
their new IPs, and if they have backups configured we would be
reconfiguring them to work with the new IPs.
As long as there's a period of at least a few days between
enabling new IPs and disabling old ones, there should be no
reason why people's backups would fail. Any that do start to
fail will be picked up by us and we'll try the obvious thing of
trying with new IP address, before opening a support ticket with
you.
Finally, anyone who is particularly concerned about their
backups should open a support ticket to get them reconfigured,
after they've enabled their new IPs but before they've disabled
the old ones.
Slightly woolly timings as I can't promise to be able to stick to a
specific date - "no earlier than" timings are usually okay.
This plan seems to avoid most likely cases of having to have actual
one-to-one conversations about it. :)
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
11:01 p.m.
As a (relative) newbie, am I already on the new IP address range?
<snip>
Otherwise it seems that my initial thought - that the bitfolk systems
could check that the new IP address is up and running, and silently
switch over - seems to have been covered.
11:17 p.m.
Hi Dom,
On Sun, Oct 30, 2011 at 01:01:56AM +0200, Dom Latter wrote:
As a (relative) newbie, am I already on the new IP
address range?
I'm afraid not; no customer is yet.
https://tools.bitfolk.com/wiki/Renumbering_out_of_212.13.194.0/23#Phase_II:…
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
11:27 p.m.
On 30/10/11 01:17, Andy Smith wrote:
Hi Dom,
On Sun, Oct 30, 2011 at 01:01:56AM +0200, Dom Latter wrote:
Ah. Well, I'll just have to tell my customers that 85.x.y.z is
a *much* classier neighbourhood than 212.x.y.z.
(We have control of DNS for most of them, but one or two manage
their own, for reasons that escape me).
As a (relative) newbie, am I already on the new
IP address range?
I'm afraid not; no customer is yet. 
30 Oct
30 Oct
12:37 p.m.
On 30/10/11 00:27, Dom Latter wrote:
On 30/10/11 01:17, Andy Smith wrote:
Well, obviously it is. It can trace its aristocratic ancestors back to
class A networks, rather than an oiky class C.
John
Hi Dom,
On Sun, Oct 30, 2011 at 01:01:56AM +0200, Dom Latter wrote:
Ah. Well, I'll just have to tell my customers that 85.x.y.z is
a *much* classier neighbourhood than 212.x.y.z. As a (relative) newbie, am I already on the new
IP address range?
I'm afraid not; no customer is yet.
4801
days inactive
4802
days old
11 comments
6 participants
participants (6)
-
Andy Smith -
Dom Latter -
John Winters -
Joseph Heenan -
Keith Williams -
Phil Stewart