--27KoNqt0fmcl1zj/
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Hi Joseph,
On Tue, Jun 26, 2012 at 11:14:43AM +0100, Joseph Heenan wrote:
> here's the graph for my > VPS:
>=20
> http://f8lure.mouselike.org/archived_graphs/button.heenan.me.uk_day25.png
>=20
> The number of huge spikes (and some packet loss, shown red) on this
> surprised me. Would this kind of result be expected?
About a month ago I was made aware of a problem with occasional
spikes of high latency, and on looking into it, it became apparent
that it had actually been the case for a long time - perhaps years -
without anyone really noticing.
What you're seeing is one or two packets out of every couple of
hundred being delayed somewhere, sometimes for hundreds of
milliseconds.
It isn't restricted to your VPS, or to any one BitFolk server. It
seems to be affecting all VPSes, but as I say, it has been doing so
for a very long time. Here's a graph that exemplifies the issue:
http://www.thinkbroadband.com/ping/share/9b7cf0ba2197b53c0aeb0f3cff42fb7e.h=
tml
Since then I've been trying to work out where it's actually
happening, and this has been a long and ongoing process.
Firstly, it *is* restricted to BitFolk. Other things hosted at the
same colo are not seeing it. Here's something else in the same rack
as some BitFolk nodes, connected to the same switches:
http://www.thinkbroadband.com/ping/share/cc1418a68757c0f78c674ca6cd0beabe.h=
tml
That lead me to wonder if it could be some form of overloading of
BitFolk's VM hosting nodes. I feel like I have by this point
discounted that possibility though, because I have been emptying off
the node "curacao" to the point where it now has just two VPSes left
on it, one of which is the "pingtest" VPS above, which still shows
the issue. So it's hard to believe that it can be overloading.
Then I wondered about proxy ARP. I worked with our colo provider to
restrict the amount of IP addresses that their routers would ARP
for, and we examined packet traces for ARP activity but that proved
to be fruitless.
So next, is it a problem inherent to Xen? Well, the "penguin" graph
above is a Xen-based VPS running on hardware similar to BitFolk's,
which was set up by me in a virtually-identical way to how I set up
BitFolk's nodes, and it doesn't show the problem.
That's where we are at the moment, and I'm continuing to work on
this. By tomorrow I'll have moved the last remaining customer off of
curacao and then I'll move that node into a different VLAN with
other (non-BitFolk) servers that aren't currently experiencing this
problem, to see what happens.
I'm afraid I can't give you any ETA on when this might be fixed as I
still don't know exactly what the problem is. I will keep you
informed of progress.
Cheers,
Andy
--=20
http://bitfolk.com/ -- No-nonsense VPS hosting
--27KoNqt0fmcl1zj/
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
iEYEAREDAAYFAk/qBmQACgkQIJm2TL8VSQurmwCg6DQPfwjNkIIqbcpfP+qOOV5w
5d8An2cNrgpz46RXGCmWu+7nyy30GZZ5
=Ijzb
-----END PGP SIGNATURE-----
--27KoNqt0fmcl1zj/--
From joseph@??? Wed Jun 27 11:21:02 2012
Received: from button.heenan.me.uk ([85.119.82.222])
by mail.bitfolk.com with esmtp (Exim 4.72)
(envelope-from <joseph@???>) id 1SjqIw-00071B-Kr
for users@???; Wed, 27 Jun 2012 11:21:02 +0000
Received: from dhcp124.sh2.org.uk (home.heenan.me.uk [212.159.108.133])
by button.heenan.me.uk (Postfix) with ESMTPSA id 783A7AC07D
for <users@???>; Wed, 27 Jun 2012 12:20:59 +0100 (BST)
Message-ID: <4FEAEC99.1030002@???>
Date: Wed, 27 Jun 2012 12:20:57 +0100
From: Joseph Heenan <joseph@???>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7;
rv:13.0) Gecko/20120614 Thunderbird/13.0.1
MIME-Version: 1.0
To: users@???
References: <4FE98B93.6000208@???>
<20120626185844.GK11695@???>
In-Reply-To: <20120626185844.GK11695@???>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Scanner: Scanned by ClamAV on mail.bitfolk.com at Wed,
27 Jun 2012 11:21:02 +0000
X-SA-Exim-Connect-IP: 85.119.82.222
X-SA-Exim-Mail-From: joseph@???
X-SA-Exim-Scanned: No (on mail.bitfolk.com); SAEximRunCond expanded to false
Subject: Re: [bitfolk] vps ping response times
X-BeenThere: users@???
X-Mailman-Version: 2.1.13
Precedence: list
List-Id: Users of BitFolk hosting <users.lists.bitfolk.com>
List-Unsubscribe: <https://lists.bitfolk.com/mailman/options/users>,
<mailto:users-request@lists.bitfolk.com?subject=unsubscribe>
List-Archive: <http://lists.bitfolk.com/lurker/list/users.html>
List-Post: <mailto:users@lists.bitfolk.com>
List-Help: <mailto:users-request@lists.bitfolk.com?subject=help>
List-Subscribe: <https://lists.bitfolk.com/mailman/listinfo/users>,
<mailto:users-request@lists.bitfolk.com?subject=subscribe>
X-List-Received-Date: Wed, 27 Jun 2012 11:21:02 -0000
Hi Andy,
On 26/06/2012 19:58, Andy Smith wrote:
[snip]
> I'm afraid I can't give you any ETA on when this might be fixed as I
> still don't know exactly what the problem is. I will keep you informed
> of progress. Cheers, Andy
That's a very detailed response - thanks!
Joseph
From andy@??? Sat Jul 07 13:05:37 2012
Received: from andy by mail.bitfolk.com with local (Exim 4.72)
(envelope-from <andy@???>) id 1SnUhd-000096-4M
for users@???; Sat, 07 Jul 2012 13:05:37 +0000
Date: Sat, 7 Jul 2012 13:05:37 +0000
From: Andy Smith <andy@???>
To: users@???
Message-ID: <20120707130537.GA11695@???>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-ripemd160;
protocol="application/pgp-signature"; boundary="jmkJtp15SxLq1SbD"
Content-Disposition: inline
OpenPGP: id=BF15490B; url=http://strugglers.net/~andy/pubkey.asc
X-URL: http://strugglers.net/wiki/User:Andy
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Virus-Scanner: Scanned by ClamAV on mail.bitfolk.com at Sat,
07 Jul 2012 13:05:37 +0000
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: andy@???
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
spamd2.lon.bitfolk.com
X-Spam-Level:
X-Spam-ASN:
X-Spam-Status: No, score=-0.0 required=5.0 tests=NO_RELAYS shortcircuit=no
autolearn=disabled version=3.3.1
X-Spam-Report: * -0.0 NO_RELAYS Informational: message was not relayed via SMTP
X-SA-Exim-Version: 4.2.1 (built Mon, 22 Mar 2010 06:51:10 +0000)
X-SA-Exim-Scanned: Yes (on mail.bitfolk.com)
Subject: [bitfolk] Proving that you are you
X-BeenThere: users@???
X-Mailman-Version: 2.1.13
Precedence: list
List-Id: Users of BitFolk hosting <users.lists.bitfolk.com>
List-Unsubscribe: <https://lists.bitfolk.com/mailman/options/users>,
<mailto:users-request@lists.bitfolk.com?subject=unsubscribe>
List-Archive: <http://lists.bitfolk.com/lurker/list/users.html>
List-Post: <mailto:users@lists.bitfolk.com>
List-Help: <mailto:users-request@lists.bitfolk.com?subject=help>
List-Subscribe: <https://lists.bitfolk.com/mailman/listinfo/users>,
<mailto:users-request@lists.bitfolk.com?subject=subscribe>
X-List-Received-Date: Sat, 07 Jul 2012 13:05:39 -0000
--jmkJtp15SxLq1SbD
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Hello,
Today a customer popped up on IRC saying that they had broken their
VPS and couldn
%20on%20this%0A>%20>%20surprised%20me.%20Would%20this%20kind%20of%20result%20be%20expected?%0A>%20%0A>%20About%20a%20month%20ago%20I%20was%20made%20aware%20of%20a%20problem%20with%20occasional%0A>%20spikes%20of%20high%20latency,%20and%20on%20looking%20into%20it,%20it%20became%20apparent%0A>%20that%20it%20had%20actually%20been%20the%20case%20for%20a%20long%20time%20-%20perhaps%20years%20-%0A>%20without%20anyone%20really%20noticing.%0A>%20%0A>%20What%20you're%20seeing%20is%20one%20or%20two%20packets%20out%20of%20every%20couple%20of%0A>%20hundred%20being%20delayed%20somewhere,%20sometimes%20for%20hundreds%20of%0A>%20milliseconds.%0A>%20%0A>%20It%20isn't%20restricted%20to%20your%20VPS,%20or%20to%20any%20one%20BitFolk%20server.%20It%0A>%20seems%20to%20be%20affecting%20all%20VPSes,%20but%20as%20I%20say,%20it%20has%20been%20doing%20so%0A>%20for%20a%20very%20long%20time.%20Here's%20a%20graph%20that%20exemplifies%20the%20issue:%0A>%20%0A>%20http://www.thinkbroadband.com/ping/share/9b7cf0ba2197b53c0aeb0f3cff42fb7e.h=%0A>%20tml%0A>%20%0A>%20Since%20then%20I've%20been%20trying%20to%20work%20out%20where%20it's%20actually%0A>%20happening,%20and%20this%20has%20been%20a%20long%20and%20ongoing%20process.%0A>%20%0A>%20Firstly,%20it%20*is*%20restricted%20to%20BitFolk.%20Other%20things%20hosted%20at%20the%0A>%20same%20colo%20are%20not%20seeing%20it.%20Here's%20something%20else%20in%20the%20same%20rack%0A>%20as%20some%20BitFolk%20nodes,%20connected%20to%20the%20same%20switches:%0A>%20%0A>%20http://www.thinkbroadband.com/ping/share/cc1418a68757c0f78c674ca6cd0beabe.h=%0A>%20tml%0A>%20%0A>%20That%20lead%20me%20to%20wonder%20if%20it%20could%20be%20some%20form%20of%20overloading%20of%0A>%20BitFolk's%20VM%20hosting%20nodes.%20I%20feel%20like%20I%20have%20by%20this%20point%0A>%20discounted%20that%20possibility%20though,%20because%20I%20have%20been%20emptying%20off%0A>%20the%20node%20%22curacao%22%20to%20the%20point%20where%20it%20now%20has%20just%20two%20VPSes%20left%0A>%20on%20it,%20one%20of%20which%20is%20the%20%22pingtest%22%20VPS%20above,%20which%20still%20shows%0A>%20the%20issue.%20So%20it's%20hard%20to%20believe%20that%20it%20can%20be%20overloading.%0A>%20%0A>%20Then%20I%20wondered%20about%20proxy%20ARP.%20I%20worked%20with%20our%20colo%20provider%20to%0A>%20restrict%20the%20amount%20of%20IP%20addresses%20that%20their%20routers%20would%20ARP%0A>%20for,%20and%20we%20examined%20packet%20traces%20for%20ARP%20activity%20but%20that%20proved%0A>%20to%20be%20fruitless.%0A>%20%0A>%20So%20next,%20is%20it%20a%20problem%20inherent%20to%20Xen?%20Well,%20the%20%22penguin%22%20graph%0A>%20above%20is%20a%20Xen-based%20VPS%20running%20on%20hardware%20similar%20to%20BitFolk's,%0A>%20which%20was%20set%20up%20by%20me%20in%20a%20virtually-identical%20way%20to%20how%20I%20set%20up%0A>%20BitFolk's%20nodes,%20and%20it%20doesn't%20show%20the%20problem.%0A>%20%0A>%20That's%20where%20we%20are%20at%20the%20moment,%20and%20I'm%20continuing%20to%20work%20on%0A>%20this.%20By%20tomorrow%20I'll%20have%20moved%20the%20last%20remaining%20customer%20off%20of%0A>%20curacao%20and%20then%20I'll%20move%20that%20node%20into%20a%20different%20VLAN%20with%0A>%20other%20(non-BitFolk)%20servers%20that%20aren't%20currently%20experiencing%20this%0A>%20problem,%20to%20see%20what%20happens.%0A>%20%0A>%20I'm%20afraid%20I%20can't%20give%20you%20any%20ETA%20on%20when%20this%20might%20be%20fixed%20as%20I%0A>%20still%20don't%20know%20exactly%20what%20the%20problem%20is.%20I%20will%20keep%20you%0A>%20informed%20of%20progress.%0A>%20%0A>%20Cheers,%0A>%20Andy%0A>%20%0A>%20--=20%0A>%20http://bitfolk.com/%20--%20No-nonsense%20VPS%20hosting%0A>%20%0A>%20--27KoNqt0fmcl1zj/%0A>%20Content-Type:%20application/pgp-signature;%20name=%22signature.asc%22%0A>%20Content-Description:%20Digital%20signature%0A>%20Content-Disposition:%20inline%0A>%20%0A>%20-----BEGIN%20PGP%20SIGNATURE-----%0A>%20Version:%20GnuPG%20v1.4.10%20(GNU/Linux)%0A>%20%0A>%20iEYEAREDAAYFAk/qBmQACgkQIJm2TL8VSQurmwCg6DQPfwjNkIIqbcpfP+qOOV5w%0A>%205d8An2cNrgpz46RXGCmWu+7nyy30GZZ5%0A>%20=Ijzb%0A>%20-----END%20PGP%20SIGNATURE-----%0A>%20%0A>%20--27KoNqt0fmcl1zj/--%0A>%20%0A>%20%0A>%20%0A>%20Hi%20Andy,%0A>%20%0A>%20On%2026/06/2012%2019:58,%20Andy%20Smith%20wrote:%0A>%20%5Bsnip%5D%0A>%20>%20I'm%20afraid%20I%20can't%20give%20you%20any%20ETA%20on%20when%20this%20might%20be%20fixed%20as%20I%20%0A>%20>%20still%20don't%20know%20exactly%20what%20the%20problem%20is.%20I%20will%20keep%20you%20informed%20%0A>%20>%20of%20progress.%20Cheers,%20Andy%0A>%20That's%20a%20very%20detailed%20response%20-%20thanks!%0A>%20%0A>%20Joseph%0A>%20%0A>%20%0A>%20%0A>%20%0A>%20%0A>%20--jmkJtp15SxLq1SbD%0A>%20Content-Type:%20text/plain;%20charset=us-ascii%0A>%20Content-Disposition:%20inline%0A>%20Content-Transfer-Encoding:%20quoted-printable%0A>%20%0A>%20Hello,%0A>%20%0A>%20Today%20a%20customer%20popped%20up%20on%20IRC%20saying%20that%20they%20had%20broken%20their%0A>%20VPS%20and%20couldn "Reply to this message")
