Andy Smith <andy@???> wrote:
>There's some work being done... It shouldn't be noticeable.
That's what they told RBS :)
cheers,
Chris
--
Chris Dennis
Fordingbridge, Hampshire
From joseph@??? Tue Jun 26 10:14:49 2012
Received: from button.heenan.me.uk ([85.119.82.222])
by mail.bitfolk.com with esmtp (Exim 4.72)
(envelope-from <joseph@???>) id 1SjSnJ-0000K8-0M
for users@???; Tue, 26 Jun 2012 10:14:49 +0000
Received: from Joseph-Heenans-MacBook-Pro.local (glasgow.emobix.co.uk
[81.187.12.227])
by button.heenan.me.uk (Postfix) with ESMTPSA id AF734AC07D
for <users@???>; Tue, 26 Jun 2012 11:14:46 +0100 (BST)
Message-ID: <4FE98B93.6000208@???>
Date: Tue, 26 Jun 2012 11:14:43 +0100
From: Joseph Heenan <joseph@???>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7;
rv:12.0) Gecko/20120428 Thunderbird/12.0.1
MIME-Version: 1.0
To: users@???
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Scanner: Scanned by ClamAV on mail.bitfolk.com at Tue,
26 Jun 2012 10:14:49 +0000
X-SA-Exim-Connect-IP: 85.119.82.222
X-SA-Exim-Mail-From: joseph@???
X-SA-Exim-Scanned: No (on mail.bitfolk.com); SAEximRunCond expanded to false
Subject: [bitfolk] vps ping response times
X-BeenThere: users@???
X-Mailman-Version: 2.1.13
Precedence: list
List-Id: Users of BitFolk hosting <users.lists.bitfolk.com>
List-Unsubscribe: <https://lists.bitfolk.com/mailman/options/users>,
<mailto:users-request@lists.bitfolk.com?subject=unsubscribe>
List-Archive: <http://lists.bitfolk.com/lurker/list/users.html>
List-Post: <mailto:users@lists.bitfolk.com>
List-Help: <mailto:users-request@lists.bitfolk.com?subject=help>
List-Subscribe: <https://lists.bitfolk.com/mailman/listinfo/users>,
<mailto:users-request@lists.bitfolk.com?subject=subscribe>
X-List-Received-Date: Tue, 26 Jun 2012 10:14:49 -0000
Hi all,
I got into a discussion with one of the people I'd recommended to
bitfolk about ping times to VPSes (he was seeing some spikes). So I
setup some graphs, but didn't see what I expected.
Here's the graph for his vps:
http://f8lure.mouselike.org/archived_graphs/baa.muuh.co.uk_day25.png
which basically looks fine (which doesn't match what he was seeing from
his own graphing, which is odd), and here's the graph for my VPS:
http://f8lure.mouselike.org/archived_graphs/button.heenan.me.uk_day25.png
The number of huge spikes (and some packet loss, shown red) on this
surprised me. Would this kind of result be expected?
Both graphs are for yesterday afternoon. Both vpses where pretty much
idle as best we can tell.
Cheers,
Joseph
From andy@??? Tue Jun 26 18:58:44 2012
Received: from andy by mail.bitfolk.com with local (Exim 4.72)
(envelope-from <andy@???>) id 1SjayK-0004po-Cw
for users@???; Tue, 26 Jun 2012 18:58:44 +0000
Date: Tue, 26 Jun 2012 18:58:44 +0000
From: Andy Smith <andy@???>
To: users@???
Message-ID: <20120626185844.GK11695@???>
References: <4FE98B93.6000208@???>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-ripemd160;
protocol="application/pgp-signature"; boundary="27KoNqt0fmcl1zj/"
Content-Disposition: inline
In-Reply-To: <4FE98B93.6000208@???>
OpenPGP: id=BF15490B; url=http://strugglers.net/~andy/pubkey.asc
X-URL: http://strugglers.net/wiki/User:Andy
User-Agent: Mutt/1.5.20 (2009-06-14)
X-Virus-Scanner: Scanned by ClamAV on mail.bitfolk.com at Tue,
26 Jun 2012 18:58:44 +0000
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: andy@???
X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on
spamd1.lon.bitfolk.com
X-Spam-Level:
X-Spam-ASN:
X-Spam-Status: No, score=-0.0 required=5.0 tests=NO_RELAYS shortcircuit=no
autolearn=disabled version=3.3.1
X-Spam-Report: * -0.0 NO_RELAYS Informational: message was not relayed via SMTP
X-SA-Exim-Version: 4.2.1 (built Mon, 22 Mar 2010 06:51:10 +0000)
X-SA-Exim-Scanned: Yes (on mail.bitfolk.com)
Subject: Re: [bitfolk] vps ping response times
X-BeenThere: users@???
X-Mailman-Version: 2.1.13
Precedence: list
List-Id: Users of BitFolk hosting <users.lists.bitfolk.com>
List-Unsubscribe: <https://lists.bitfolk.com/mailman/options/users>,
<mailto:users-request@lists.bitfolk.com?subject=unsubscribe>
List-Archive: <http://lists.bitfolk.com/lurker/list/users.html>
List-Post: <mailto:users@lists.bitfolk.com>
List-Help: <mailto:users-request@lists.bitfolk.com?subject=help>
List-Subscribe: <https://lists.bitfolk.com/mailman/listinfo/users>,
<mailto:users-request@lists.bitfolk.com?subject=subscribe>
X-List-Received-Date: Tue, 26 Jun 2012 18:58:44 -0000
--27KoNqt0fmcl1zj/
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
Hi Joseph,
On Tue, Jun 26, 2012 at 11:14:43AM +0100, Joseph Heenan wrote:
> here's the graph for my > VPS:
>=20
> http://f8lure.mouselike.org/archived_graphs/button.heenan.me.uk_day25.png
>=20
> The number of huge spikes (and some packet loss, shown red) on this
> surprised me. Would this kind of result be expected?
About a month ago I was made aware of a problem with occasional
spikes of high latency, and on looking into it, it became apparent
that it had actually been the case for a long time - perhaps years -
without anyone really noticing.
What you're seeing is one or two packets out of every couple of
hundred being delayed somewhere, sometimes for hundreds of
milliseconds.
It isn't restricted to your VPS, or to any one BitFolk server. It
seems to be affecting all VPSes, but as I say, it has been doing so
for a very long time. Here's a graph that exemplifies the issue:
http://www.thinkbroadband.com/ping/share/9b7cf0ba2197b53c0aeb0f3cff42fb7e.h=
tml
Since then I've been trying to work out where it's actually
happening, and this has been a long and ongoing process.
Firstly, it *is* restricted to BitFolk. Other things hosted at the
same colo are not seeing it. Here's something else in the same rack
as some BitFolk nodes, connected to the same switches:
http://www.thinkbroadband.com/ping/share/cc1418a68757c0f78c674ca6cd0beabe.h=
tml
That lead me to wonder if it could be some form of overloading of
BitFolk's VM hosting nodes. I feel like I have by this point
discounted that possibility though, because I have been emptying off
the node "curacao" to the point where it now has just two VPSes left
on it, one of which is the "pingtest" VPS above, which still shows
the issue. So it's hard to believe that it can be overloading.
Then I wondered about proxy ARP. I worked with our colo provider to
restrict the amount of IP addresses that their routers would ARP
for, and we examined packet traces for ARP activity but that proved
to be fruitless.
So next, is it a problem inherent to Xen? Well, the "penguin" graph
above is a Xen-based VPS running on hardware similar to BitFolk's,
which was set up by me in a virtually-identical way to how I set up
BitFolk's nodes, and it doesn't show the problem.
That's where we are at the moment, and I'm continuing to work on
this. By tomorrow I'll have moved the last remaining customer off of
curacao and then I'll move that node into a different VLAN with
other (non-BitFolk) servers that aren't currently experiencing this
problem, to see what happens.
I'm afraid I can't give you any ETA on when this might be fixed as I
still don't know exactly what the problem is. I will keep you
informed of progress.
Cheers,
Andy
--=20
http://bitfolk.com/ -- No-nonsense VPS hosting
--27KoNqt0fmcl1zj/
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
iEYEAREDAAYFAk/qBmQACgkQIJm2TL8VSQurmwCg6DQPfwjNkIIqbcpfP+qOOV5w
5d8An2cNrgpz46RXGCmWu+7nyy30GZZ5
=Ijzb
-----END PGP SIGNATURE-----
--27KoNqt0fmcl1zj/--
From joseph@??? Wed Jun 27 11:21:02 2012
Received: from button.heenan.me.uk ([85.119.82.222])
by mail.bitfolk.com with esmtp (Exim 4.72)
(envelope-from <joseph@???>) id 1SjqIw-00071B-Kr
for users@???; Wed, 27 Jun 2012 11:21:02 +0000
Received: from dhcp124.sh2.org.uk (home.heenan.me.uk [212.159.108.133])
by button.heenan.me.uk (Postfix) with ESMTPSA id 783A7AC07D
%0A>%20%0A>%20cheers,%20%0A>%20%0A>%20Chris%0A>%20%0A>%20--%20%0A>%20Chris%20Dennis%0A>%20Fordingbridge,%20Hampshire%0A>%20%0A>%20%0A>%20%0A>%20Hi%20all,%0A>%20%0A>%20I%20got%20into%20a%20discussion%20with%20one%20of%20the%20people%20I'd%20recommended%20to%20%0A>%20bitfolk%20about%20ping%20times%20to%20VPSes%20(he%20was%20seeing%20some%20spikes).%20So%20I%20%0A>%20setup%20some%20graphs,%20but%20didn't%20see%20what%20I%20expected.%0A>%20%0A>%20Here's%20the%20graph%20for%20his%20vps:%0A>%20%0A>%20http://f8lure.mouselike.org/archived_graphs/baa.muuh.co.uk_day25.png%0A>%20%0A>%20which%20basically%20looks%20fine%20(which%20doesn't%20match%20what%20he%20was%20seeing%20from%20%0A>%20his%20own%20graphing,%20which%20is%20odd),%20and%20here's%20the%20graph%20for%20my%20VPS:%0A>%20%0A>%20http://f8lure.mouselike.org/archived_graphs/button.heenan.me.uk_day25.png%0A>%20%0A>%20The%20number%20of%20huge%20spikes%20(and%20some%20packet%20loss,%20shown%20red)%20on%20this%20%0A>%20surprised%20me.%20Would%20this%20kind%20of%20result%20be%20expected?%0A>%20%0A>%20Both%20graphs%20are%20for%20yesterday%20afternoon.%20Both%20vpses%20where%20pretty%20much%20%0A>%20idle%20as%20best%20we%20can%20tell.%0A>%20%0A>%20Cheers,%0A>%20%0A>%20Joseph%0A>%20%0A>%20%0A>%20%0A>%20%0A>%20%0A>%20--27KoNqt0fmcl1zj/%0A>%20Content-Type:%20text/plain;%20charset=us-ascii%0A>%20Content-Disposition:%20inline%0A>%20Content-Transfer-Encoding:%20quoted-printable%0A>%20%0A>%20Hi%20Joseph,%0A>%20%0A>%20On%20Tue,%20Jun%2026,%202012%20at%2011:14:43AM%20+0100,%20Joseph%20Heenan%20wrote:%0A>%20>%20here's%20the%20graph%20for%20my%20>%20VPS:%0A>%20>=20%0A>%20>%20http://f8lure.mouselike.org/archived_graphs/button.heenan.me.uk_day25.png%0A>%20>=20%0A>%20>%20The%20number%20of%20huge%20spikes%20(and%20some%20packet%20loss,%20shown%20red)%20on%20this%0A>%20>%20surprised%20me.%20Would%20this%20kind%20of%20result%20be%20expected?%0A>%20%0A>%20About%20a%20month%20ago%20I%20was%20made%20aware%20of%20a%20problem%20with%20occasional%0A>%20spikes%20of%20high%20latency,%20and%20on%20looking%20into%20it,%20it%20became%20apparent%0A>%20that%20it%20had%20actually%20been%20the%20case%20for%20a%20long%20time%20-%20perhaps%20years%20-%0A>%20without%20anyone%20really%20noticing.%0A>%20%0A>%20What%20you're%20seeing%20is%20one%20or%20two%20packets%20out%20of%20every%20couple%20of%0A>%20hundred%20being%20delayed%20somewhere,%20sometimes%20for%20hundreds%20of%0A>%20milliseconds.%0A>%20%0A>%20It%20isn't%20restricted%20to%20your%20VPS,%20or%20to%20any%20one%20BitFolk%20server.%20It%0A>%20seems%20to%20be%20affecting%20all%20VPSes,%20but%20as%20I%20say,%20it%20has%20been%20doing%20so%0A>%20for%20a%20very%20long%20time.%20Here's%20a%20graph%20that%20exemplifies%20the%20issue:%0A>%20%0A>%20http://www.thinkbroadband.com/ping/share/9b7cf0ba2197b53c0aeb0f3cff42fb7e.h=%0A>%20tml%0A>%20%0A>%20Since%20then%20I've%20been%20trying%20to%20work%20out%20where%20it's%20actually%0A>%20happening,%20and%20this%20has%20been%20a%20long%20and%20ongoing%20process.%0A>%20%0A>%20Firstly,%20it%20*is*%20restricted%20to%20BitFolk.%20Other%20things%20hosted%20at%20the%0A>%20same%20colo%20are%20not%20seeing%20it.%20Here's%20something%20else%20in%20the%20same%20rack%0A>%20as%20some%20BitFolk%20nodes,%20connected%20to%20the%20same%20switches:%0A>%20%0A>%20http://www.thinkbroadband.com/ping/share/cc1418a68757c0f78c674ca6cd0beabe.h=%0A>%20tml%0A>%20%0A>%20That%20lead%20me%20to%20wonder%20if%20it%20could%20be%20some%20form%20of%20overloading%20of%0A>%20BitFolk's%20VM%20hosting%20nodes.%20I%20feel%20like%20I%20have%20by%20this%20point%0A>%20discounted%20that%20possibility%20though,%20because%20I%20have%20been%20emptying%20off%0A>%20the%20node%20%22curacao%22%20to%20the%20point%20where%20it%20now%20has%20just%20two%20VPSes%20left%0A>%20on%20it,%20one%20of%20which%20is%20the%20%22pingtest%22%20VPS%20above,%20which%20still%20shows%0A>%20the%20issue.%20So%20it's%20hard%20to%20believe%20that%20it%20can%20be%20overloading.%0A>%20%0A>%20Then%20I%20wondered%20about%20proxy%20ARP.%20I%20worked%20with%20our%20colo%20provider%20to%0A>%20restrict%20the%20amount%20of%20IP%20addresses%20that%20their%20routers%20would%20ARP%0A>%20for,%20and%20we%20examined%20packet%20traces%20for%20ARP%20activity%20but%20that%20proved%0A>%20to%20be%20fruitless.%0A>%20%0A>%20So%20next,%20is%20it%20a%20problem%20inherent%20to%20Xen?%20Well,%20the%20%22penguin%22%20graph%0A>%20above%20is%20a%20Xen-based%20VPS%20running%20on%20hardware%20similar%20to%20BitFolk's,%0A>%20which%20was%20set%20up%20by%20me%20in%20a%20virtually-identical%20way%20to%20how%20I%20set%20up%0A>%20BitFolk's%20nodes,%20and%20it%20doesn't%20show%20the%20problem.%0A>%20%0A>%20That's%20where%20we%20are%20at%20the%20moment,%20and%20I'm%20continuing%20to%20work%20on%0A>%20this.%20By%20tomorrow%20I'll%20have%20moved%20the%20last%20remaining%20customer%20off%20of%0A>%20curacao%20and%20then%20I'll%20move%20that%20node%20into%20a%20different%20VLAN%20with%0A>%20other%20(non-BitFolk)%20servers%20that%20aren't%20currently%20experiencing%20this%0A>%20problem,%20to%20see%20what%20happens.%0A>%20%0A>%20I'm%20afraid%20I%20can't%20give%20you%20any%20ETA%20on%20when%20this%20might%20be%20fixed%20as%20I%0A>%20still%20don't%20know%20exactly%20what%20the%20problem%20is.%20I%20will%20keep%20you%0A>%20informed%20of%20progress.%0A>%20%0A>%20Cheers,%0A>%20Andy%0A>%20%0A>%20--=20%0A>%20http://bitfolk.com/%20--%20No-nonsense%20VPS%20hosting%0A>%20%0A>%20--27KoNqt0fmcl1zj/%0A>%20Content-Type:%20application/pgp-signature;%20name=%22signature.asc%22%0A>%20Content-Description:%20Digital%20signature%0A>%20Content-Disposition:%20inline%0A>%20%0A>%20-----BEGIN%20PGP%20SIGNATURE-----%0A>%20Version:%20GnuPG%20v1.4.10%20(GNU/Linux)%0A>%20%0A>%20iEYEAREDAAYFAk/qBmQACgkQIJm2TL8VSQurmwCg6DQPfwjNkIIqbcpfP+qOOV5w%0A>%205d8An2cNrgpz46RXGCmWu+7nyy30GZZ5%0A>%20=Ijzb%0A>%20-----END%20PGP%20SIGNATURE-----%0A>%20%0A>%20--27KoNqt0fmcl1zj/--%0A>%20%0A>%20%0A>%20%20%20%20%20 "Reply to this message")
