20 Sep
2011
20 Sep
'11
8:20 p.m.
Hi Mathew,
On Tue, Sep 20, 2011 at 08:04:30PM +0100, Mathew Newton wrote:
You may recall recently a couple of us had time issues
on Dunkel and I
gave your suggestion a try:
[...]
Unfortunately the problem is back!
It's now looking like it's a problem with that host (dunkel). I
suspect software rather than hardware problem because it has run for
a year+ under Xen 3.x before it was recently taken out of service
and upgraded to Xen 4.x.
Yours is not the only VM on there to have seen a sudden 47s skew in
time; two of BitFolk's own VMs saw similar at around the same time
(15:54Z today):
panel0:
Sep 20 16:00:07 panel0 ntpd[1802]: no servers reachable
[...]
Sep 20 16:11:39 panel0 ntpd[1802]: time reset -46.869813 s
resntp2:
Sep 20 15:54:56 resntp2 heartbeat: [1407]: WARN: Gmain_timeout_dispatch: Dispatch function
for send local status was delayed 46000 ms (> 510 ms) before being called (GSource:
0x86a9720)
Sep 20 15:54:56 resntp2 heartbeat: [1407]: WARN: Late heartbeat: Node resntp2: interval
47000 ms
Sep 20 15:54:56 resntp2 heartbeat: [1407]: WARN: node resntp0: is dead
Sep 20 15:54:56 resntp2 heartbeat: [1407]: WARN: node resntp1: is dead
Sep 20 15:54:56 resntp2 heartbeat: [1407]: WARN: node resntp3: is dead
(yes, heartbeat got *really* upset and caused a cluster failover)
There's only three BitFolk VMs on that server: panel0, resntp2,
spamd0. There's an ntpd on all of the VMs, but only the ntpd of
panel0 complained. Only heartbeat on resntp2 complained, not ntpd.
Nothing complained on spamd0 but that could just be because it has
no software which noticed.
| Distro | Kernel | Clock | Available
| version | version | source | clock sources
--------+---------+---------------------+--------+----------------
panel0 | lenny | 2.6.26-2-686-bigmem | xen | xen tsc jiffies
--------+---------+---------------------+--------+----------------
resntp2 | squeeze | 2.6.32-5-686-bigmem | tsc | xen tsc
--------+---------+---------------------+--------+----------------
spamd0 | squeeze | 2.6.32-5-686-bigmem | xen | xen tsc
--------+---------+---------------------+--------+----------------
Interesting how it is always ~47s.
Also:
panel0 $ zgrep "time reset" /var/log/syslog*
/var/log/syslog:Sep 20 16:11:39 panel0 ntpd[1802]: time reset -46.869813 s
/var/log/syslog.2.gz:Sep 6 12:58:41 panel0 ntpd[1742]: time reset -46.869301 s
/var/log/syslog.32.gz:Jul 27 11:19:35 panel0 ntpd[1742]: time reset -46.869786 s
/var/log/syslog.59.gz:Jun 29 19:05:10 panel0 ntpd[1708]: time reset -46.868480 s
resntp2 $ zgrep "Late heartbeat" /var/log/syslog*
/var/log/syslog:Sep 20 15:54:56 resntp2 heartbeat: [1407]: WARN: Late heartbeat: Node
resntp2: interval 47000 ms
/var/log/syslog:Sep 20 15:54:56 resntp2 heartbeat: [1407]: WARN: Late heartbeat: Node
resntp0: interval 47780 ms
/var/log/syslog:Sep 20 15:54:56 resntp2 heartbeat: [1407]: WARN: Late heartbeat: Node
resntp1: interval 47780 ms
/var/log/syslog:Sep 20 15:54:57 resntp2 heartbeat: [1407]: WARN: Late heartbeat: Node
resntp3: interval 48280 ms
/var/log/syslog.2.gz:Sep 6 12:43:02 resntp2 heartbeat: [1407]: WARN: Late heartbeat: Node
resntp2: interval 47590 ms
/var/log/syslog.2.gz:Sep 6 12:43:02 resntp2 heartbeat: [1407]: WARN: Late heartbeat: Node
resntp0: interval 47970 ms
/var/log/syslog.2.gz:Sep 6 12:43:02 resntp2 heartbeat: [1407]: WARN: Late heartbeat: Node
resntp3: interval 47590 ms
/var/log/syslog.2.gz:Sep 6 12:43:02 resntp2 heartbeat: [1407]: WARN: Late heartbeat: Node
resntp1: interval 48100 ms
(logs on this host only back to Aug 20th)
dunkel was last booted on June 16th so I suspect this has always
been the case on this host and software combination.
Can anyone else experiencing similar things please send me an email
off-list with what you saw and the distribution, kernel versions and
clocksources? You can find the clocksource info in:
/sys/devices/system/clocksource0/current_clocksource
/sys/devices/system/clocksource0/available_clocksource
I'm afraid this could be tricky to fix and I'm going to have to ask
you and others to bear with me while I investigate.
I would rather not take the whole server out of service right now,
but if you or anyone else wants to be moved to another server please
send in a support ticket and it will be done.
If I don't find a solution in a reasonable period of time then I am
absolutely willing to put you on different hardware and take it out
of service (and probably bring it home and beat it to pieces in the
front garden, Office Space style).
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting