Hi Alastair, On Wed, Sep 07, 2011 at 06:51:02AM +0100, Alastair Sherringham wrote: But you are on a different host, running under a different version of Xen. Cheers, Andy

Hi Andy, On Tue, September 6, 2011 11:22 pm, Andy Smith wrote: Forgive my ignorance, but having just done aptitude install linux-image-2.6-686-bigmem I get the following during the configuration phase: You are attempting to install an initrd kernel image (version 2.6.26-2-686-bigmem). This will not work unless the boot loader is configured to use an initrd. In order to configure LILO, you need to add 'initrd=/initrd.img' to the image=/vmlinuz stanza of /etc/lilo.conf. The boot loader must be configured to use such images and the system will not boot until this is done. This message will appear for any new kernel installation unless the following is added to /etc/kernel-img.conf: "do_initrd = Yes" The next screen then asks if I want to abort the initrd kernel installation (default yes). Is there anything I need to do, or should I just let it proceed? I didn't want to make an uneducated guess... Regards, Mathew

On Wed, September 7, 2011 9:14 pm, Andy Smith wrote: Thanks Andy. Kernel installed (and now running) with clocksource=jiffies removed. Will see how I get on! Thanks for all your help, Mathew

Hi Mathew, On Tue, Sep 20, 2011 at 08:04:30PM +0100, Mathew Newton wrote: [...] It's now looking like it's a problem with that host (dunkel). I suspect software rather than hardware problem because it has run for a year+ under Xen 3.x before it was recently taken out of service and upgraded to Xen 4.x. Yours is not the only VM on there to have seen a sudden 47s skew in time; two of BitFolk's own VMs saw similar at around the same time (15:54Z today): panel0: Sep 20 16:00:07 panel0 ntpd[1802]: no servers reachable [...] Sep 20 16:11:39 panel0 ntpd[1802]: time reset -46.869813 s resntp2: Sep 20 15:54:56 resntp2 heartbeat: [1407]: WARN: Gmain_timeout_dispatch: Dispatch function for send local status was delayed 46000 ms (> 510 ms) before being called (GSource: 0x86a9720) Sep 20 15:54:56 resntp2 heartbeat: [1407]: WARN: Late heartbeat: Node resntp2: interval 47000 ms Sep 20 15:54:56 resntp2 heartbeat: [1407]: WARN: node resntp0: is dead Sep 20 15:54:56 resntp2 heartbeat: [1407]: WARN: node resntp1: is dead Sep 20 15:54:56 resntp2 heartbeat: [1407]: WARN: node resntp3: is dead (yes, heartbeat got *really* upset and caused a cluster failover) There's only three BitFolk VMs on that server: panel0, resntp2, spamd0. There's an ntpd on all of the VMs, but only the ntpd of panel0 complained. Only heartbeat on resntp2 complained, not ntpd. Nothing complained on spamd0 but that could just be because it has no software which noticed. | Distro | Kernel | Clock | Available | version | version | source | clock sources --------+---------+---------------------+--------+---------------- panel0 | lenny | 2.6.26-2-686-bigmem | xen | xen tsc jiffies --------+---------+---------------------+--------+---------------- resntp2 | squeeze | 2.6.32-5-686-bigmem | tsc | xen tsc --------+---------+---------------------+--------+---------------- spamd0 | squeeze | 2.6.32-5-686-bigmem | xen | xen tsc --------+---------+---------------------+--------+---------------- Interesting how it is always ~47s. Also: panel0 $ zgrep "time reset" /var/log/syslog* /var/log/syslog:Sep 20 16:11:39 panel0 ntpd[1802]: time reset -46.869813 s /var/log/syslog.2.gz:Sep 6 12:58:41 panel0 ntpd[1742]: time reset -46.869301 s /var/log/syslog.32.gz:Jul 27 11:19:35 panel0 ntpd[1742]: time reset -46.869786 s /var/log/syslog.59.gz:Jun 29 19:05:10 panel0 ntpd[1708]: time reset -46.868480 s resntp2 $ zgrep "Late heartbeat" /var/log/syslog* /var/log/syslog:Sep 20 15:54:56 resntp2 heartbeat: [1407]: WARN: Late heartbeat: Node resntp2: interval 47000 ms /var/log/syslog:Sep 20 15:54:56 resntp2 heartbeat: [1407]: WARN: Late heartbeat: Node resntp0: interval 47780 ms /var/log/syslog:Sep 20 15:54:56 resntp2 heartbeat: [1407]: WARN: Late heartbeat: Node resntp1: interval 47780 ms /var/log/syslog:Sep 20 15:54:57 resntp2 heartbeat: [1407]: WARN: Late heartbeat: Node resntp3: interval 48280 ms /var/log/syslog.2.gz:Sep 6 12:43:02 resntp2 heartbeat: [1407]: WARN: Late heartbeat: Node resntp2: interval 47590 ms /var/log/syslog.2.gz:Sep 6 12:43:02 resntp2 heartbeat: [1407]: WARN: Late heartbeat: Node resntp0: interval 47970 ms /var/log/syslog.2.gz:Sep 6 12:43:02 resntp2 heartbeat: [1407]: WARN: Late heartbeat: Node resntp3: interval 47590 ms /var/log/syslog.2.gz:Sep 6 12:43:02 resntp2 heartbeat: [1407]: WARN: Late heartbeat: Node resntp1: interval 48100 ms (logs on this host only back to Aug 20th) dunkel was last booted on June 16th so I suspect this has always been the case on this host and software combination. Can anyone else experiencing similar things please send me an email off-list with what you saw and the distribution, kernel versions and clocksources? You can find the clocksource info in: /sys/devices/system/clocksource0/current_clocksource /sys/devices/system/clocksource0/available_clocksource I'm afraid this could be tricky to fix and I'm going to have to ask you and others to bear with me while I investigate. I would rather not take the whole server out of service right now, but if you or anyone else wants to be moved to another server please send in a support ticket and it will be done. If I don't find a solution in a reasonable period of time then I am absolutely willing to put you on different hardware and take it out of service (and probably bring it home and beat it to pieces in the front garden, Office Space style). Cheers, Andy -- http://bitfolk.com/ -- No-nonsense VPS hosting

Hello, On Tue, Sep 20, 2011 at 10:45:23PM +0100, Mathew Newton wrote: In the mean time you could try starting ntpd with the -x argument (keeping the usual -g as well): -x Normally, the time is slewed if the offset is less than the step threshold, which is 128 ms by default, and stepped if above the threshold. This option sets the threshold to 600 s, which is well within the accuracy window to set the clock manually. Note: Since the slew rate of typical Unix kernels is limited to 0.5 ms/s, each second of adjustment requires an amortization interval of 2000 s. Thus, an adjustment as much as 600 s will take almost 14 days to complete. This option can be used with the -g and -q options. Note: The kernel time discipline is disabled with this option. AFAIK that would cause your ntpd to slow time down in order to correct a ~47 second gain, instead of stepping it backwards all at once. The downside would of course be that your clock would be fast by a decreasing fraction of 47 seconds for the next ~26 hours. On Debian and Ubuntu you can find the default ntpd arguments in /etc/default/ntpd. Cheers, Andy -- http://bitfolk.com/ -- No-nonsense VPS hosting