1 Feb
2010
1 Feb
'10
8:11 a.m.
Hi there
I'm running Ubuntu 8.04 on the VPS (image provided by Bitfolk) and periodically the
VPS seems to lock up and I get the following scrolling really fast on the console:
[1366226.746076] =======================
[1366238.351907] BUG: soft lockup - CPU#0 stuck for 11s! [apache2:27046]
[1366238.351920]
[1366238.351925] Pid: 27046, comm: apache2 Tainted: G D (2.6.24-24-xen #1)
[1366238.351929] EIP: 0061:[<c03285fa>] EFLAGS: 00200286 CPU: 0
[1366238.351938] EIP is at _spin_lock+0xa/0x10
[1366238.351940] EAX: d6002dd8 EBX: d6002d94 ECX: c156a9c0 EDX: 00000000
[1366238.351943] ESI: c156a9c0 EDI: 00000000 EBP: d4a4bdd8 ESP: d4a4bc3c
[1366238.351946] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
[1366238.351954] CR0: 80050033 CR2: 08e080c8 CR3: 04d26000 CR4: 00002620
[1366238.351958] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[1366238.351961] DR6: ffff0ff0 DR7: 00000400
[1366238.351965] [<c01a6ebd>] try_to_free_buffers+0x2d/0x90
[1366238.351978] [<c0167c75>] shrink_page_list+0x4c5/0x600
[1366238.351985] [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[1366238.351991] [<c0167ecf>] shrink_inactive_list+0x11f/0x3b0
[1366238.351998] [<c01681fc>] shrink_zone+0x9c/0x100
[1366238.352002] [<c0168d42>] try_to_free_pages+0x152/0x250
[1366238.352008] [<c0162f9b>] __alloc_pages+0x14b/0x390
[1366238.352013] [<c011959c>] kmap_atomic+0x1c/0x30
[1366238.352020] [<c01713dc>] handle_mm_fault+0x8ac/0x1360
[1366238.352025] [<c0164cf2>] test_clear_page_writeback+0x72/0x100
[1366238.352030] [<c0275361>] __add_entropy_words+0x61/0x1f0
[1366238.352037] [<c011e34e>] __wake_up+0x3e/0x60
[1366238.352042] [<c027571b>] add_timer_randomness+0xdb/0x190
[1366238.352047] [<c032a18e>] do_page_fault+0x35e/0xe70
[1366238.352052] [<c01581d1>] handle_IRQ_event+0x41/0x80
[1366238.352056] [<c0159ad6>] handle_level_irq+0xb6/0xf0
[1366238.352061] [<c0254290>] evtchn_do_upcall+0xc0/0x1a0
[1366238.352067] [<c0329e30>] do_page_fault+0x0/0xe70
[1366238.352071] [<c0328ad5>] error_code+0x35/0x40
[1366238.352077] =======================
I've increased the swapfile size to 1GB as I noticed we were using it pretty heavily.
The VPS runs a couple of websites, backed by MySQL.
Any ideas?
Alex
This email carries a disclaimer, a copy of which may be read at
http://learning.longhill.org.uk/disclaimer
1 Feb
1 Feb
2:07 p.m.
Hi,
I'd say apache pid 27046 is waiting for IO after it requested more
memory and the kernel trying to free some by utilising swap.
After 11 seconds (of doing IO) it complained.
_If_ my analysis is correct, you'd have the following Options:
* add more ram
* reduce amount of ram required
* increase IO Speed (new kernel/drivers/etc..)
I wouldn't expect the increase in swapspace to solve it. *Perhaps*
lessen the frequency of this to happen though.
Conrad
On Mon, 2010-02-01 at 08:11 +0000, Alex Harrington wrote:
Hi there
I'm running Ubuntu 8.04 on the VPS (image provided by Bitfolk) and periodically the
VPS seems to lock up and I get the following scrolling really fast on the console:
[1366226.746076] =======================
[1366238.351907] BUG: soft lockup - CPU#0 stuck for 11s! [apache2:27046]
[1366238.351920]
[1366238.351925] Pid: 27046, comm: apache2 Tainted: G D (2.6.24-24-xen #1)
[1366238.351929] EIP: 0061:[<c03285fa>] EFLAGS: 00200286 CPU: 0
[1366238.351938] EIP is at _spin_lock+0xa/0x10
[1366238.351940] EAX: d6002dd8 EBX: d6002d94 ECX: c156a9c0 EDX: 00000000
[1366238.351943] ESI: c156a9c0 EDI: 00000000 EBP: d4a4bdd8 ESP: d4a4bc3c
[1366238.351946] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069
[1366238.351954] CR0: 80050033 CR2: 08e080c8 CR3: 04d26000 CR4: 00002620
[1366238.351958] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
[1366238.351961] DR6: ffff0ff0 DR7: 00000400
[1366238.351965] [<c01a6ebd>] try_to_free_buffers+0x2d/0x90
[1366238.351978] [<c0167c75>] shrink_page_list+0x4c5/0x600
[1366238.351985] [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0
[1366238.351991] [<c0167ecf>] shrink_inactive_list+0x11f/0x3b0
[1366238.351998] [<c01681fc>] shrink_zone+0x9c/0x100
[1366238.352002] [<c0168d42>] try_to_free_pages+0x152/0x250
[1366238.352008] [<c0162f9b>] __alloc_pages+0x14b/0x390
[1366238.352013] [<c011959c>] kmap_atomic+0x1c/0x30
[1366238.352020] [<c01713dc>] handle_mm_fault+0x8ac/0x1360
[1366238.352025] [<c0164cf2>] test_clear_page_writeback+0x72/0x100
[1366238.352030] [<c0275361>] __add_entropy_words+0x61/0x1f0
[1366238.352037] [<c011e34e>] __wake_up+0x3e/0x60
[1366238.352042] [<c027571b>] add_timer_randomness+0xdb/0x190
[1366238.352047] [<c032a18e>] do_page_fault+0x35e/0xe70
[1366238.352052] [<c01581d1>] handle_IRQ_event+0x41/0x80
[1366238.352056] [<c0159ad6>] handle_level_irq+0xb6/0xf0
[1366238.352061] [<c0254290>] evtchn_do_upcall+0xc0/0x1a0
[1366238.352067] [<c0329e30>] do_page_fault+0x0/0xe70
[1366238.352071] [<c0328ad5>] error_code+0x35/0x40
[1366238.352077] =======================
I've increased the swapfile size to 1GB as I noticed we were using it pretty heavily.
The VPS runs a couple of websites, backed by MySQL.
Any ideas?
Alex
This email carries a disclaimer, a copy of which may be read at
http://learning.longhill.org.uk/disclaimer
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
2:18 p.m.
On Mon, Feb 01, 2010 at 03:07:26PM +0100, Conrad Wood wrote:
I'd say apache pid 27046 is waiting for IO after
it requested more
memory and the kernel trying to free some by utilising swap.
After 11 seconds (of doing IO) it complained.
_If_ my analysis is correct, you'd have the following Options:
* add more ram
* reduce amount of ram required
* increase IO Speed (new kernel/drivers/etc..)
* Reduce the footprint of Apache to fit in the RAM you've got.
If you're using the prefork MPM, the defaults are really quite
large for a busy site in limited space. For example, my home site (low
traffic, in the smallest Bitfolk VM) uses:
<IfModule mpm_prefork_module>
StartServers 2
MinSpareServers 1
MaxSpareServers 2
MaxClients 7
MaxRequestsPerChild 0
</IfModule>
What happens is, when you get spike in load (from a badly-written
spider, or a little light slashdotting), apache spawns more processes
to cope. This takes up RAM, and eventually pushes you into swap. Of
course, that then slows down the processing by several orders of
magnitude, ensuring that it falls even further behind, attempting to
spawn even more processes to keep up, and finally the system collapses
completely.
Bringing down the maximum number of processes that Apache will
consider serving so that they all fit in RAM prevents this from
happening.
You could also consider only loading the Apache modules you're
actually using, which saves yet more space.
Hugo.
--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- Great oxymorons of the world, no. 7: The Simple Truth ---
2:47 p.m.
If you're using the prefork MPM, the defaults
are really
quite large for a busy site in limited space. For example, my
home site (low traffic, in the smallest Bitfolk VM) uses:
Hi Hugo
Thanks - I've just done some tuning there so fingers crossed.
Alex
This email carries a disclaimer, a copy of which may be read at
http://learning.longhill.org.uk/disclaimer
3:02 p.m.
** Hugo Mills <hugo-bf(a)carfax.org.uk> [2010-02-01 14:20]:
On Mon, Feb 01, 2010 at 03:07:26PM +0100, Conrad Wood
wrote:
<snip>
I'd say apache pid 27046 is waiting for IO
after it requested more
memory and the kernel trying to free some by utilising swap.
After 11 seconds (of doing IO) it complained.
_If_ my analysis is correct, you'd have the following Options:
* add more ram
* reduce amount of ram required
* increase IO Speed (new kernel/drivers/etc..)
* Reduce the footprint of Apache to fit in the RAM you've got.
What happens is, when you get spike in load (from a badly-written
spider, or a little light slashdotting), apache spawns more processes
to cope. This takes up RAM, and eventually pushes you into swap. Of
course, that then slows down the processing by several orders of
magnitude, ensuring that it falls even further behind, attempting to
spawn even more processes to keep up, and finally the system collapses
completely.
Bringing down the maximum number of processes that Apache will
consider serving so that they all fit in RAM prevents this from
happening.
<snip>
** end quote [Hugo Mills]
Interesting thought. I've been experiencing this sort of problem on a regular
basis, although not had the time to investigate. I've noted similar errors, but
not based on a specific application, and after finding some references to
problems with Ubuntu virtualisation with Xen have assumed that when I get round
to it I'll be looking for a base OS configuration issue.
I've migrated from another company who originally used UML, although they
experimented with Xen and seem to be useing KVM now. I believe I was still on
the UML setup, since I don't remember being advised of a mirgation. That ran
the same services with no problems on the same basic virtual hardware spec.
I really need to start investigating now as although I'm monitoring it, it is
far from ideal. Being impressed with a 2 day uptime is a sad state of affairs!
--
Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/ | 023 9238 0001
======================================================================
Registered in England | Company No: 4905028 | Registered Office:
Crawford House, Hambledon Road, Denmead, Waterlooville, Hants, PO7 6NU
2:46 p.m.
Hello,
Maybe it is something in userland, but I would be interested to know
what happens if you switched to a kernel from Debian Lenny e.g.:
http://packages.debian.org/lenny/linux-image-2.6.26-2-xen-686
http://packages.debian.org/lenny/linux-modules-2.6.26-2-xen-686
or an upstream kernel.org.one.
Cheers,
Andy
3:13 p.m.
On 01/02/10 14:46, Andy Smith wrote:
Hello,
Maybe it is something in userland, but I would be interested to know
what happens if you switched to a kernel from Debian Lenny e.g.:
http://packages.debian.org/lenny/linux-image-2.6.26-2-xen-686
http://packages.debian.org/lenny/linux-modules-2.6.26-2-xen-686
or an upstream kernel.org.one.
Cheers,
Andy
I had this problem a few months back, and a switch to the lenny kernel
seemed to solve it. No problems since I've switched.
6:17 p.m.
** Andy Smith <andy(a)bitfolk.com> [2010-02-01 14:49]:
Hello,
Maybe it is something in userland, but I would be interested to know
what happens if you switched to a kernel from Debian Lenny e.g.:
http://packages.debian.org/lenny/linux-image-2.6.26-2-xen-686
http://packages.debian.org/lenny/linux-modules-2.6.26-2-xen-686
or an upstream kernel.org.one.
** end quote [Andy Smith]
Is this just a case of downloading and installing with dpkg? Possibly using a
custom line in the apt sources to pick up updates as well (obviously avoiding
any other packages being pulled in!). Just wondering whether updating the
kernel is a little more controlled than other packages, but I may be used to my
old UML setup and the even more restricted system I was on before (which one
escapes me!) where the kernel was updated as part of the VPS hosting platform.
Any pointers or suggestions as to ones to try, or good references on compiling
Xen kernels? I can experiment a bit on my VPS to see whether it improves
things.
--
Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/ | 023 9238 0001
======================================================================
Registered in England | Company No: 4905028 | Registered Office:
Crawford House, Hambledon Road, Denmead, Waterlooville, Hants, PO7 6NU
7:57 p.m.
Hi Paul,
On Mon, Feb 01, 2010 at 06:17:09PM +0000, Paul Tansom wrote:
** Andy Smith <andy(a)bitfolk.com> [2010-02-01
14:49]:
Yes, dpkg -i on both the above .debs should just install them.
Hello,
Maybe it is something in userland, but I would be interested to know
what happens if you switched to a kernel from Debian Lenny e.g.:
http://packages.debian.org/lenny/linux-image-2.6.26-2-xen-686
http://packages.debian.org/lenny/linux-modules-2.6.26-2-xen-686
or an upstream kernel.org.one.
** end quote [Andy Smith]
Is this just a case of downloading and installing with dpkg? Possibly using a
custom line in the apt sources to pick up updates as well (obviously avoiding
any other packages being pulled in!).
I think this is possible but haven't looked into how.
Just wondering whether updating the
kernel is a little more controlled than other packages,
Kernel updates in debian/ubuntu are just updated packages with the
package pre/post install scripts handling initrd, grub etc.
You could for example just subscribe to the Debian security mailing
list then you would know when a new kernel .deb was out.
but I may be used to my old UML setup and the even
more restricted
system I was on before (which one escapes me!) where the kernel
was updated as part of the VPS hosting platform.
Your previous emails suggest Bytemark's UML setup.
I could enforce usage of one kernel on everyone which would make it
my problem, but then that's a bit less flexible for everyone else.
I am still considering doing something a bit like slicehost where
you have to request a kernel upgrade and it happens during a
boot/shutdown (don't forget that modules need installing in the
filesystem).
Any pointers or suggestions as to ones to try, or good
references on compiling
Xen kernels? I can experiment a bit on my VPS to see whether it improves
things.
I have successfully compiled an upstream kernel.org kernel like
this:
1. Download kernel source from kernel.org, unpack, cd into it.
2. apt-get install build-essential fakeroot libncurses5-dev kernel-package
3. Obtain a .config from somewhere. You could try your existing
kernel's config that you might find somewhere in /boot/config*
then do "make oldconfig" to update it, though the difference
between something like 2.6.18 and 2.6.32 will be quite
long-winded. I've attached one that I have used successfully
with 2.6.32.7.
4. fakeroot make-kpkg --arch=i386 --revision=pauls-kernel.1 --initrd kernel_image
kernel_headers
The "pauls-kernel.1" part will appear in the package name but not
in the name of the kernel so if you made another kernel from the
same source it would overwrite this one. If you want to make
different packages that can co-exist, use --append-to-version=...
Avoid using "_" in these strings.
You should end up with linux-image and linux-headers .debs in the
directory above the current one.
5. Install the .debs, make sure you have an initrd created. If not
you can make one with e.g.
update-initramfs -v -k 2.6.32.7 -c
6. update-grub and check it appears correctly in the
/boot/grub/menu.lst. If using an upstream kernel then avoid
clocksource=jiffies in the kernel command line.
7. Go to xen shell, shut down, boot, select appropriate kernel, hope
it works. :)
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
8:27 p.m.
On Mon, Feb 01, 2010 at 07:57:39PM +0000, Andy Smith wrote:
Hi Paul,
On Mon, Feb 01, 2010 at 06:17:09PM +0000, Paul Tansom wrote:
I think apt-pinning is what you want.
Thanks,
Rodrigo
** Andy Smith <andy(a)bitfolk.com>
[2010-02-01 14:49]:
Possibly using a
custom line in the apt sources to pick up updates as well (obviously avoiding
any other packages being pulled in!).
I think this is possible but haven't looked into how.
3 Feb
3 Feb
2:53 p.m.
** Rodrigo Campos <rodrigo(a)sdfg.com.ar> [2010-02-01 20:33]:
On Mon, Feb 01, 2010 at 07:57:39PM +0000, Andy Smith
wrote:
** end quote [Rodrigo Campos]
Doesn't pinning hold a specific package at a defined version? I was thinking
more of the issue of duplicate packages being available in the Debian sources
as well as the Ubuntu ones, and therefore clashing.
--
Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/ | 023 9238 0001
======================================================================
Registered in England | Company No: 4905028 | Registered Office:
Crawford House, Hambledon Road, Denmead, Waterlooville, Hants, PO7 6NU
On Mon, Feb 01, 2010 at 06:17:09PM +0000, Paul
Tansom wrote:
I think apt-pinning is what you want. ** Andy Smith <andy(a)bitfolk.com>
[2010-02-01 14:49]:
Possibly using a
custom line in the apt sources to pick up updates as well (obviously avoiding
any other packages being pulled in!).
I think this is possible but haven't looked into how.
3:31 p.m.
On Wed, Feb 03, 2010 at 02:53:25PM +0000, Paul Tansom wrote:
** Rodrigo Campos <rodrigo(a)sdfg.com.ar>
[2010-02-01 20:33]:
I think more generally it holds a package to the version in a
specific repository (rather than the latest available over all
repositories).
Hugo.
--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- This chap Anon is writing some perfectly lovely stuff ---
at the moment.
On Mon, Feb 01, 2010 at 07:57:39PM +0000, Andy
Smith wrote:
** end quote [Rodrigo Campos]
Doesn't pinning hold a specific package at a defined version? I was thinking
more of the issue of duplicate packages being available in the Debian sources
as well as the Ubuntu ones, and therefore clashing. On Mon, Feb 01, 2010 at 06:17:09PM +0000, Paul
Tansom wrote:
I think apt-pinning is what you want. ** Andy Smith <andy(a)bitfolk.com>
[2010-02-01 14:49]:
Possibly using a
custom line in the apt sources to pick up updates as well (obviously avoiding
any other packages being pulled in!).
I think this is possible but haven't looked into how.
1 Feb
1 Feb
9:31 p.m.
Yes, dpkg -i on both the above .debs should just
install them.
Thanks Andy. I've switched to the Debian kernel you suggested. I'll let you know
if this comes up again.
Regards
Alex
This email carries a disclaimer, a copy of which may be read at
http://learning.longhill.org.uk/disclaimer
3 Feb
3 Feb
2:50 p.m.
** Andy Smith <andy(a)bitfolk.com> [2010-02-01 20:00]:
I've got a feeling that in the deb line in sources.list you could specify a
particular package to pick from the repository and not take the lot. I can't
find the reference now to see if my memory serves correctly, backports.org
seems the most likely source!
Already on that, given I've used Debian for some years now, so will likely use
that for now.
I was avoiding mentioning my previous provider, in case I say something less
complimentary about them at some point!
<snip>
Thanks, it is a while since I did much kernel compilation, in fact I'm not 100%
sure I've bothered since I moved to Debian. Back on Red Hat it was, from my
first Linux install, a requirement so that I could use the Token Ring NIC
drivers! Those were the days!
** end quote [Andy Smith]
--
Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/ | 023 9238 0001
======================================================================
Registered in England | Company No: 4905028 | Registered Office:
Crawford House, Hambledon Road, Denmead, Waterlooville, Hants, PO7 6NU
On Mon, Feb 01, 2010 at 06:17:09PM +0000, Paul Tansom
wrote:
Thankfully quite happy with dpkg, so it may be time to give it a go. Having
just popped back up from lower down (will make sense if you continue reading!)
are the module packages to install with these, or given that the kernel
versions match, are the Ubuntu ones adequate? I'm assuming the latter, but if I
use a newer kernel then modules would be required.
** Andy Smith <andy(a)bitfolk.com>
[2010-02-01 14:49]:
Yes, dpkg -i on both the above .debs should just install them. Maybe it is something in userland, but I would be
interested to know
what happens if you switched to a kernel from Debian Lenny e.g.:
http://packages.debian.org/lenny/linux-image-2.6.26-2-xen-686
http://packages.debian.org/lenny/linux-modules-2.6.26-2-xen-686
or an upstream kernel.org.one.
** end quote [Andy Smith]
Is this just a case of downloading and installing with dpkg? Possibly using
a
custom line in the apt sources to pick up updates as well (obviously avoiding
any other packages being pulled in!).
I think this is possible but haven't looked into how. Just wondering
whether updating the
kernel is a little more controlled than other packages,
Kernel updates in debian/ubuntu are just updated packages with the
package pre/post install scripts handling initrd, grub etc.
You could for example just subscribe to the Debian security mailing
list then you would know when a new kernel .deb was out. but I may be
used to my old UML setup and the even more restricted
system I was on before (which one escapes me!) where the kernel
was updated as part of the VPS hosting platform.
Your previous emails suggest Bytemark's UML setup. I could enforce usage of one kernel on everyone which
would make it
my problem, but then that's a bit less flexible for everyone else.
I am still considering doing something a bit like slicehost where
you have to request a kernel upgrade and it happens during a
boot/shutdown (don't forget that modules need installing in the
filesystem).
Personally I have no issue with being responsible, so long as I have some
guidelines to work to regarding the underlying VM requirements :) On which note
I'll head back up this reply with a comment!
Any pointers
or suggestions as to ones to try, or good references on compiling
Xen kernels? I can experiment a bit on my VPS to see whether it improves
things.
I have successfully compiled an upstream kernel.org kernel like
this:
5436
days inactive
5438
days old
13 comments
7 participants
participants (7)
-
Alex Harrington -
Andrew Gee -
Andy Smith -
Conrad Wood -
Hugo Mills -
Paul Tansom -
Rodrigo Campos