CPU Soft Lockup

List overview All Threads
Download

newer

older

Checking VPS performance

panel password reset / change...

Alex Harrington

1 Feb 2010 1 Feb '10

8:11 a.m.

Hi there I'm running Ubuntu 8.04 on the VPS (image provided by Bitfolk) and periodically the VPS seems to lock up and I get the following scrolling really fast on the console: [1366226.746076] ======================= [1366238.351907] BUG: soft lockup - CPU#0 stuck for 11s! [apache2:27046] [1366238.351920] [1366238.351925] Pid: 27046, comm: apache2 Tainted: G D (2.6.24-24-xen #1) [1366238.351929] EIP: 0061:[<c03285fa>] EFLAGS: 00200286 CPU: 0 [1366238.351938] EIP is at _spin_lock+0xa/0x10 [1366238.351940] EAX: d6002dd8 EBX: d6002d94 ECX: c156a9c0 EDX: 00000000 [1366238.351943] ESI: c156a9c0 EDI: 00000000 EBP: d4a4bdd8 ESP: d4a4bc3c [1366238.351946] DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0069 [1366238.351954] CR0: 80050033 CR2: 08e080c8 CR3: 04d26000 CR4: 00002620 [1366238.351958] DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000 [1366238.351961] DR6: ffff0ff0 DR7: 00000400 [1366238.351965] [<c01a6ebd>] try_to_free_buffers+0x2d/0x90 [1366238.351978] [<c0167c75>] shrink_page_list+0x4c5/0x600 [1366238.351985] [<c0166e0f>] isolate_lru_pages+0x5f/0x1c0 [1366238.351991] [<c0167ecf>] shrink_inactive_list+0x11f/0x3b0 [1366238.351998] [<c01681fc>] shrink_zone+0x9c/0x100 [1366238.352002] [<c0168d42>] try_to_free_pages+0x152/0x250 [1366238.352008] [<c0162f9b>] __alloc_pages+0x14b/0x390 [1366238.352013] [<c011959c>] kmap_atomic+0x1c/0x30 [1366238.352020] [<c01713dc>] handle_mm_fault+0x8ac/0x1360 [1366238.352025] [<c0164cf2>] test_clear_page_writeback+0x72/0x100 [1366238.352030] [<c0275361>] __add_entropy_words+0x61/0x1f0 [1366238.352037] [<c011e34e>] __wake_up+0x3e/0x60 [1366238.352042] [<c027571b>] add_timer_randomness+0xdb/0x190 [1366238.352047] [<c032a18e>] do_page_fault+0x35e/0xe70 [1366238.352052] [<c01581d1>] handle_IRQ_event+0x41/0x80 [1366238.352056] [<c0159ad6>] handle_level_irq+0xb6/0xf0 [1366238.352061] [<c0254290>] evtchn_do_upcall+0xc0/0x1a0 [1366238.352067] [<c0329e30>] do_page_fault+0x0/0xe70 [1366238.352071] [<c0328ad5>] error_code+0x35/0x40 [1366238.352077] ======================= I've increased the swapfile size to 1GB as I noticed we were using it pretty heavily. The VPS runs a couple of websites, backed by MySQL. Any ideas? Alex This email carries a disclaimer, a copy of which may be read at http://learning.longhill.org.uk/disclaimer

Show replies by thread

Conrad Wood

1 Feb 1 Feb

2:07 p.m.

Hi, I'd say apache pid 27046 is waiting for IO after it requested more memory and the kernel trying to free some by utilising swap. After 11 seconds (of doing IO) it complained. _If_ my analysis is correct, you'd have the following Options: * add more ram * reduce amount of ram required * increase IO Speed (new kernel/drivers/etc..) I wouldn't expect the increase in swapspace to solve it. *Perhaps* lessen the frequency of this to happen though. Conrad On Mon, 2010-02-01 at 08:11 +0000, Alex Harrington wrote:

...

Hugo Mills

2:18 p.m.

On Mon, Feb 01, 2010 at 03:07:26PM +0100, Conrad Wood wrote:

...

I'd say apache pid 27046 is waiting for IO after it requested more memory and the kernel trying to free some by utilising swap. After 11 seconds (of doing IO) it complained. _If_ my analysis is correct, you'd have the following Options: * add more ram * reduce amount of ram required * increase IO Speed (new kernel/drivers/etc..)

* Reduce the footprint of Apache to fit in the RAM you've got. If you're using the prefork MPM, the defaults are really quite large for a busy site in limited space. For example, my home site (low traffic, in the smallest Bitfolk VM) uses: <IfModule mpm_prefork_module> StartServers 2 MinSpareServers 1 MaxSpareServers 2 MaxClients 7 MaxRequestsPerChild 0 </IfModule> What happens is, when you get spike in load (from a badly-written spider, or a little light slashdotting), apache spawns more processes to cope. This takes up RAM, and eventually pushes you into swap. Of course, that then slows down the processing by several orders of magnitude, ensuring that it falls even further behind, attempting to spawn even more processes to keep up, and finally the system collapses completely. Bringing down the maximum number of processes that Apache will consider serving so that they all fit in RAM prevents this from happening. You could also consider only loading the Apache modules you're actually using, which saves yet more space. Hugo. -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- Great oxymorons of the world, no. 7: The Simple Truth ---

Andy Smith

2:46 p.m.

Hello, Maybe it is something in userland, but I would be interested to know what happens if you switched to a kernel from Debian Lenny e.g.: http://packages.debian.org/lenny/linux-image-2.6.26-2-xen-686 http://packages.debian.org/lenny/linux-modules-2.6.26-2-xen-686 or an upstream kernel.org.one. Cheers, Andy

Alex Harrington

2:47 p.m.

...

If you're using the prefork MPM, the defaults are really quite large for a busy site in limited space. For example, my home site (low traffic, in the smallest Bitfolk VM) uses:

Hi Hugo Thanks - I've just done some tuning there so fingers crossed. Alex This email carries a disclaimer, a copy of which may be read at http://learning.longhill.org.uk/disclaimer

Paul Tansom

3:02 p.m.

** Hugo Mills <hugo-bf(a)carfax.org.uk> [2010-02-01 14:20]:

...

On Mon, Feb 01, 2010 at 03:07:26PM +0100, Conrad Wood wrote:

* Reduce the footprint of Apache to fit in the RAM you've got.

<snip>

...

What happens is, when you get spike in load (from a badly-written spider, or a little light slashdotting), apache spawns more processes to cope. This takes up RAM, and eventually pushes you into swap. Of course, that then slows down the processing by several orders of magnitude, ensuring that it falls even further behind, attempting to spawn even more processes to keep up, and finally the system collapses completely. Bringing down the maximum number of processes that Apache will consider serving so that they all fit in RAM prevents this from happening.

<snip> ** end quote [Hugo Mills] Interesting thought. I've been experiencing this sort of problem on a regular basis, although not had the time to investigate. I've noted similar errors, but not based on a specific application, and after finding some references to problems with Ubuntu virtualisation with Xen have assumed that when I get round to it I'll be looking for a base OS configuration issue. I've migrated from another company who originally used UML, although they experimented with Xen and seem to be useing KVM now. I believe I was still on the UML setup, since I don't remember being advised of a mirgation. That ran the same services with no problems on the same basic virtual hardware spec. I really need to start investigating now as although I'm monitoring it, it is far from ideal. Being impressed with a 2 day uptime is a sad state of affairs! -- Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/ | 023 9238 0001 ====================================================================== Registered in England | Company No: 4905028 | Registered Office: Crawford House, Hambledon Road, Denmead, Waterlooville, Hants, PO7 6NU

Andrew Gee

3:13 p.m.

On 01/02/10 14:46, Andy Smith wrote:

...

I had this problem a few months back, and a switch to the lenny kernel seemed to solve it. No problems since I've switched.

Paul Tansom

6:17 p.m.

** Andy Smith <andy(a)bitfolk.com> [2010-02-01 14:49]:

...

** end quote [Andy Smith] Is this just a case of downloading and installing with dpkg? Possibly using a custom line in the apt sources to pick up updates as well (obviously avoiding any other packages being pulled in!). Just wondering whether updating the kernel is a little more controlled than other packages, but I may be used to my old UML setup and the even more restricted system I was on before (which one escapes me!) where the kernel was updated as part of the VPS hosting platform. Any pointers or suggestions as to ones to try, or good references on compiling Xen kernels? I can experiment a bit on my VPS to see whether it improves things. -- Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/ | 023 9238 0001 ====================================================================== Registered in England | Company No: 4905028 | Registered Office: Crawford House, Hambledon Road, Denmead, Waterlooville, Hants, PO7 6NU

Andy Smith

7:57 p.m.

Hi Paul, On Mon, Feb 01, 2010 at 06:17:09PM +0000, Paul Tansom wrote:

...

** Andy Smith <andy(a)bitfolk.com> [2010-02-01 14:49]:

** end quote [Andy Smith] Is this just a case of downloading and installing with dpkg?

Yes, dpkg -i on both the above .debs should just install them.

...

Possibly using a custom line in the apt sources to pick up updates as well (obviously avoiding any other packages being pulled in!).

I think this is possible but haven't looked into how.

...

Just wondering whether updating the kernel is a little more controlled than other packages,

Kernel updates in debian/ubuntu are just updated packages with the package pre/post install scripts handling initrd, grub etc. You could for example just subscribe to the Debian security mailing list then you would know when a new kernel .deb was out.

...

but I may be used to my old UML setup and the even more restricted system I was on before (which one escapes me!) where the kernel was updated as part of the VPS hosting platform.

Your previous emails suggest Bytemark's UML setup. I could enforce usage of one kernel on everyone which would make it my problem, but then that's a bit less flexible for everyone else. I am still considering doing something a bit like slicehost where you have to request a kernel upgrade and it happens during a boot/shutdown (don't forget that modules need installing in the filesystem).

...

Any pointers or suggestions as to ones to try, or good references on compiling Xen kernels? I can experiment a bit on my VPS to see whether it improves things.

I have successfully compiled an upstream kernel.org kernel like this: 1. Download kernel source from kernel.org, unpack, cd into it. 2. apt-get install build-essential fakeroot libncurses5-dev kernel-package 3. Obtain a .config from somewhere. You could try your existing kernel's config that you might find somewhere in /boot/config* then do "make oldconfig" to update it, though the difference between something like 2.6.18 and 2.6.32 will be quite long-winded. I've attached one that I have used successfully with 2.6.32.7. 4. fakeroot make-kpkg --arch=i386 --revision=pauls-kernel.1 --initrd kernel_image kernel_headers The "pauls-kernel.1" part will appear in the package name but not in the name of the kernel so if you made another kernel from the same source it would overwrite this one. If you want to make different packages that can co-exist, use --append-to-version=... Avoid using "_" in these strings. You should end up with linux-image and linux-headers .debs in the directory above the current one. 5. Install the .debs, make sure you have an initrd created. If not you can make one with e.g. update-initramfs -v -k 2.6.32.7 -c 6. update-grub and check it appears correctly in the /boot/grub/menu.lst. If using an upstream kernel then avoid clocksource=jiffies in the kernel command line. 7. Go to xen shell, shut down, boot, select appropriate kernel, hope it works. :) Cheers, Andy -- http://bitfolk.com/ -- No-nonsense VPS hosting

Rodrigo Campos

8:27 p.m.

On Mon, Feb 01, 2010 at 07:57:39PM +0000, Andy Smith wrote:

...

Hi Paul, On Mon, Feb 01, 2010 at 06:17:09PM +0000, Paul Tansom wrote:

** Andy Smith <andy(a)bitfolk.com> [2010-02-01 14:49]: Possibly using a custom line in the apt sources to pick up updates as well (obviously avoiding any other packages being pulled in!).

I think this is possible but haven't looked into how.

I think apt-pinning is what you want. Thanks, Rodrigo

Alex Harrington

9:31 p.m.

...

Yes, dpkg -i on both the above .debs should just install them.

Thanks Andy. I've switched to the Debian kernel you suggested. I'll let you know if this comes up again. Regards Alex This email carries a disclaimer, a copy of which may be read at http://learning.longhill.org.uk/disclaimer

Paul Tansom

3 Feb 3 Feb

2:50 p.m.

** Andy Smith <andy(a)bitfolk.com> [2010-02-01 20:00]:

...

On Mon, Feb 01, 2010 at 06:17:09PM +0000, Paul Tansom wrote:

** Andy Smith <andy(a)bitfolk.com> [2010-02-01 14:49]:

Maybe it is something in userland, but I would be interested to know what happens if you switched to a kernel from Debian Lenny e.g.: http://packages.debian.org/lenny/linux-image-2.6.26-2-xen-686 http://packages.debian.org/lenny/linux-modules-2.6.26-2-xen-686 or an upstream kernel.org.one.

** end quote [Andy Smith] Is this just a case of downloading and installing with dpkg?

Yes, dpkg -i on both the above .debs should just install them.

Thankfully quite happy with dpkg, so it may be time to give it a go. Having just popped back up from lower down (will make sense if you continue reading!) are the module packages to install with these, or given that the kernel versions match, are the Ubuntu ones adequate? I'm assuming the latter, but if I use a newer kernel then modules would be required.

...

Possibly using a custom line in the apt sources to pick up updates as well (obviously avoiding any other packages being pulled in!).

I think this is possible but haven't looked into how.

I've got a feeling that in the deb line in sources.list you could specify a particular package to pick from the repository and not take the lot. I can't find the reference now to see if my memory serves correctly, backports.org seems the most likely source!

...

Just wondering whether updating the kernel is a little more controlled than other packages,

Already on that, given I've used Debian for some years now, so will likely use that for now.

...

but I may be used to my old UML setup and the even more restricted system I was on before (which one escapes me!) where the kernel was updated as part of the VPS hosting platform.

Your previous emails suggest Bytemark's UML setup.

I was avoiding mentioning my previous provider, in case I say something less complimentary about them at some point!

...

I could enforce usage of one kernel on everyone which would make it my problem, but then that's a bit less flexible for everyone else. I am still considering doing something a bit like slicehost where you have to request a kernel upgrade and it happens during a boot/shutdown (don't forget that modules need installing in the filesystem).

Personally I have no issue with being responsible, so long as I have some guidelines to work to regarding the underlying VM requirements :) On which note I'll head back up this reply with a comment!

...

Any pointers or suggestions as to ones to try, or good references on compiling Xen kernels? I can experiment a bit on my VPS to see whether it improves things.

I have successfully compiled an upstream kernel.org kernel like this:

<snip> Thanks, it is a while since I did much kernel compilation, in fact I'm not 100% sure I've bothered since I moved to Debian. Back on Red Hat it was, from my first Linux install, a requirement so that I could use the Token Ring NIC drivers! Those were the days! ** end quote [Andy Smith] -- Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/ | 023 9238 0001 ====================================================================== Registered in England | Company No: 4905028 | Registered Office: Crawford House, Hambledon Road, Denmead, Waterlooville, Hants, PO7 6NU

Paul Tansom

2:53 p.m.

** Rodrigo Campos <rodrigo(a)sdfg.com.ar> [2010-02-01 20:33]:

...

On Mon, Feb 01, 2010 at 07:57:39PM +0000, Andy Smith wrote:

On Mon, Feb 01, 2010 at 06:17:09PM +0000, Paul Tansom wrote:

** Andy Smith <andy(a)bitfolk.com> [2010-02-01 14:49]: Possibly using a custom line in the apt sources to pick up updates as well (obviously avoiding any other packages being pulled in!).

I think this is possible but haven't looked into how.

I think apt-pinning is what you want.

** end quote [Rodrigo Campos] Doesn't pinning hold a specific package at a defined version? I was thinking more of the issue of duplicate packages being available in the Debian sources as well as the Ubuntu ones, and therefore clashing. -- Paul Tansom | Aptanet Ltd. | http://www.aptanet.com/ | 023 9238 0001 ====================================================================== Registered in England | Company No: 4905028 | Registered Office: Crawford House, Hambledon Road, Denmead, Waterlooville, Hants, PO7 6NU

Hugo Mills

3:31 p.m.

On Wed, Feb 03, 2010 at 02:53:25PM +0000, Paul Tansom wrote:

...

** Rodrigo Campos <rodrigo(a)sdfg.com.ar> [2010-02-01 20:33]:

On Mon, Feb 01, 2010 at 07:57:39PM +0000, Andy Smith wrote:

On Mon, Feb 01, 2010 at 06:17:09PM +0000, Paul Tansom wrote:

** Andy Smith <andy(a)bitfolk.com> [2010-02-01 14:49]: Possibly using a custom line in the apt sources to pick up updates as well (obviously avoiding any other packages being pulled in!).

I think this is possible but haven't looked into how.

I think apt-pinning is what you want.

I think more generally it holds a package to the version in a specific repository (rather than the latest available over all repositories). Hugo. -- === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk === PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk --- This chap Anon is writing some perfectly lovely stuff --- at the moment.

5215

days inactive

5217

days old

users@mailman.bitfolk.com

Manage subscription

13 comments

7 participants

tags (0)

participants (7)

Alex Harrington
Andrew Gee
Andy Smith
Conrad Wood
Hugo Mills
Paul Tansom
Rodrigo Campos