27 Mar
2011
27 Mar
'11
1:01 a.m.
Hi everyone
My VPS running Debian Lenny has recently started to crash frequently,
about 6 times in the last week. I've had out of memory conditions in the
past, though reasonably infrequently, so that was my first I thought.
I've bumped my RAM allocation, added extra swap and removed a few
services which weren't strictly necessary but the crashing continues. My
resource graphs don't suggest any memory usage build up before the crash
or anything else untoward going on and the system logs report nothing,
it seems to just go more or less instantaneously.
I managed to capture some of the console output when I got to the machine:
http://pastebin.com/TjZRCebQ
But it had been dead for about 6 hours as I'd been out for the evening,
so it's possible the crucial part was lost before I got to it.
I'm not much of an expert on interpreting kernel panics so I thought I'd
ask if anybody has any greater insight on the cause or how I can capture
the output to disk.
At this point, my only options seem to be to migrate services off to
another system until the system stops crashing, or add more RAM, or
start afresh on a new VPS to see if it goes away.
Regards,
Adam Sweet
--
http://blog.adamsweet.org/
27 Mar
27 Mar
8:54 a.m.
On 27 Mar 2011, at 02:01, Adam Sweet <adam(a)adamsweet.org> wrote:
My VPS running Debian Lenny has recently started to
crash frequently,
about 6 times in the last week. I've had out of memory conditions in the
past, though reasonably infrequently, so that was my first I thought.
Are you running apache2, wordpress 3.x and php 3.5.x?
My vps would OOM frequently, and now the ubunutu-uk vps is doing the same. I could not
identify the exact cause, but switching from apache to lighttpd has stopped it happening
completely.
Al.
10:32 a.m.
Hi Adam,
On Sun, Mar 27, 2011 at 02:01:10AM +0100, Adam Sweet wrote:
Hi everyone
My VPS running Debian Lenny has recently started to crash frequently,
about 6 times in the last week. I've had out of memory conditions in the
past, though reasonably infrequently, so that was my first I thought.
Any ideas when it started happening?
I've bumped my RAM allocation, added extra swap
and removed a few
services which weren't strictly necessary but the crashing continues. My
resource graphs don't suggest any memory usage build up before the crash
or anything else untoward going on and the system logs report nothing,
it seems to just go more or less instantaneously.
I don't think it will be RAM exhaustion then, sadly.
I managed to capture some of the console output when I
got to the machine:
http://pastebin.com/TjZRCebQ
But it had been dead for about 6 hours as I'd been out for the evening,
so it's possible the crucial part was lost before I got to it.
Sorry, this is a new one to me. Is it always apache2 task that gets
blocked?
As you might imagine, I'm running quite a few Debian Lenny VMs under
Xen at BitFolk for infrastructure purposes and I'm not seeing this
myself.
What are your kernel boot arguments?
At this point, my only options seem to be to migrate
services off to
another system until the system stops crashing, or add more RAM, or
start afresh on a new VPS to see if it goes away.
We could try moving your VPS to another host. Probably worth trying
that before migrating/reinstalling.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
8:44 p.m.
Alan Pope said:
Are you running apache2, wordpress 3.x and php 3.5.x?
My vps would OOM frequently, and now the ubunutu-uk vps is doing the same. I could not
identify the exact cause, but switching from apache to lighttpd has stopped it happening
completely.
It's possible it was the apache settings to blame, but I don't seem to
have problems running those three on Lenny.
Ian
28 Mar
28 Mar
8:39 a.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 27/03/11 09:54, Alan Pope wrote:
On 27 Mar 2011, at 02:01, Adam Sweet <adam(a)adamsweet.org> wrote:
Yes, apart from PHP 5.2.6 with Suhosin. Either way, it might be worth me
investigating whether lighttpd can do everything I need.
Regards,
Adam Sweet
- --
http://blog.adamsweet.org/
My VPS running Debian Lenny has recently started
to crash frequently,
about 6 times in the last week. I've had out of memory conditions in the
past, though reasonably infrequently, so that was my first I thought.
Are you running apache2, wordpress 3.x and php 3.5.x?
My vps would OOM frequently, and now the ubunutu-uk vps is doing the same. I could not
identify the exact cause, but switching from apache to lighttpd has stopped it happening
completely. -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk2QSTIACgkQRi1ZcmvD37dmqQCcDRoB5Kh51tyqRf3IfEBJ0xPq
dB4An3ng+v9uhiD7xmVu9XAck1eXgylH
=K3iW
-----END PGP SIGNATURE-----
9:05 a.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 27/03/11 11:32, Andy Smith wrote:
Yes, always. On previous occurrences I did see that the OOM handler had
killed apache2 and mysqld, but I didn't expect the kernel to panic and
the system to become inoperable in this way.
I've driven my Apache MaxClient down quite hard over time, to around 10
so I'm surprised that having added way more swap and RAM than
MaxClients*Apache process size that I've still seen another crash,
though admittedly the OOM wasn't in the output I was able to catch.
I'd be happy for you to try that, I'd prefer to arrange a time out of
office hours so I can let people know that their mail won't work during
the move.
Regards,
Adam Sweet
- --
http://blog.adamsweet.org/
Hi Adam,
On Sun, Mar 27, 2011 at 02:01:10AM +0100, Adam Sweet wrote:
Hmm, about 19th March when it started crashing every 12-36 hours, 3rd
March is the next previous I have a record of but there have been
intermittent crashes for a while which I put down to running too much in
too little (ie ClamAV).
Hi everyone
My VPS running Debian Lenny has recently started to crash frequently,
about 6 times in the last week. I've had out of memory conditions in the
past, though reasonably infrequently, so that was my first I thought.
Any ideas when it started happening? I've
bumped my RAM allocation, added extra swap and removed a few
services which weren't strictly necessary but the crashing continues. My
resource graphs don't suggest any memory usage build up before the crash
or anything else untoward going on and the system logs report nothing,
it seems to just go more or less instantaneously.
I don't think it will be RAM exhaustion then, sadly.
I managed to capture some of the console output
when I got to the machine:
http://pastebin.com/TjZRCebQ
But it had been dead for about 6 hours as I'd been out for the evening,
so it's possible the crucial part was lost before I got to it.
Sorry, this is a new one to me. Is it always apache2 task that gets
blocked? As you might imagine, I'm running quite a few
Debian Lenny VMs under
Xen at BitFolk for infrastructure purposes and I'm not seeing this
myself.
What are your kernel boot arguments?
kernel /boot/vmlinuz-2.6.26-2-xen-686 root=/dev/xvda1 console=hvc0 ro
At this point,
my only options seem to be to migrate services off to
another system until the system stops crashing, or add more RAM, or
start afresh on a new VPS to see if it goes away.
We could try moving your VPS to another host. Probably worth trying
that before migrating/reinstalling. -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk2QT2cACgkQRi1ZcmvD37dJYACeP+voa1TvIBUySTZNs1MO1AgE
g+0Ani4s47P92ig9kSia4d+3Fnuqc+eQ
=+VIj
-----END PGP SIGNATURE-----
9:05 a.m.
On 27 March 2011 21:44, Ian <ian(a)lovingboth.com> wrote:
It's possible it was the apache settings to blame,
but I don't seem to have
problems running those three on Lenny.
Indeed, it's possible. I did tweak the settings and ramp apache down
very low. It would still eat hundreds of MB per process, which on a
machine with ~400MB RAM is an untenable situation.
Al.
10:25 a.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 28/03/11 10:05, Alan Pope wrote:
On 27 March 2011 21:44, Ian <ian(a)lovingboth.com>
wrote:
That matches my experience and assumption. The Apache process size for
my blog (which is the busiest and heaviest site I run) is:
VSZ RSS
37408 18344
with a MaxClients setting of 10, though it was 15 until recently. I also
run MySQL, Exim and Courier IMAP and I'm currently hovering around 140MB
RAM used out of 480MB + 1GB swap.
I'm currently watching htop on my VPS, in the hope that I might catch a
sudden resource spike in the event of a crash but I imagine it either
won't crash while I'm paying attention or it will be dead before I
notice :-/
Regards,
Adam Sweet
- --
http://blog.adamsweet.org/
It's possible it was the apache settings to
blame, but I don't seem to have
problems running those three on Lenny.
Indeed, it's possible. I did tweak the settings and ramp apache down
very low. It would still eat hundreds of MB per process, which on a
machine with ~400MB RAM is an untenable situation. -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk2QYg0ACgkQRi1ZcmvD37eTtQCgtJ02uc9cpbIHGnxifWwOXg6w
+50An3KEEIQVaPhOKyINgQEYfY1kyst2
=D57G
-----END PGP SIGNATURE-----
11:10 a.m.
Hi Adam,
On Mon, Mar 28, 2011 at 10:05:43AM +0100, Adam Sweet wrote:
I don't expect it to fix what you're seeing, but on Lenny I would
also recommend clocksource=jiffies. (not for pvops kernels like in
squeeze though)
$ cat /sys/devices/system/clocksource/*/current_clocksource
jiffies
Again I wouldn't expect this to fix anything, but it's worth a try
because it's so easy. The down time would only be on the order of
10-15 minutes. Please send a support ticket if you'd like to try
this, giving some ideas of what sorts of times you'd be OK with it
happening.
Cheers,
Andy
--
"I'm /extremely/ miffed about today's events and in my quest to try to make
you understand the level of my unhappiness, I'm likely to use an awful lot
of what we would call /violent sexual imagery/ and I just wanted to check
that neither of you would be terribly offended by that." -- Malcolm Tucker
On 27/03/11 11:32, Andy Smith wrote:
The kernel hasn't paniced as such. What's happened is a task has been
stuck inside the kernel for 120 seconds or whatever and this
automatically produces the backtrace you saw.
If it's always apache2 then it could actually be apache using all
the memory and going into swap, then being too slow to do anything.
It might be something else though. Or it might not be memory-related
at all.
Sorry, this is a new one to me. Is it always
apache2 task that gets
blocked?
Yes, always. On previous occurrences I did see that the OOM handler had
killed apache2 and mysqld, but I didn't expect the kernel to panic and
the system to become inoperable in this way. What are your
kernel boot arguments?
kernel /boot/vmlinuz-2.6.26-2-xen-686 root=/dev/xvda1 console=hvc0 ro We could try
moving your VPS to another host. Probably worth trying
that before migrating/reinstalling.
I'd be happy for you to try that, I'd prefer to arrange a time out of
office hours so I can let people know that their mail won't work during
the move. 
30 Mar
30 Mar
8:08 p.m.
On Mon, Mar 28, 2011 at 11:10:59AM +0000, Andy Smith wrote:
If it's always apache2 then it could actually be
apache using all
the memory and going into swap, then being too slow to do anything.
It might be something else though. Or it might not be memory-related
at all.
If it keeps happening maybe you could put Apache in a memory cgroup,
that way if it begins to use too much memory it'll be contained and the
VPS will be fully usable. It will also prevent the OOM killer touch
processes not in that cgroup, and give you a decent environment to debug
the issue.
We've been doing that in our VPS for about a week and it's been working
great for us.
Thanks,
Alberto
4 Apr
4 Apr
9:04 a.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 30/03/11 21:08, Alberto Bertogli wrote:
On Mon, Mar 28, 2011 at 11:10:59AM +0000, Andy Smith
wrote:
That's a nice idea, I'd not heard of that before, I'm going to look into
that.
For some closure, I should say that I've not had a crash in the week
since my original post, having driven the Apache MaxClients down to 10,
which is kind of annoying :-/
It seems incredible that you can't serve more than 10 simultaneous
clients on ~500MB RAM so perhaps it is time to consider an alternative
HTTP server.
Regards,
Adam Sweet
- --
http://blog.adamsweet.org/
If it's always apache2 then it could actually
be apache using all
the memory and going into swap, then being too slow to do anything.
It might be something else though. Or it might not be memory-related
at all.
If it keeps happening maybe you could put Apache in a memory cgroup,
that way if it begins to use too much memory it'll be contained and the
VPS will be fully usable. It will also prevent the OOM killer touch
processes not in that cgroup, and give you a decent environment to debug
the issue.
We've been doing that in our VPS for about a week and it's been working
great for us. -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk2ZiZUACgkQRi1ZcmvD37c+oQCfTG+QiHL45DaG0bbtjqVlS41C
DTYAn2B2Ghvplfgh5/C6qyuimtrHBKxN
=iDDq
-----END PGP SIGNATURE-----
1:59 p.m.
Hi Adam,
On Mon, Apr 04, 2011 at 10:04:21AM +0100, Adam Sweet wrote:
For some closure, I should say that I've not had a
crash in the week
since my original post, having driven the Apache MaxClients down to 10,
which is kind of annoying :-/
It seems incredible that you can't serve more than 10 simultaneous
clients on ~500MB RAM so perhaps it is time to consider an alternative
HTTP server.
How big is each apache process right now?
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
5:29 p.m.
On Mon, Apr 04, 2011 at 10:04:21AM +0100, Adam Sweet wrote:
On 30/03/11 21:08, Alberto Bertogli wrote:
It's possible that you may be able to put a proxy in front of your heavy
backend Apache, split the processing up a little so dirs that require
more dynamic work get off loaded to a heavyweight Apache, with lower
concurrent "listeners", whilst your static content gets handled by the
lighter Apache, which has more "listeners".
By the way, LugRadio was perhaps one of the best things to ever happen
in the UK. You provided many, many hours of entertainment to me. Huge
respect.
--
Best regards,
Ed http://www.s5h.net/
On Mon, Mar 28, 2011 at 11:10:59AM +0000, Andy
Smith wrote:
That's a nice idea, I'd not heard of that before, I'm going to look into
that.
For some closure, I should say that I've not had a crash in the week
since my original post, having driven the Apache MaxClients down to 10,
which is kind of annoying :-/
It seems incredible that you can't serve more than 10 simultaneous
clients on ~500MB RAM so perhaps it is time to consider an alternative
HTTP server. If it's always apache2 then it could actually
be apache using all
the memory and going into swap, then being too slow to do anything.
It might be something else though. Or it might not be memory-related
at all.
If it keeps happening maybe you could put Apache in a memory cgroup,
that way if it begins to use too much memory it'll be contained and the
VPS will be fully usable. It will also prevent the OOM killer touch
processes not in that cgroup, and give you a decent environment to debug
the issue.
We've been doing that in our VPS for about a week and it's been working
great for us.
8 Apr
8 Apr
11:14 a.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 04/04/11 14:59, Andy Smith wrote:
Hi Adam,
On Mon, Apr 04, 2011 at 10:04:21AM +0100, Adam Sweet wrote:
A process that has actually served some data is around:
VSZ RSS
37324 18304
Regards,
Adam Sweet
- --
http://blog.adamsweet.org/
For some closure, I should say that I've not
had a crash in the week
since my original post, having driven the Apache MaxClients down to 10,
which is kind of annoying :-/
It seems incredible that you can't serve more than 10 simultaneous
clients on ~500MB RAM so perhaps it is time to consider an alternative
HTTP server.
How big is each apache process right now? -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk2e7h8ACgkQRi1ZcmvD37dBqwCeIvvkX6bpdEayK9YEnAdeFsr1
qO0AoJIkOTP+QZJKcznYV9g2VcMU3Nwc
=7ipf
-----END PGP SIGNATURE-----
11:35 a.m.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On 04/04/11 18:29, ed wrote:
I'll look into that. It's been a long time since I took any care of PHP
and Apache's performance tuning so it will probably pay to put them on a
diet. The crashing seems to have gone away for now so at least I've
bought myself some time to investigate the solutions recommended here.
On Mon, Apr 04, 2011 at 10:04:21AM +0100, Adam Sweet
wrote:
It seems
incredible that you can't serve more than 10 simultaneous
clients on ~500MB RAM so perhaps it is time to consider an alternative
HTTP server.
It's possible that you may be able to put a proxy in front of your heavy
backend Apache, split the processing up a little so dirs that require
more dynamic work get off loaded to a heavyweight Apache, with lower
concurrent "listeners", whilst your static content gets handled by the
lighter Apache, which has more "listeners". By the way, LugRadio was perhaps one of the best
things to ever happen
in the UK. You provided many, many hours of entertainment to me. Huge
respect.
Hahaha, well I'm not sure I can take too much credit for that dubious
honour :)
Thanks for the advice everyone.
Regards,
Adam Sweet
- --
http://blog.adamsweet.org/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAk2e8usACgkQRi1ZcmvD37c68QCdF0AtkxlA9aixJ6FJszcBjjEC
n/8An0xQdQ/PWkW0SQ6fv3PVq28LtkEj
=NL6R
-----END PGP SIGNATURE-----
4769
days inactive
4781
days old
14 comments
6 participants
participants (6)
-
Adam Sweet -
Alan Pope -
Alberto Bertogli -
Andy Smith -
ed -
Ian