Every morning I read through the logwatch reports from my 2 VPS and do any adjustments that the reports seem to suggest. This morning though there was a very long report which seems serious but which I just do not understand. As this is from the VPS I use for the big charity web site I host, I am quite concerned. I have pasted the relevant part below and apologise that it is so long. --------------------- Kernel Begin ------------------------ WARNING: Kernel Errors Present [<c02cc64d>] error_code+0x35/0x3c ...: 4 Time(s) 5 Time(s): 00000000 c0133308 28e31a64 d1464f0c 28e31a64 c14f81a8 c1237b40 00e78000 4 Time(s): 00000006 ca626005 00003939 d2a57e50 d142b038 c012eeab d142b038 d142b000 4 Time(s): c0116099 d282e488 0000c6c6 d14dff70 d142b074 c012eeab d142b000 d142b014 4 Time(s): c018d837 c0133308 28e3200a c3c002dc 28e3200a c14f9054 c1237b40 00e78000 4 Time(s): c01d2a24 c0133308 28e3255e d14643cc 28e3255e c14f9768 c1237b40 00e78000 4 Time(s): c5ab0ac0 00000001 c5ab0ac0 0003c999 00000000 00000005 005b756f b7465000 4 Time(s): c5ab0ac0 c0102eb7 aad1decd c035a1a0 c035a3ec d0f36c0c 00000000 001b8003 4 Time(s): c7038940 00000282 d66fc523 d24427c8 d24fce48 c7038ac8 c1237b40 00000000 4 Time(s): ca626013 00000310 00003a3a ca635e78 d142b038 c012eeab d142b038 d142b000 4 Time(s): ca646a00 00000286 c0117946 00000003 00000001 ca646b88 c1237b40 00000000 4 Time(s): cba89580 d26fb2c0 00003b3b ca6d5e00 d142b038 c012eeab d142b038 d142b000 4 Time(s): cce68040 80000005 00000001 0002bbf6 00000000 00000005 005bd0f2 001c000d 5 Time(s): cd87c040 d1a560c0 c1237b40 00003290 00000000 00000005 005bd0cd 001b8003 2 Time(s): cd87cc80 d14c1000 aad2e620 c0178c86 d28110c0 d24cc5b8 d2826c40 ca4c9f14 4 Time(s): d0f369c0 00000246 00000000 c01d2525 01000000 d0f36b48 c1237b40 00000000 5 Time(s): d0f37200 00000286 0028b327 001bb000 00000000 d0f37388 c1237b40 00000000 2 Time(s): d0f37a40 00000282 d14c1013 c0183b45 ca4c9f14 d0f37bc8 c1237b40 00000000 4 Time(s): d10182c0 00000286 d28a9b2c 51705f65 0028ba00 d1018448 c1237b40 00000000 4 Time(s): d147b4e0 00000286 d403f170 31d6d75a 0028b9e9 d147b668 c1237b40 00000000 4 Time(s): d15203c0 0000033f d14eb248 0000bc01 00000000 00000005 005b5ce2 d14eb248 4 Time(s): d1520900 a9d3de3b 0028b9dd 00000c66 00000000 d288b588 c0116199 c1237b74 4 Time(s): d1520900 d253eb1c 00000065 0001642c 00000000 00000005 005bced5 00000000 4 Time(s): d15466a0 00000286 00000065 2793d7b4 0028b9e9 d1546828 c1237b40 00000000 4 Time(s): d18a8200 ca747ff8 aad1ceb7 00000003 ca7f6294 d1868240 0000000a fffffff4 5 Time(s): d18e3180 00200282 00000000 018cd819 0028b9e2 d18e3308 c1237b40 00000000 4 Time(s): d1a74040 016f084c aad1c0e0 f97e34b5 ca635f14 c76364ac c0178b58 ca626021 5 Time(s): d1a743c0 d8f3f9d2 aad1a2c5 ca71fd3c d8f3f9d2 0028bacb 00000000 d1127200 4 Time(s): d1bcf1c0 c016d34b 00003c3c ca747e00 d142b038 c012eeab d142b038 d142b000 4 Time(s): d2353200 00200282 c5e0ba14 1f80d62d 0028b9e5 d2353388 c1237b40 00000000 5 Time(s): d2402d98 c0133308 28e31a64 c3c00d5c 28e31a64 c14f81a8 c1237b40 00e78000 4 Time(s): d2402d98 c0133308 28e31ba2 cacb836c 28e31ba2 c14f9228 c1237b40 00e78000 4 Time(s): d288b560 00000246 0028bb0d 2793bc59 0028b9e9 d288b6e8 c1237b40 00000000 2 Time(s): ffffff9c d2826c44 c0178f05 d14c1000 ffffffe9 00000310 c7636524 c763652c 44 Time(s): ======================= 4 Time(s): [<c0102eb7>] __switch_to+0x376/0x42e 36 Time(s): [<c0103f76>] syscall_call+0x7/0xb 8 Time(s): [<c010425f>] kernel_thread_helper+0x7/0x10 4 Time(s): [<c0105f74>] get_nsec_offset+0xe/0x6a 4 Time(s): [<c0106149>] get_runstate_snapshot+0x69/0xec 4 Time(s): [<c01061e1>] sched_clock+0x15/0x47 4 Time(s): [<c010e4e6>] do_page_fault+0x0/0xb2e 14 Time(s): [<c010eaeb>] do_page_fault+0x605/0xb2e 10 Time(s): [<c010eb51>] do_page_fault+0x66b/0xb2e 4 Time(s): [<c0116099>] __dequeue_entity+0x1f/0x71 4 Time(s): [<c0116199>] dequeue_entity+0x13/0x9b 4 Time(s): [<c0117946>] try_to_wake_up+0xe8/0xf1 4 Time(s): [<c011b492>] dequeue_task_fair+0x13/0x27 4 Time(s): [<c011f4be>] do_syslog+0x161/0x3e5 5 Time(s): [<c012659c>] process_timeout+0x0/0x5 4 Time(s): [<c01268ed>] try_to_del_timer_sync+0x44/0x4a 8 Time(s): [<c012eca1>] kthread+0x0/0x5f 8 Time(s): [<c012ecd9>] kthread+0x38/0x5f 50 Time(s): [<c012ed9c>] autoremove_wake_function+0x0/0x2d 22 Time(s): [<c012edc9>] wake_bit_function+0x0/0x3c 20 Time(s): [<c012eeab>] prepare_to_wait+0x12/0x4d 22 Time(s): [<c0133308>] getnstimeofday+0x37/0xbc 8 Time(s): [<c014e5b6>] __lock_page+0x4e/0x54 8 Time(s): [<c014e5c4>] sync_page+0x0/0x36 8 Time(s): [<c014e5f7>] sync_page+0x33/0x36 4 Time(s): [<c014ed2f>] __filemap_fdatawrite_range+0x65/0x70 4 Time(s): [<c014ef27>] filemap_fdatawrite+0x12/0x16 12 Time(s): [<c01503f7>] generic_file_aio_read+0x49a/0x4e1 8 Time(s): [<c0153775>] __writepage+0x0/0x21 4 Time(s): [<c015377d>] __writepage+0x8/0x21 4 Time(s): [<c0153bfb>] write_cache_pages+0xff/0x27b 4 Time(s): [<c0153c66>] write_cache_pages+0x16a/0x27b 8 Time(s): [<c0153d91>] generic_writepages+0x1a/0x21 8 Time(s): [<c0153dc1>] do_writepages+0x29/0x30 4 Time(s): [<c015468b>] wb_kupdate+0x0/0xdb 4 Time(s): [<c0154706>] wb_kupdate+0x7b/0xdb 4 Time(s): [<c0154a69>] pdflush+0x0/0x253 4 Time(s): [<c0154bf2>] pdflush+0x189/0x253 4 Time(s): [<c015ee5b>] handle_mm_fault+0x8e8/0xd9e 4 Time(s): [<c016d34b>] cache_alloc_refill+0x57/0x52a 2 Time(s): [<c016e9b5>] get_unused_fd_flags+0x4f/0xd8 6 Time(s): [<c016ea7e>] do_sys_open+0x40/0xb0 6 Time(s): [<c016eb32>] sys_open+0x1e/0x23 4 Time(s): [<c016fdc2>] rw_copy_check_uvector+0x5c/0xb0 5 Time(s): [<c016ff3f>] do_sync_write+0x0/0x100 5 Time(s): [<c016fffe>] do_sync_write+0xbf/0x100 17 Time(s): [<c017003f>] do_sync_read+0x0/0xfe 17 Time(s): [<c01700fe>] do_sync_read+0xbf/0xfe 5 Time(s): [<c0170770>] vfs_write+0x83/0x120 21 Time(s): [<c017088e>] vfs_read+0x81/0x11e 21 Time(s): [<c0170cdf>] sys_read+0x3c/0x63 5 Time(s): [<c0170d42>] sys_write+0x3c/0x63 13 Time(s): [<c0172f1c>] cp_new_stat64+0xfc/0x10e 5 Time(s): [<c0173497>] sys_lstat64+0x1e/0x23 5 Time(s): [<c0176180>] pipe_write+0x379/0x3c7 5 Time(s): [<c0176519>] pipe_read+0x31d/0x32c 4 Time(s): [<c0177491>] vfs_create+0xb4/0x117 4 Time(s): [<c0178b58>] __link_path_walk+0xb2e/0xbf5 2 Time(s): [<c0178c86>] path_walk+0x67/0x70 2 Time(s): [<c0178f05>] do_path_lookup+0x122/0x184 2 Time(s): [<c0179a6f>] do_filp_open+0x105/0x684 4 Time(s): [<c0179b17>] do_filp_open+0x1ad/0x684 5 Time(s): [<c0181c88>] file_update_time+0x99/0xc2 17 Time(s): [<c018200d>] touch_atime+0xc7/0xce 2 Time(s): [<c0183b45>] mntput_no_expire+0x13/0x120 4 Time(s): [<c018a139>] __writeback_single_inode+0x160/0x28a 4 Time(s): [<c018a563>] sync_sb_inodes+0x17c/0x24d 22 Time(s): [<c018a7ed>] __mark_inode_dirty+0x21/0x148 4 Time(s): [<c018a988>] writeback_inodes+0x74/0xf9 4 Time(s): [<c018cc6e>] do_fsync+0x26/0x83 4 Time(s): [<c018cce8>] __do_fsync+0x1d/0x2b 4 Time(s): [<c018d837>] submit_bh+0xe0/0xed 28 Time(s): [<c018e4da>] sync_buffer+0x0/0x33 14 Time(s): [<c018e50a>] sync_buffer+0x30/0x33 14 Time(s): [<c018e6c7>] __lock_buffer+0x21/0x24 4 Time(s): [<c01a00c8>] proc_reg_read+0x0/0xa8 4 Time(s): [<c01a0158>] proc_reg_read+0x90/0xa8 4 Time(s): [<c01a721a>] kmsg_read+0x0/0x36 22 Time(s): [<c01b96dd>] security_file_permission+0xc/0xd 4 Time(s): [<c01d2525>] submit_bio+0xc6/0xcd 4 Time(s): [<c01d2a24>] generic_unplug_device+0x1e/0x5a 10 Time(s): [<c02c0000>] setup_cpu_cache+0x60/0x11f 8 Time(s): [<c02cacf5>] schedule+0x631/0x6b1 22 Time(s): [<c02caebc>] io_schedule+0x49/0x80 22 Time(s): [<c02cafe8>] __wait_on_bit_lock+0x2a/0x52 14 Time(s): [<c02cb06f>] out_of_line_wait_on_bit_lock+0x5f/0x67 2 Time(s): [<c02cb140>] mutex_lock+0xa/0xb 2 Time(s): [<c02cb314>] __mutex_lock_slowpath+0x67/0xa9 4 Time(s): [<d403f170>] __journal_file_buffer+0xa2/0x123 [jbd] 14 Time(s): [<d403f536>] do_get_write_access+0x4c/0x367 [jbd] 14 Time(s): [<d403f869>] journal_get_write_access+0x18/0x26 [jbd] 16 Time(s): [<d403faab>] start_this_handle+0x234/0x397 [jbd] 16 Time(s): [<d403fc8b>] journal_start+0x7d/0xa9 [jbd] 4 Time(s): [<d4040c4b>] journal_commit_transaction+0xdd/0xcae [jbd] 4 Time(s): [<d4043f98>] kjournald+0x0/0x225 [jbd] 4 Time(s): [<d4044054>] kjournald+0xbc/0x225 [jbd] 14 Time(s): [<d40735e7>] __ext3_get_inode_loc+0xcf/0x26c [ext3] 14 Time(s): [<d40737c2>] ext3_reserve_inode_write+0x2d/0x5d [ext3] 14 Time(s): [<d4073803>] ext3_mark_inode_dirty+0x11/0x27 [ext3] 4 Time(s): [<d40758cd>] ext3_ordered_writepage+0x43/0x13d [ext3] 8 Time(s): [<d4075f85>] ext3_dirty_inode+0x21/0x63 [ext3] 14 Time(s): [<d4075fb4>] ext3_dirty_inode+0x50/0x63 [ext3] 4 Time(s): [<d40788a9>] ext3_create+0x41/0xdc [ext3] 14 Time(s): [<d407e5bb>] __ext3_journal_get_write_access+0x13/0x32 [ext3] 44 Time(s): "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. 44 Time(s): Call Trace: 4 Time(s): INFO: task fail2ban-server:2408 blocked for more than 120 seconds. 4 Time(s): INFO: task kjournald:502 blocked for more than 120 seconds. 4 Time(s): INFO: task klogd:1282 blocked for more than 120 seconds. 5 Time(s): INFO: task master:6589 blocked for more than 120 seconds. 4 Time(s): INFO: task miniserv.pl:1569 blocked for more than 120 seconds. 4 Time(s): INFO: task monitor.pl:19711 blocked for more than 120 seconds. 2 Time(s): INFO: task monitor.pl:19727 blocked for more than 120 seconds. 4 Time(s): INFO: task pdflush:6591 blocked for more than 120 seconds. 5 Time(s): INFO: task qmgr:6592 blocked for more than 120 seconds. 4 Time(s): INFO: task syslogd:1273 blocked for more than 120 seconds. 4 Time(s): INFO: task tlsmgr:8187 blocked for more than 120 seconds. 4 Time(s): fail2ban-serv D 31d6d75a 0 2408 1 4 Time(s): kjournald D 2793bc59 0 502 2 4 Time(s): klogd D 51705f65 0 1282 1 5 Time(s): master D 018cd819 0 6589 1 4 Time(s): miniserv.pl D 1f80d62d 0 1569 1 2 Time(s): monitor.pl D c0183b45 0 19727 19726 4 Time(s): monitor.pl D d24427c8 0 19711 19710 4 Time(s): pdflush D c01d2525 0 6591 2 5 Time(s): qmgr D 001bb000 0 6592 6589 4 Time(s): syslogd D 2793d7b4 0 1273 1 4 Time(s): tlsmgr D 00000003 0 8187 6589 ---------------------- Kernel End ------------------------- I am running Debian Lenny. All software is the latest versions that are held in the repositories (I run automatic apt-get update, apt-get upgrade daily) I am running Apache2, and a small Postfix installation purely for sending reports and replies from the site, fail2ban and webmin. Does this suggest an imminent disaster? Or am I just panicking Many thanks in advance Keith -- If the only tool you have is a hammer, you tend to see every problem as a nail. - Abraham Maslow