Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

My server crashed today, suddenly.

those are the logs from /var/log/messages before it crashed. please help me undestand what is the reason

Apr 10 21:40:37 stephan kernel: audit_log_start: 242 callbacks suppressed
Apr 10 21:40:37 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 21:40:37 stephan kernel: audit: audit_lost=95563 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 21:40:37 stephan kernel: audit: backlog limit exceeded
Apr 10 21:40:37 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 21:40:37 stephan kernel: audit: audit_lost=95564 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 21:40:37 stephan kernel: audit: backlog limit exceeded
Apr 10 21:40:38 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 21:40:38 stephan kernel: audit: audit_lost=95565 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 21:40:38 stephan kernel: audit: backlog limit exceeded
Apr 10 21:40:38 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 21:40:43 stephan kernel: audit_log_start: 230 callbacks suppressed
Apr 10 21:40:43 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 21:40:43 stephan kernel: audit: audit_lost=95643 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 21:40:43 stephan kernel: audit: backlog limit exceeded
Apr 10 21:40:43 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 21:40:43 stephan kernel: audit: audit_lost=95644 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 21:40:43 stephan kernel: audit: backlog limit exceeded
Apr 10 21:40:43 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 21:40:43 stephan kernel: audit: audit_lost=95645 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 21:40:43 stephan kernel: audit: backlog limit exceeded
Apr 10 21:40:43 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 21:40:48 stephan kernel: audit_log_start: 281 callbacks suppressed
Apr 10 21:40:48 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 21:40:48 stephan kernel: audit: audit_lost=95740 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 21:40:48 stephan kernel: audit: backlog limit exceeded
Apr 10 21:40:48 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 21:40:48 stephan kernel: audit: audit_lost=95741 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 21:40:48 stephan kernel: audit: backlog limit exceeded
Apr 10 21:40:48 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 21:40:48 stephan kernel: audit: audit_lost=95742 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 21:40:48 stephan kernel: audit: backlog limit exceeded
Apr 10 21:40:48 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 21:40:53 stephan kernel: audit_log_start: 278 callbacks suppressed
Apr 10 21:40:53 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 21:40:53 stephan kernel: audit: audit_lost=95836 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 21:40:53 stephan kernel: audit: backlog limit exceeded
Apr 10 21:40:53 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 21:40:53 stephan kernel: audit: audit_lost=95837 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 21:40:53 stephan kernel: audit: backlog limit exceeded
Apr 10 21:40:53 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 21:40:53 stephan kernel: audit: audit_lost=95838 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 22:06:24 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:24 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:24 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:24 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:24 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:24 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:24 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:24 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:24 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:24 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:29 stephan kernel: net_ratelimit: 284 callbacks suppressed
Apr 10 22:06:29 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:29 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:29 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:29 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:29 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:29 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:29 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:29 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:29 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:29 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:06:42 stephan kernel: net_ratelimit: 42 callbacks suppressed
Apr 10 22:09:09 stephan auditd[1913]: Audit daemon rotating log files
Apr 10 22:09:13 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:09:13 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:09:13 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:09:13 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:09:13 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:09:13 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:09:13 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:09:13 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:09:14 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:09:14 stephan kernel: TCP: Possible SYN flooding on port 80. Sending cookies.
Apr 10 22:11:44 stephan kernel: net_ratelimit: 11 callbacks suppressed
Apr 10 22:11:44 stephan kernel: Redirect from 217.84.219.203 on bond0 about 87.186.224.62 ignored.
Apr 10 22:11:44 stephan kernel:  Advised path = 82.144.220.146 -> 0.0.0.0
Apr 10 22:15:35 stephan auditd[1913]: Audit daemon rotating log files
Apr 10 22:21:01 stephan auditd[1913]: Audit daemon rotating log files
Apr 10 22:21:49 stephan kernel: audit_log_start: 659 callbacks suppressed
Apr 10 22:21:49 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:21:49 stephan kernel: audit: audit_lost=96599 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 22:21:49 stephan kernel: audit: backlog limit exceeded
Apr 10 22:21:49 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:21:49 stephan kernel: audit: audit_lost=96600 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 22:21:49 stephan kernel: audit: backlog limit exceeded
Apr 10 22:21:49 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:21:49 stephan kernel: audit: audit_lost=96601 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 22:21:49 stephan kernel: audit: backlog limit exceeded
Apr 10 22:21:49 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:21:54 stephan kernel: audit_log_start: 13358 callbacks suppressed
Apr 10 22:21:54 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:21:54 stephan kernel: audit: audit_lost=101055 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 22:21:54 stephan kernel: audit: backlog limit exceeded
Apr 10 22:21:54 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:21:54 stephan kernel: audit: audit_lost=101056 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 22:21:54 stephan kernel: audit: backlog limit exceeded
Apr 10 22:21:54 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:21:54 stephan kernel: audit: audit_lost=101057 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 22:21:54 stephan kernel: audit: backlog limit exceeded
Apr 10 22:21:54 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:21:59 stephan kernel: audit_log_start: 512 callbacks suppressed
Apr 10 22:21:59 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:21:59 stephan kernel: audit: audit_lost=101229 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 22:21:59 stephan kernel: audit: backlog limit exceeded
Apr 10 22:21:59 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:21:59 stephan kernel: audit: audit_lost=101230 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 22:21:59 stephan kernel: audit: backlog limit exceeded
Apr 10 22:21:59 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:21:59 stephan kernel: audit: audit_lost=101231 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 22:21:59 stephan kernel: audit: backlog limit exceeded
Apr 10 22:21:59 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:22:04 stephan kernel: audit_log_start: 371 callbacks suppressed
Apr 10 22:22:04 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:22:04 stephan kernel: audit: audit_lost=101356 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 22:22:04 stephan kernel: audit: backlog limit exceeded
Apr 10 22:22:04 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:22:04 stephan kernel: audit: audit_lost=101357 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 22:22:04 stephan kernel: audit: backlog limit exceeded
Apr 10 22:22:04 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:22:04 stephan kernel: audit: audit_lost=101358 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 22:22:04 stephan kernel: audit: backlog limit exceeded
Apr 10 22:22:04 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:22:09 stephan kernel: audit_log_start: 422 callbacks suppressed
Apr 10 22:22:09 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:22:09 stephan kernel: audit: audit_lost=101500 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 22:22:09 stephan kernel: audit: backlog limit exceeded
Apr 10 22:22:09 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:22:09 stephan kernel: audit: audit_lost=101501 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 22:22:09 stephan kernel: audit: backlog limit exceeded
Apr 10 22:22:09 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:22:09 stephan kernel: audit: audit_lost=101502 audit_rate_limit=0 audit_backlog_limit=5000
Apr 10 22:22:09 stephan kernel: audit: backlog limit exceeded
Apr 10 22:22:09 stephan kernel: audit: audit_backlog=5001 > audit_backlog_limit=5000
Apr 10 22:23:14 stephan kernel: ------------[ cut here ]------------
Apr 10 22:23:14 stephan kernel: WARNING: at lib/list_debug.c:53 __list_del_entry+0xa1/0xd0()
Apr 10 22:23:14 stephan kernel: Hardware name: S5520SC
Apr 10 22:23:14 stephan kernel: list_del corruption. prev->next should be ffff880c798a3000, but was ffff88171f2c47f8
Apr 10 22:23:14 stephan kernel: Modules linked in: bonding iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 xfs exportfs pcspkr i2c_i801 iTCO_wdt iTCO_vendor_support ses enclosure sg ioatdma i7core_edac edac_core snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc igb dca ext4 mbcache jbd2 sd_mod crc_t10dif firewire_ohci firewire_core crc_itu_t ahci libahci mpt2sas scsi_transport_sas raid_class nouveau ttm drm_kms_helper drm i2c_algo_bit i2c_core mxm_wmi wmi video dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Apr 10 22:23:14 stephan kernel: Pid: 66, comm: kswapd1 Not tainted 3.0.0+ #1
Apr 10 22:23:14 stephan kernel: Call Trace:
Apr 10 22:23:14 stephan kernel: [<ffffffff81062b2f>] warn_slowpath_common+0x7f/0xc0
Apr 10 22:23:14 stephan kernel: [<ffffffff81062c26>] warn_slowpath_fmt+0x46/0x50
Apr 10 22:23:14 stephan kernel: [<ffffffff81268c91>] __list_del_entry+0xa1/0xd0
Apr 10 22:23:14 stephan kernel: [<ffffffff81268cd1>] list_del+0x11/0x40
Apr 10 22:23:14 stephan kernel: [<ffffffff8114ba5b>] free_block+0xcb/0x180
Apr 10 22:23:14 stephan kernel: [<ffffffff8114b8e0>] kmem_cache_free+0x290/0x2b0
Apr 10 22:23:14 stephan kernel: [<ffffffff8118b802>] free_buffer_head+0x22/0x40
Apr 10 22:23:14 stephan kernel: [<ffffffff8118bc29>] try_to_free_buffers+0x79/0xc0
Apr 10 22:23:14 stephan kernel: [<ffffffffa03e6ad0>] xfs_vm_releasepage+0x50/0xc0 [xfs]
Apr 10 22:23:14 stephan kernel: [<ffffffff811018f2>] try_to_release_page+0x32/0x60
Apr 10 22:23:14 stephan kernel: [<ffffffff81113b1c>] shrink_page_list+0x4bc/0x5b0
Apr 10 22:23:14 stephan kernel: [<ffffffff811145a2>] shrink_inactive_list+0x172/0x460
Apr 10 22:23:14 stephan kernel: [<ffffffff8105b4e3>] ? perf_event_task_sched_out+0x33/0xa0
Apr 10 22:23:14 stephan kernel: [<ffffffff81112458>] ? get_scan_count+0x1c8/0x2c0
Apr 10 22:23:14 stephan kernel: [<ffffffff81114deb>] shrink_zone+0x20b/0x2c0
Apr 10 22:23:14 stephan kernel: [<ffffffff811162c9>] balance_pgdat+0x6e9/0x8f0
Apr 10 22:23:14 stephan kernel: [<ffffffff81116696>] kswapd+0x1c6/0x210
Apr 10 22:23:14 stephan kernel: [<ffffffff811164d0>] ? balance_pgdat+0x8f0/0x8f0
Apr 10 22:23:14 stephan kernel: [<ffffffff81084d16>] kthread+0x96/0xa0
Apr 10 22:23:14 stephan kernel: [<ffffffff81503084>] kernel_thread_helper+0x4/0x10
Apr 10 22:23:14 stephan kernel: [<ffffffff81084c80>] ? kthread_worker_fn+0x1a0/0x1a0
Apr 10 22:23:14 stephan kernel: [<ffffffff81503080>] ? gs_change+0x13/0x13
Apr 10 22:23:14 stephan kernel: ---[ end trace 3120f0f10d7ed084 ]---
Apr 10 22:28:10 stephan auditd[1913]: Audit daemon rotating log files
Apr 10 22:28:24 stephan kernel: ------------[ cut here ]------------
Apr 10 22:28:24 stephan kernel: WARNING: at lib/list_debug.c:56 __list_del_entry+0x82/0xd0()
Apr 10 22:28:24 stephan kernel: Hardware name: S5520SC
Apr 10 22:28:24 stephan kernel: list_del corruption. next->prev should be ffff88171f2c4000, but was ffff8816f87d0000
Apr 10 22:28:24 stephan kernel: Modules linked in: bonding iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 xfs exportfs pcspkr i2c_i801 iTCO_wdt iTCO_vendor_support ses enclosure sg ioatdma i7core_edac edac_core snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer snd soundcore snd_page_alloc igb dca ext4 mbcache jbd2 sd_mod crc_t10dif firewire_ohci firewire_core crc_itu_t ahci libahci mpt2sas scsi_transport_sas raid_class nouveau ttm drm_kms_helper drm i2c_algo_bit i2c_core mxm_wmi wmi video dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
Apr 10 22:28:24 stephan kernel: Pid: 28172, comm: httpd Tainted: G        W   3.0.0+ #1
Apr 10 22:28:24 stephan kernel: Call Trace:
Apr 10 22:28:24 stephan kernel: [<ffffffff81062b2f>] warn_slowpath_common+0x7f/0xc0
Apr 10 22:28:24 stephan kernel: [<ffffffff81062c26>] warn_slowpath_fmt+0x46/0x50
Apr 10 22:28:24 stephan kernel: [<ffffffff81268c72>] __list_del_entry+0x82/0xd0
Apr 10 22:28:24 stephan kernel: [<ffffffff81268cd1>] list_del+0x11/0x40
Apr 10 22:28:24 stephan kernel: [<ffffffff8114ba5b>] free_block+0xcb/0x180
Apr 10 22:28:24 stephan kernel: [<ffffffff8114b8e0>] kmem_cache_free+0x290/0x2b0
Apr 10 22:28:24 stephan kernel: [<ffffffff8118b802>] free_buffer_head+0x22/0x40
Apr 10 22:28:24 stephan kernel: [<ffffffff8118bc29>] try_to_free_buffers+0x79/0xc0
Apr 10 22:28:24 stephan kernel: [<ffffffffa01e1a9f>] jbd2_journal_invalidatepage+0xcf/0xf0 [jbd2]
Apr 10 22:28:24 stephan kernel: [<ffffffffa0205b70>] ext4_invalidatepage+0x80/0x190 [ext4]
Apr 10 22:28:24 stephan kernel: [<ffffffffa0204112>] ? ext4_da_release_space+0xd2/0x190 [ext4]
Apr 10 22:28:24 stephan kernel: [<ffffffffa0205cff>] ext4_da_invalidatepage+0x7f/0xa0 [ext4]
Apr 10 22:28:24 stephan kernel: [<ffffffff8110f635>] do_invalidatepage+0x25/0x30
Apr 10 22:28:24 stephan kernel: [<ffffffff8110fa1a>] truncate_inode_page+0x9a/0xb0
Apr 10 22:28:24 stephan kernel: [<ffffffff8110fd90>] truncate_inode_pages_range+0x160/0x480
Apr 10 22:28:24 stephan kernel: [<ffffffffa02067d1>] ? ext4_mark_inode_dirty+0x91/0x230 [ext4]
Apr 10 22:28:24 stephan kernel: [<ffffffff811100c5>] truncate_inode_pages+0x15/0x20
Apr 10 22:28:24 stephan kernel: [<ffffffffa020be1e>] ext4_evict_inode+0x8e/0x3a0 [ext4]
Apr 10 22:28:24 stephan kernel: [<ffffffff811789ee>] evict+0x7e/0x170
Apr 10 22:28:24 stephan kernel: [<ffffffff81178e68>] iput_final+0xc8/0x180
Apr 10 22:28:24 stephan kernel: [<ffffffff81178f58>] iput+0x38/0x50
Apr 10 22:28:24 stephan kernel: [<ffffffff8116eeec>] do_unlinkat+0x15c/0x1c0
Apr 10 22:28:24 stephan kernel: [<ffffffff810c23ff>] ? audit_syscall_entry+0x1bf/0x1f0
Apr 10 22:28:24 stephan kernel: [<ffffffff8116ef66>] sys_unlink+0x16/0x20
Apr 10 22:28:24 stephan kernel: [<ffffffff81501f42>] system_call_fastpath+0x16/0x1b
Apr 10 22:28:24 stephan kernel: ---[ end trace 3120f0f10d7ed085 ]---
share|improve this question
    
How about some information about what the server runs and what it was doing in the time leading up to the crash? –  ewwhite Apr 10 '13 at 21:03
    
Bonding has always been problematic to me, anyway if it's not a memory problem, it may be worth updating your NIC firmwares :) –  Martino Dino Apr 10 '13 at 21:33

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.