Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

This is the content of /var/messages when things crumble:

Dec 21 19:47:45 localhost kernel: ------------[ cut here ]------------
Dec 21 19:47:45 localhost kernel: WARNING: at net/sched/sch_generic.c:261 dev_watchdog+0x26d/0x280() (Not tainted)
Dec 21 19:47:45 localhost kernel: Hardware name: KGP(M)E-D16
Dec 21 19:47:45 localhost kernel: NETDEV WATCHDOG: eth0 (e1000e): transmit queue 0 timed out
Dec 21 19:47:45 localhost kernel: Modules linked in: ipt_REDIRECT iptable_nat nf_nat xt_multiport xt_owner ext3 jbd nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack iptable_filter ip_tables autofs4 sunrpc cpufreq_ondemand powernow_k8 freq_table mperf ipv6 e1000e microcode serio_raw k10temp edac_core edac_mce_amd i2c_piix4 i2c_core sg shpchp ext4 mbcache jbd2 sd_mod crc_t10dif ata_generic pata_acpi pata_atiixp ahci dm_mirror dm_region_hash dm_log dm_mod [last unloaded: nf_conntrack]
Dec 21 19:47:45 localhost kernel: Pid: 0, comm: swapper Not tainted 2.6.32-220.el6.x86_64 #1
Dec 21 19:47:45 localhost kernel: Call Trace:
Dec 21 19:47:45 localhost kernel: <IRQ>  [<ffffffff81069b77>] ? warn_slowpath_common+0x87/0xc0
Dec 21 19:47:45 localhost kernel: [<ffffffff81069c66>] ? warn_slowpath_fmt+0x46/0x50
Dec 21 19:47:45 localhost kernel: [<ffffffff8144a54d>] ? dev_watchdog+0x26d/0x280
Dec 21 19:47:45 localhost kernel: [<ffffffff8144a2e0>] ? dev_watchdog+0x0/0x280
Dec 21 19:47:45 localhost kernel: [<ffffffff8107c957>] ? run_timer_softirq+0x197/0x340
Dec 21 19:47:45 localhost kernel: [<ffffffff810a0b70>] ? tick_sched_timer+0x0/0xc0
Dec 21 19:47:45 localhost kernel: [<ffffffff8102ad2d>] ? lapic_next_event+0x1d/0x30
Dec 21 19:47:45 localhost kernel: [<ffffffff81072161>] ? __do_softirq+0xc1/0x1d0
Dec 21 19:47:45 localhost kernel: [<ffffffff81095770>] ? hrtimer_interrupt+0x140/0x250
Dec 21 19:47:45 localhost kernel: [<ffffffff8100c24c>] ? call_softirq+0x1c/0x30
Dec 21 19:47:45 localhost kernel: [<ffffffff8100de85>] ? do_softirq+0x65/0xa0
Dec 21 19:47:45 localhost kernel: [<ffffffff81071f45>] ? irq_exit+0x85/0x90
Dec 21 19:47:45 localhost kernel: [<ffffffff814f4de0>] ? smp_apic_timer_interrupt+0x70/0x9b
Dec 21 19:47:45 localhost kernel: [<ffffffff8100bc13>] ? apic_timer_interrupt+0x13/0x20
Dec 21 19:47:45 localhost kernel: <EOI>  [<ffffffff810375ab>] ? native_safe_halt+0xb/0x10
Dec 21 19:47:45 localhost kernel: [<ffffffff810145dd>] ? default_idle+0x4d/0xb0
Dec 21 19:47:45 localhost kernel: [<ffffffff81009e06>] ? cpu_idle+0xb6/0x110
Dec 21 19:47:45 localhost kernel: [<ffffffff814d411a>] ? rest_init+0x7a/0x80
Dec 21 19:47:45 localhost kernel: [<ffffffff81c1ff76>] ? start_kernel+0x424/0x430
Dec 21 19:47:45 localhost kernel: [<ffffffff81c1f33a>] ? x86_64_start_reservations+0x125/0x129
Dec 21 19:47:45 localhost kernel: [<ffffffff81c1f438>] ? x86_64_start_kernel+0xfa/0x109
Dec 21 19:47:45 localhost kernel: ---[ end trace 1c035fe603219926 ]---
Dec 21 19:47:45 localhost kernel: e1000e 0000:03:00.0: eth0: Reset adapter
Dec 21 19:47:46 localhost abrt-dump-oops: Reported 1 kernel oopses to Abrt
Dec 21 19:47:46 localhost abrtd: Directory 'oops-2012-12-21-19:47:46-12170-0' creation detected
Dec 21 19:47:47 localhost abrtd: Can't open file '/var/spool/abrt/oops-2012-12-21-19:47:46-12170-0/uid': No such file or directory
Dec 21 19:47:54 localhost kernel: Bridge firewalling registered
Dec 21 19:49:05 localhost abrtd: Sending an email...
Dec 21 19:49:05 localhost abrtd: Email was sent to: root@localhost
Dec 21 19:49:05 localhost abrtd: New problem directory /var/spool/abrt/oops-2012-12-21-19:47:46-12170-0, processing
Dec 21 19:49:05 localhost abrtd: Can't open file '/var/spool/abrt/oops-2012-12-21-19:47:46-12170-0/uid': No such file or directory

Looks like a hardware named KGP(M)E-D16 stop or something. Looking that up at google shows that it's the motherboard.

What else should I check? I reported this to fdcservers.net already.

They claimed it's kernel bug. Instead of hardware issue. What kernel bug? Why does it cause server to crash? What should I do?

Checking driver for the network card I got this

root@host [/var/log]# ethtool -i eth0
driver: e1000e
version: 1.9.5-k
firmware-version: 1.8-0
bus-info: 0000:03:00.0
root@host [/var/log]# ethtool -i eth1
driver: e1000e
version: 1.9.5-k
firmware-version: 1.8-0
bus-info: 0000:02:00.0
root@host [/var/log]# ethtool -i eth2
Cannot get driver information: No such device

That being said,

Hardware name: KGP(M)E-D16 is asus motherboard. Also if you search Hardware name: KGP(M)E-D16 this page rank in top 3.

share|improve this question
1  
My guess would be a defective network adapter driver. What version of the e1000 driver is running? –  Greg Askew Dec 22 '12 at 19:17
    
ugh how to know that? –  Sharen Eayrs Dec 22 '12 at 19:42
1  
ethtool -i ethX [Replace X with 0, 1 or whatever you have] –  Soham Chakraborty Dec 22 '12 at 19:45
add comment

1 Answer 1

up vote 12 down vote accepted

The problem is as it states itself. Line 261 of net/sched/sch_generic.c which is generic packet scheduler routine.

The panic itself is here

Dec 21 19:47:45 localhost kernel: [<ffffffff8144a54d>] ? dev_watchdog+0x26d/0x280

So, the network device timed out. As the source code says, some queue was blocked and the timer expired. It was supposed to hold the device for certain particular time but the counter got over. Here is the relevant portion of the code.

if (!mod_timer(&dev->watchdog_timer,
258 round_jiffies(jiffies +
259 dev->watchdog_timeo)))
260 dev_hold(dev);

You see there is a watchdog timer and the counter is measured in jiffies. When this timer is over, it throws warning.

This has something to do with your network card or driver. I will straight away reject the theory of kernel bug unless they can prove it. And there is no way to tell it, unless someone has reported the exact call trace or Intel knows about this trace and it happens on the same hardware, same driver, same firmware. In a nutshell, without checking the kernel dump or vmcore, no experienced person will tell this is kernel bug. The part of kernel that deals with timer, is carefully crafted out and e1000 is not an obscure driver to deal with.

I don't want to diss your server guys but this is my take. It would be worthwhile to check your ethtool -S ethX outputs to see whether there are any drops, overruns, timeouts etc.

share|improve this answer
    
Is this responsibility of me or my host? I mean they provide the hardware and set up the centosh. Also how can such a simple timeout cause server not to respond? The server is still running and log is still being created but something sites are no longer served and ping doesn't work either –  Sharen Eayrs Dec 22 '12 at 21:00
    
you said the network devide time out even though the log gives a code for motherboard. Are you sure? –  Sharen Eayrs Dec 22 '12 at 22:31
    
More importantly, what should I do? –  Sharen Eayrs Dec 22 '12 at 22:34
    
@SharenEayrs That might be because the Ethernet is embedded onto the motherboard. In any case, the thing to do is to replace the defective hardware. –  Michael Hampton Dec 22 '12 at 22:40
    
Yeah, change the hardware and see. Also, CentOS 5 or 6 should provide a tool called kdump (search about it). If you set up kdump rightly, you can collect vmcore which you can send to your hosts's way to analyze for the definitive reason to find out the reason of crash. –  Soham Chakraborty Dec 23 '12 at 1:50
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.