Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

After an impressive uptime of 550 days without trouble our quad opteron system turned into a corrupted system a few days after I mounted an nfs-share from a FreeNAS-box.

Ongoing sessions remained active, but new logins failed since neither getting a shell or chroot worked (authentification worked, as I saw in /var/log/secure - put the login procedure just stucked afterwards..).

/var/log/messages pointed to a stucking write process to the mounting write process:

 BUG: soft lockup - CPU#38 stuck for 61s! [maq:27850]
Modules linked in: nfs lockd fscache nfs_acl auth_rpcgss sunrpc sit tunnel4 tun fuse autofs4 cpu
freq_ondemand powernow_k8 freq_table ipt_REJECT nf_conntrack_ipv4 nf_defra
g_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log sg amd64_edac_mod edac_core edac_mce_amd i2c_piix4 i2c_core igb dca ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom 3w_sas ata_generic pata_acpi pata_atiixp ahci qla2xxx scsi_transport_fc scsi_tgt dm_mod [last unloaded: scsi_wait_scan]
CPU 38:
Modules linked in: nfs lockd fscache nfs_acl auth_rpcgss sunrpc sit tunnel4 tun fuse autofs4 cpufreq_ondemand powernow_k8 freq_table ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log sg amd64_edac_mod edac_core edac_mce_amd i2c_piix4 i2c_core igb dca ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom 3w_sas ata_generic pata_acpi pata_atiixp ahci qla2xxx scsi_transport_fc scsi_tgt dm_mod [last unloaded: scsi_wait_scan]
Pid: 27850, comm: maq Not tainted 2.6.32-71.29.1.el6.x86_64 #1 H8QG6
RIP: 0010:[<ffffffff814cbab7>]  [<ffffffff814cbab7>] _spin_unlock_irqrestore+0x17/0x20
RSP: 0018:ffff886c12ddd8f8  EFLAGS: 00000202
RAX: 0000000000000202 RBX: ffff886c12ddd8f8 RCX: 000000000000f14b
RDX: ffff886060611448 RSI: 0000000000000202 RDI: 0000000000000202
RBP: ffffffff81013c8e R08: ffff886060611450 R09: 94adb435c189d607
R10: 0000000000000000 R11: 0000000000000001 R12: ffff881027c2d200
R13: 0000000000000026 R14: ffff88015521a8c0 R15: ffff8835e94e3200
FS:  00007f8aa27be720(0000) GS:ffff886060880000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007ff69c45a000 CR3: 0000003289c86000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
 [<ffffffff810920ae>] ? prepare_to_wait_exclusive+0x4e/0x80
 [<ffffffff814ca168>] ? __wait_on_bit_lock+0x48/0xc0
 [<ffffffff81091dd7>] ? bit_waitqueue+0x87/0xd0
 [<ffffffffa0338560>] ? nfs_wait_bit_killable+0x0/0x40 [nfs]
 [<ffffffff814ca258>] ? out_of_line_wait_on_bit_lock+0x78/0x90
 [<ffffffff81091ee0>] ? wake_bit_function+0x0/0x50
 [<ffffffffa0345c9a>] ? nfs_commit_inode+0xaa/0x1c0 [nfs]
 [<ffffffffa0345e29>] ? nfs_wb_page+0x79/0xd0 [nfs]
 [<ffffffffa0344170>] ? nfs_page_find_request+0x50/0x70 [nfs]
 [<ffffffffa0345ec0>] ? nfs_flush_incompatible+0x40/0x70 [nfs]
 [<ffffffffa0334aa3>] ? nfs_write_begin+0x93/0x220 [nfs]
 [<ffffffff8110cbce>] ? generic_file_buffered_write+0x10e/0x2a0
 [<ffffffff8110e520>] ? __generic_file_aio_write+0x250/0x480
 [<ffffffff8110e7bf>] ? generic_file_aio_write+0x6f/0xe0
 [<ffffffffa033571a>] ? nfs_file_write+0xda/0x1e0 [nfs]
 [<ffffffff8116d05a>] ? do_sync_write+0xfa/0x140
 [<ffffffff81091ea0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff8120caab>] ? selinux_file_permission+0xfb/0x150
 [<ffffffff811fff16>] ? security_file_permission+0x16/0x20
 [<ffffffff8116d358>] ? vfs_write+0xb8/0x1a0
 [<ffffffff8116dd91>] ? sys_write+0x51/0x90

I tried to kill (-9) the process which raised the error ('maq') but the process was stucking and couldn't be killed (the filesystem mounted via nfs was stucking too). Restarting all services in concern on the local machine and of the nfs service on the freenas server didn't help. After two hours of work on resisting processes on a more or less instable system, I had to reboot the system (worked without any hang ups).

When I checked the logfiles, I saw that the auto-update function of yum was enabled and that the selinux-policy stuff was updated. Could this have caused the login problems?

Could nfs cause a more or less completely stucking system in case of (short?) write latencies (can I prevent a new soft lockup by using 'mount -o soft')? Or might this be a kernel bug?

uname -r
2.6.32-71.29.1.el6.x86_64
cat /etc/issue
Scientific Linux release 6.0 (Carbon)

Many thanks!

share|improve this question
    
...and this is one of the reasons 550 days of uptime is a bad idea. –  HopelessN00b Jan 18 '13 at 11:51
    
Yes, any kind of soft lockup means something wrong on kernel side. Mounting or other things won't change this. Gimme some time, I am pulling the git repo. I doubt I will find something but I can try. –  Soham Chakraborty Jan 18 '13 at 15:04

1 Answer 1

Upgrade your OS.

(e.g. it does not make sense to debug errors from an old kernel like this)

This applies to CentOS, RHEL, Scientific Linux, etc. EL6.0 was full of issues/bugs and nearly unusable for my purposes. EL6.1 was a little better, but my systems really stabilized at the EL 6.2 and 6.3 levels.

You've probably run into an issue fixed in an upstream RHEL errata kernel.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.