After an impressive uptime of 550 days without trouble our quad opteron system turned into a corrupted system a few days after I mounted an nfs-share from a FreeNAS-box.
Ongoing sessions remained active, but new logins failed since neither getting a shell or chroot worked (authentification worked, as I saw in /var/log/secure - put the login procedure just stucked afterwards..).
/var/log/messages pointed to a stucking write process to the mounting write process:
BUG: soft lockup - CPU#38 stuck for 61s! [maq:27850]
Modules linked in: nfs lockd fscache nfs_acl auth_rpcgss sunrpc sit tunnel4 tun fuse autofs4 cpu
freq_ondemand powernow_k8 freq_table ipt_REJECT nf_conntrack_ipv4 nf_defra
g_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log sg amd64_edac_mod edac_core edac_mce_amd i2c_piix4 i2c_core igb dca ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom 3w_sas ata_generic pata_acpi pata_atiixp ahci qla2xxx scsi_transport_fc scsi_tgt dm_mod [last unloaded: scsi_wait_scan]
CPU 38:
Modules linked in: nfs lockd fscache nfs_acl auth_rpcgss sunrpc sit tunnel4 tun fuse autofs4 cpufreq_ondemand powernow_k8 freq_table ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 dm_mirror dm_region_hash dm_log sg amd64_edac_mod edac_core edac_mce_amd i2c_piix4 i2c_core igb dca ext4 mbcache jbd2 sd_mod crc_t10dif sr_mod cdrom 3w_sas ata_generic pata_acpi pata_atiixp ahci qla2xxx scsi_transport_fc scsi_tgt dm_mod [last unloaded: scsi_wait_scan]
Pid: 27850, comm: maq Not tainted 2.6.32-71.29.1.el6.x86_64 #1 H8QG6
RIP: 0010:[<ffffffff814cbab7>] [<ffffffff814cbab7>] _spin_unlock_irqrestore+0x17/0x20
RSP: 0018:ffff886c12ddd8f8 EFLAGS: 00000202
RAX: 0000000000000202 RBX: ffff886c12ddd8f8 RCX: 000000000000f14b
RDX: ffff886060611448 RSI: 0000000000000202 RDI: 0000000000000202
RBP: ffffffff81013c8e R08: ffff886060611450 R09: 94adb435c189d607
R10: 0000000000000000 R11: 0000000000000001 R12: ffff881027c2d200
R13: 0000000000000026 R14: ffff88015521a8c0 R15: ffff8835e94e3200
FS: 00007f8aa27be720(0000) GS:ffff886060880000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007ff69c45a000 CR3: 0000003289c86000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Call Trace:
[<ffffffff810920ae>] ? prepare_to_wait_exclusive+0x4e/0x80
[<ffffffff814ca168>] ? __wait_on_bit_lock+0x48/0xc0
[<ffffffff81091dd7>] ? bit_waitqueue+0x87/0xd0
[<ffffffffa0338560>] ? nfs_wait_bit_killable+0x0/0x40 [nfs]
[<ffffffff814ca258>] ? out_of_line_wait_on_bit_lock+0x78/0x90
[<ffffffff81091ee0>] ? wake_bit_function+0x0/0x50
[<ffffffffa0345c9a>] ? nfs_commit_inode+0xaa/0x1c0 [nfs]
[<ffffffffa0345e29>] ? nfs_wb_page+0x79/0xd0 [nfs]
[<ffffffffa0344170>] ? nfs_page_find_request+0x50/0x70 [nfs]
[<ffffffffa0345ec0>] ? nfs_flush_incompatible+0x40/0x70 [nfs]
[<ffffffffa0334aa3>] ? nfs_write_begin+0x93/0x220 [nfs]
[<ffffffff8110cbce>] ? generic_file_buffered_write+0x10e/0x2a0
[<ffffffff8110e520>] ? __generic_file_aio_write+0x250/0x480
[<ffffffff8110e7bf>] ? generic_file_aio_write+0x6f/0xe0
[<ffffffffa033571a>] ? nfs_file_write+0xda/0x1e0 [nfs]
[<ffffffff8116d05a>] ? do_sync_write+0xfa/0x140
[<ffffffff81091ea0>] ? autoremove_wake_function+0x0/0x40
[<ffffffff8120caab>] ? selinux_file_permission+0xfb/0x150
[<ffffffff811fff16>] ? security_file_permission+0x16/0x20
[<ffffffff8116d358>] ? vfs_write+0xb8/0x1a0
[<ffffffff8116dd91>] ? sys_write+0x51/0x90
I tried to kill (-9) the process which raised the error ('maq') but the process was stucking and couldn't be killed (the filesystem mounted via nfs was stucking too). Restarting all services in concern on the local machine and of the nfs service on the freenas server didn't help. After two hours of work on resisting processes on a more or less instable system, I had to reboot the system (worked without any hang ups).
When I checked the logfiles, I saw that the auto-update function of yum was enabled and that the selinux-policy stuff was updated. Could this have caused the login problems?
Could nfs cause a more or less completely stucking system in case of (short?) write latencies (can I prevent a new soft lockup by using 'mount -o soft')? Or might this be a kernel bug?
uname -r
2.6.32-71.29.1.el6.x86_64
cat /etc/issue
Scientific Linux release 6.0 (Carbon)