I have fileserver running FreeBSD 10.1. The server's storage consists of two HDDs in a mirrored ZFS pool.
This server is running Samba 4.1, as domain member. The domain controller is running on another server (Windows Server 2012). I use winbind for AD integration.
The problem is that from time to time (quite often actually) smbd locks up. From a client's perspective, the Samba share is unresponsive.
What is really weird, is that in such an occasion, the offending smbd processes are caught up in a D-state: uninterruptible sleep. I don't understand what could cause this. I don't use NFS mounts on this server and all Samba shares are located on the local ZFS pool.
This problem is especially annoying because I can't fix it by restarting Samba: the frozen smbd processes don't respond to SIGKILL. The only workaround is rebooting.
Here is some ps aux information that could help:
root 628 0.0 0.7 295348 22688 - Is 10:50AM 0:05.09 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
root 641 0.0 0.7 295856 22832 - I 10:50AM 0:01.66 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
p######### 812 0.0 0.9 326684 27132 - I 11:04AM 0:03.33 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 839 0.0 0.9 329220 27064 - D 11:07AM 0:07.86 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 939 0.0 0.8 324996 26208 - I 11:37AM 0:01.22 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 946 0.0 0.9 324988 26504 - I 11:40AM 0:02.12 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1077 0.0 0.8 325016 26232 - I 11:46AM 0:02.00 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1084 0.0 0.8 325640 26268 - I 11:49AM 0:01.17 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1107 0.0 0.9 326200 26864 - I 11:55AM 0:01.73 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1141 0.0 0.9 325960 26964 - I 12:02PM 0:02.20 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1167 0.0 0.9 326004 26608 - I 12:08PM 0:01.14 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1188 0.0 0.9 326192 27044 - I 12:11PM 0:01.03 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1195 0.0 0.9 326240 27080 - I 12:15PM 0:01.15 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1209 0.0 0.9 326256 26948 - I 12:20PM 0:01.62 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1250 0.0 0.8 322384 25768 - D 12:25PM 0:00.39 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1257 0.0 0.8 322384 25908 - I 12:29PM 0:00.68 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
p######### 1773 0.0 0.8 322752 26004 - I 2:57PM 0:00.45 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
p######### 1805 0.0 0.8 322752 25188 - I 3:02PM 0:00.33 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
p######### 1809 0.0 0.8 322752 26040 - I 3:04PM 0:00.74 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
p######### 1819 0.0 0.8 326980 26276 - I 3:06PM 0:00.47 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
p######### 1825 0.0 0.8 322752 25952 - I 3:08PM 0:00.72 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
root 2134 0.0 0.9 323208 26608 - I 4:58PM 0:02.00 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
root 2160 0.0 0.8 322684 24588 - I 5:06PM 0:00.78 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
root 2236 0.0 0.9 332140 28256 - I 5:34PM 0:06.50 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
As can be seen, there are two smbd processes in the D-state. Further examination using procstat:
# procstat -t 1250
PID TID COMM TDNAME CPU PRI STATE WCHAN
1250 100817 smbd - 0 120 sleep zfs
# procstat -k 1250
PID TID COMM TDNAME KSTACK
1250 100817 smbd - mi_switch sleepq_wait sleeplk __lockmgr_args vop_stdlock VOP_LOCK1_APV _vn_lock vacl_get_acl sys___acl_get_file amd64_syscall Xfast_syscall
# procstat -t 839
PID TID COMM TDNAME CPU PRI STATE WCHAN
839 100819 smbd - 1 120 sleep filedesc
839 100820 smbd - 0 120 sleep zfs#
# procstat -k 839
PID TID COMM TDNAME KSTACK
839 100819 smbd - mi_switch sleepq_wait _sx_slock_hard namei vn_open_cred zfs_getextattr VOP_GETEXTATTR_APV extattr_get_vp sys_extattr_get_file amd64_syscall Xfast_syscall
839 100820 smbd - mi_switch sleepq_wait sleeplk __lockmgr_args vop_stdlock VOP_LOCK1_APV _vn_lock knlist_remove_kq filt_vfsdetach knote_fdclose closefp amd64_syscall Xfast_syscall
procstat -f 839 hangs indefinitely...
I don't know what to draw from this. I'm pretty sure my ZFS pool is clean since I recently scrubbed it:
scan: scrub repaired 0 in 3h10m with 0 errors on Fri Apr 17 13:36:50 2015
Both HDDs are in good health. Maybe some filesystem-level corruption that a scrub can't detect/fix? Is there a way in which I can see on what files smbd is freezing?
