3

I have fileserver running FreeBSD 10.1. The server's storage consists of two HDDs in a mirrored ZFS pool.

This server is running Samba 4.1, as domain member. The domain controller is running on another server (Windows Server 2012). I use winbind for AD integration.

The problem is that from time to time (quite often actually) smbd locks up. From a client's perspective, the Samba share is unresponsive.

What is really weird, is that in such an occasion, the offending smbd processes are caught up in a D-state: uninterruptible sleep. I don't understand what could cause this. I don't use NFS mounts on this server and all Samba shares are located on the local ZFS pool.

This problem is especially annoying because I can't fix it by restarting Samba: the frozen smbd processes don't respond to SIGKILL. The only workaround is rebooting.

Here is some ps aux information that could help:

root            628   0.0  0.7 295348 22688  -  Is   10:50AM   0:05.09 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
root            641   0.0  0.7 295856 22832  -  I    10:50AM   0:01.66 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
p#########    812   0.0  0.9 326684 27132  -  I    11:04AM   0:03.33 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c#########  839   0.0  0.9 329220 27064  -  D    11:07AM   0:07.86 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c#########  939   0.0  0.8 324996 26208  -  I    11:37AM   0:01.22 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c#########  946   0.0  0.9 324988 26504  -  I    11:40AM   0:02.12 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1077   0.0  0.8 325016 26232  -  I    11:46AM   0:02.00 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1084   0.0  0.8 325640 26268  -  I    11:49AM   0:01.17 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1107   0.0  0.9 326200 26864  -  I    11:55AM   0:01.73 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1141   0.0  0.9 325960 26964  -  I    12:02PM   0:02.20 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1167   0.0  0.9 326004 26608  -  I    12:08PM   0:01.14 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1188   0.0  0.9 326192 27044  -  I    12:11PM   0:01.03 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1195   0.0  0.9 326240 27080  -  I    12:15PM   0:01.15 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1209   0.0  0.9 326256 26948  -  I    12:20PM   0:01.62 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1250   0.0  0.8 322384 25768  -  D    12:25PM   0:00.39 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
c######### 1257   0.0  0.8 322384 25908  -  I    12:29PM   0:00.68 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
p#########   1773   0.0  0.8 322752 26004  -  I     2:57PM   0:00.45 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
p#########   1805   0.0  0.8 322752 25188  -  I     3:02PM   0:00.33 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
p#########   1809   0.0  0.8 322752 26040  -  I     3:04PM   0:00.74 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
p#########   1819   0.0  0.8 326980 26276  -  I     3:06PM   0:00.47 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
p#########   1825   0.0  0.8 322752 25952  -  I     3:08PM   0:00.72 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
root           2134   0.0  0.9 323208 26608  -  I     4:58PM   0:02.00 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
root           2160   0.0  0.8 322684 24588  -  I     5:06PM   0:00.78 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf
root           2236   0.0  0.9 332140 28256  -  I     5:34PM   0:06.50 /usr/local/sbin/smbd --daemon --configfile=/usr/local/etc/smb4.conf

As can be seen, there are two smbd processes in the D-state. Further examination using procstat:

# procstat -t 1250
PID    TID COMM             TDNAME           CPU  PRI STATE   WCHAN
1250 100817 smbd             -                  0  120 sleep   zfs

# procstat -k 1250
PID    TID COMM             TDNAME           KSTACK
1250 100817 smbd             -                mi_switch sleepq_wait sleeplk __lockmgr_args vop_stdlock VOP_LOCK1_APV _vn_lock vacl_get_acl sys___acl_get_file amd64_syscall Xfast_syscall


# procstat -t 839
PID    TID COMM             TDNAME           CPU  PRI STATE   WCHAN
839 100819 smbd             -                  1  120 sleep   filedesc
839 100820 smbd             -                  0  120 sleep   zfs# 

# procstat -k 839
PID    TID COMM             TDNAME           KSTACK
839 100819 smbd             -                mi_switch sleepq_wait _sx_slock_hard namei vn_open_cred zfs_getextattr VOP_GETEXTATTR_APV extattr_get_vp sys_extattr_get_file amd64_syscall Xfast_syscall
839 100820 smbd             -                mi_switch sleepq_wait sleeplk __lockmgr_args vop_stdlock VOP_LOCK1_APV _vn_lock knlist_remove_kq filt_vfsdetach knote_fdclose closefp amd64_syscall Xfast_syscall

procstat -f 839 hangs indefinitely...

I don't know what to draw from this. I'm pretty sure my ZFS pool is clean since I recently scrubbed it:

scan: scrub repaired 0 in 3h10m with 0 errors on Fri Apr 17 13:36:50 2015

Both HDDs are in good health. Maybe some filesystem-level corruption that a scrub can't detect/fix? Is there a way in which I can see on what files smbd is freezing?

3

I have a similar configuration with an AD joined server sharing from zfs, and have experienced the 'sleep' issue in FreeBSD 10.1 and 11.0, and several versions of Samba 3 and 4. I began to notice the lockup would occur mainly on changes to existing files, and not creation or deletion. After running very high samba debug levels, and catching a couple of processes just as they turned zombie, I found something that lead me to look at a problem in inotify. Adding this to samba config has ended a year and a half of problems for me. "kernel change notify = no"

| improve this answer | |
  • Thanks for sharing! I just applied the fix/workaround, I hope it works. I'll try to remember to report back if it worked or not! – Compizfox Aug 12 '15 at 15:50
  • It's been over 12 days without problems now. I think I can confidently say your workaround is working! Thanks! – Compizfox Aug 25 '15 at 14:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.