Server: RHEL 5.9 / smbd 3.0.33 - Clients: various, though all were using current mount.cifs (5.2)
I already solved this problem, but it was such a nightmare to hunt down these error codes I felt like it needed universal documenting.
Symptoms: Unpredictable, intermittent mount failure from one specific cifs client to linux samba server. All my linux clients pam_mount user homes at login. Randomly, and sporadically the home dir mounts began to fail on one machine. Logins and mounts continued to work flawlessly on all other clients. Initially I thought an unusual amount of activity on the broken client was causing smbd to freak, but intermittent failures lingered even after usage died down.
Attempting to mount by hand fails and reports:
Errors from underlying mount program mount error(12): Cannot allocate memory Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
<debug enable="1"/> in /etc/security/pam_mount.conf.xml to get more info from pam_mount:
command: 'mount' '-t' 'cifs' '//my_server/watdo' '/home/watdo' '-o' 'user=watdo,uid=666,gid=666' pam_mount(misc.c:38): set_myuid<pre>: (ruid/rgid=0/0, e=0/0) pam_mount(misc.c:38): set_myuid<post>: (ruid/rgid=0/0, e=0/0) pam_mount(mount.c:64): Errors from underlying mount program: pam_mount(mount.c:68): mount error(12): Cannot allocate memory pam_mount(mount.c:68): Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)`
/var/log/kern.log also reported on this event:
kernel: [4316790.256149] CIFS VFS: cifs_mount failed w/return code = -12
'echo 1 > /proc/fs/cifs/cifsFYI' cranks up mount.cifs debug (writes to /var/log/debug). Here's the good part (look familiar?):
CIFS Session Established successfully For smb_command 117 Sending smb: total_len 88 cifs_sync_mid_result: cmd=117 mid=54307 state=4 Mapping smb error code 0xc0000205 to POSIX err -12
At this point there is literally no other information available on the client side. cifs mount request goes out and the client dies almost immediately. mount.cifs error(12) is pretty uninformative (man page doesn't help, thx guys). Extensive internet searching reveals this is a common error code, also confirms it as uninformative.
Time to check on the server! Set
log level = 3 for smbd in /etc/samba/smb.conf (from the Using Samba book: "Levels above 3 are for use by the developers and dump enormous amounts of cryptic information." lol!). Here's the relevant line:
[2013/02/08 10:18:03, 3] smbd/error.c:error_packet_set(106)
error packet at smbd/reply.c(514) cmd=117 (SMBtconX) NT_STATUS_INSUFF_SERVER_RESOURCES
Almost there... from smb mailing list archive I found someone reporting a sort of similar issue, identified as a pegged share limit on an individual smb connection. List open shares on the server:
smbstatus -S | grep <serverIP> | wc -l returned 2048. Very conspicuous.
Actually examining the output of
smbstatus -S revealed thousands of entries for 'IPC$'. Samba docs on IPC$ reveal it is involved with anonymous share browsing and access to "some other resources". I set host deny on the server in /etc/samba/smb.conf:
[IPC$] hosts deny = 0.0.0.0/0
Works great now. OK, hopefully something here helps some poor soul some time in the future.
I guess in the spirit of the site I will ask a question: Why would smbd not clean up IPC$ shares? Why establish one IPC$ per user connection to a share rather than one per client connection? Can you disable IPC$ share creation from the client side? Is there a way to increase the max # connections per share (not that this would have helped in this case)? I didn't see it in the docs.