Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I hate PAM since it came up.

How do I switch on PAM debugging in Debian Squeeze at the admin level?

I have checked every resource I was able to find. Google, manpages, whatever. The only thing I haven't tried yet (I simply not dare to, did I mention that I hate PAM?) is digging into the PAM's library source.

I tried to google for a solution, nothing. What I found so far:

http://www.bitbull.ch/wiki/index.php/Pam_debugging_funktion (/etc/pam_debug) and http://nixdoc.net/man-pages/HP-UX/man4/pam.conf.4.html (debug option on PAM entries in /etc/pam.d/).

Nope, does not work. No PAM output, nothing, absolute silence.

While searching for a solution I even followed links to Pam, that are gas stations here in Germany. Well, yes, perhaps in all those billion of hits might hiding a clue, but shoot me I'd be dead before I discover.

Rest is FYI:

What problem did I have?

After upgrading to Debian Squeeze something got weird (well, hey, it once was, uh, what was right over the Etch .. ah, yes, Woody). So it's probably not Debian's fault, just a long lived screwed up setup. I immediately had the impression it has to do something with PAM, but I really did not know what's going on. I was completely in the dark, left alone, helpless as a baby, YKWIM. Some ssh logins worked, some not. It was kind of funny. No clues in ssh -v, no clues in /var/log/*, nothing. Just "auth succeeded" or "auth fail", sometimes the same user logging in parallely succeeded with one session and failed with the other, at the same time. And nothing you really can get hold of.

After digging trainloads of other options I was able to find out. There is nullok and nullok_secure, a Debian special. Something screwed with /etc/securetty and depending on the tty (which is somewhat random) a login was rejected or not. REALLY NICE, phew!

The fix was easy and everything's now fine again.

However this left me with the question, how to debug such a mess in future. It's not the first time PAM drives me nuts. So I would like to see a final solution. Final as in "solved", not final as in "armageddon". Thanks.

Ah, BTW, this again strengthened my belief in that it's good to hate PAM since it came up. Did I mention that I do?

share|improve this question
    
Here is how to create this trouble yourself on Debian: passwd -d user and then try to ssh into the box as this user. The output "failed password" in syslog has nothing to do with PAM debugging at all, so PAM stays silent. –  Tino Jul 21 '11 at 15:40
    
I forgot to mention that you must set PermitEmptyPasswords yes in /etc/ssh/sshd_config of course, then PAM outputs something like pam_unix(sshd:auth): authentication failure, but still nothing to the debug channel nor any hint which PAM module caused the failure. –  Tino Jul 21 '11 at 15:49
    
Does debian have a /var/log/auth.log file? I recently discovered that Ubuntu has it, and logs all the pam related stuff there. None of the answers here helped me, but looking in /var/log/auth.log helped me fix my problem. –  LordOfThePigs Jul 10 at 9:41
    
/var/log/auth.log is syslog. The problem is not logging but debugging. If for example the PAM stack fails early, you will not see anything, because the modules which output to syslog are not invoked at all. Or something fails and something not, but both log exactly the same lines. It is correct that, I guess, 95% of all cases can be solved by looking into the usual logs, but 5% cannot, as there simply is no trace of what really happens behind the scenes. –  Tino Jul 12 at 13:10

4 Answers 4

up vote 10 down vote accepted

A couple of things for you to try:

Did you enable logging of debug messages in syslog?

cp /etc/syslog.conf /etc/syslog.conf.original
vi /etc/syslog.conf

Add the following line:

*.debug     /var/log/debug.log

Exit with :wq!.

touch /var/log/debug.log
service syslog restart

You can enable debugging for all modules like so:

touch /etc/pam_debug

OR you can enable debugging only for the modules you're interested in by adding "debug" to the end of the relevant lines in /etc/pam.d/system-auth or the other /etc/pam.d/* files:

login   auth    required    pam_unix.so debug

Then debugging messages should start appearing in /var/log/debug.log. Hope this helps you out!

share|improve this answer
    
Good answer but I think I had syslog debugging on. I will check it out. –  Tino Jul 21 '11 at 15:18
    
I checked it, sorry, your answer is not the solution. PAM still stays silent. Perhaps this is a nullok special that just this module is lacking debugging. Let me say it with this words: Lacking debugging on such a crucial central piece of code is a worse nightmare than just being haunted by Freddy Kruger. –  Tino Jul 21 '11 at 15:33
    
OK, well, I think you answer accutally is correct! It's not the answer's fault that PAM is mute. So for the time being I accept it as "solution" until PAM capitulates. Thanks. –  Tino Jul 21 '11 at 15:58

Debian and Ubuntu (and maybe other distros) have a special log file into which all pam output is logged:

/var/log/auth.log

I've been struggling with a pam related problem for a day and a half, finally found out about this log file, and saved myself from insanity.

Here's a sample of the contents of this file when things don't go as planned.

Jul 10 09:31:14 vagrant-ubuntu-trusty-64 pamtester: pam_userdb(vsftpd:auth): user_lookup: could not open database `/etc/vsftpd_users.db': No such file or directory
Jul 10 09:36:20 vagrant-ubuntu-trusty-64 sudo:  vagrant : TTY=pts/1 ; PWD=/home/vagrant ; USER=root ; COMMAND=/bin/cat /var/log/auth.log

Here's how it looks when it works:

Jul 10 09:47:00 vagrant-ubuntu-trusty-64 sshd[5222]: pam_unix(sshd:session): session closed for user vagrant
Jul 10 09:50:58 vagrant-ubuntu-trusty-64 sshd[5584]: error: Could not load host key: /etc/ssh/ssh_host_ed25519_key
Jul 10 09:50:58 vagrant-ubuntu-trusty-64 sshd[5584]: Accepted publickey for vagrant from 10.0.2.2 port 54652 ssh2: RSA dd:3b:b8:2e:85:04:06:e9:ab:ff:a8:0a:c0:04:6e:d6
Jul 10 09:50:58 vagrant-ubuntu-trusty-64 sshd[5584]: pam_unix(sshd:session): session opened for user vagrant by (uid=0)
Jul 10 09:51:13 vagrant-ubuntu-trusty-64 sudo:  vagrant : TTY=pts/1 ; PWD=/home/vagrant ; USER=root ; COMMAND=/bin/cat /var/log/auth.log
Jul 10 09:51:13 vagrant-ubuntu-trusty-64 sudo: pam_unix(sudo:session): session opened for user root by vagrant(uid=0)
Jul 10 09:51:13 vagrant-ubuntu-trusty-64 sudo: pam_unix(sudo:session): session closed for user root
Jul 10 09:51:41 vagrant-ubuntu-trusty-64 pamtester: pam_userdb(vsftpd:auth): user 'foo' granted access
Jul 10 09:51:44 vagrant-ubuntu-trusty-64 sudo:  vagrant : TTY=pts/1 ; PWD=/home/vagrant ; USER=root ; COMMAND=/bin/cat /var/log/auth.log
Jul 10 09:51:44 vagrant-ubuntu-trusty-64 sudo: pam_unix(sudo:session): session opened for user root by vagrant(uid=0)

Note that none of the other possibilities for enabling pam debug logging worked for me.

share|improve this answer
    
Please note that all lines like pam_* actually are PAM realated. The other lines are output by the tools anyway, regardless if they use PAM or not. This means: If PAM denies, for any reason, it is really hard to find the real cause, if it is in PAM. The non-PAM lines are not helpful (as the problem sits in PAM) and the PAM lines are often not helpful, too, as they are often too silent. With the presence of many PAM modules you really have a hard time guessing which module might be the culprit, let alone to find out how to enable debugging, as this is different for each single one. –  Tino Jul 12 at 13:20

At least on CentOS 6.4, /etc/pam_debug is NOT used.

If the pam_warn.so module is installed, you can get some logging output this way:

auth required pam_warn.so

success required pam_warn.so

etc. This module ensures that it will not interfere with the authentication process at any point, but it logs meaningful stuff via syslog.

Update

After examining the code and doing some compiling, I found that (1) it's possible to enable this debug mode through the source, and (2) a RHEL patch makes the feature nearly unusable (at least the pam_unix module) and (3) it's probably better to patch the code anyway.

To get this to work for RHEL, you can get the Linux-PAM ... src.rpm (for any 1.1 version) and change the spec file as follows:

  • Find the line beginning with

    %configure \

and after it, add --enable-debug \

  • Remove the line or comment-out the line (above the previous one) that begins with %patch2

Then build the rpm and install (with force, to overwrite existing packages). Now create the file /var/run/pam-debug.log:

install -m 622 /var/run/pam-debug.log

If the file does not exist, debug output will be sent to stderr.

  • This sending out to stderr is, in my opinion, stupid, and is what causes the patch conflict. You can change that behavior by going into the file libpam/include/security/_pam_macros.h and replacing the 4 lines of

    logfile = stderr;

with

return;

On build, you'll get warnings about unreachable statements, but they can be ignored. You can make this change in a sed script (and put it in the %prep section of the RPM after the patches)...

sed -i 's/logfile = stderr;$/return;/' libpam/include/security/_pam_macros.h

IF you do this little patch, you can restore the %patch2, as it should work again properly.

share|improve this answer
    
Thanks. Good hint. I'll try if I ever have problems again. So hopefully never .. ;) –  Tino Feb 20 at 4:48

Asket... I really loved your post :) I was fighting with stuff like this for the last 15 hours... (I might have had a 30 min break though)

Somehow i got it working by doing all the stuff you did, which means i have a /etc/pam_debug and debug on pam entries. BUT as in my case i was struggling with pam_mysql, I had to put another verbose=1 after debug on my pam entries:

mailsystem:~# cat /etc/pam.d/smtp

auth required pam_mysql.so debug sqllog=1 verbose=1 user=mail passwd=imverysecret host=127.0.0.1 db=mail table=mailbox usercolumn=username passwdcolumn=password crypt=1 logtable=log_auth logmsgcolumn=msg logusercolumn=user logpidcolumn=pid loghostcolumn=host logtimecolumn=time

account sufficient pam_mysql.so debug sqllog=1 verbose=1 user=mail passwd=imverysecret host=127.0.0.1 db=mail table=mailbox usercolumn=username passwdcolumn=password crypt=1 logtable=log_auth logmsgcolumn=msg logusercolumn=user logpidcolumn=pid loghostcolumn=host logtimecolumn=times

That "sqllog" is just to write logs in the DB.

So maybe this helps you just a little bit.

We all hate PAM. Good luck!

share|improve this answer
1  
Thanks for the hint, but unfortunately this does not help: pam_unix(sshd:auth): unrecognized option [verbose=1] –  Tino Jul 21 '11 at 15:51

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.