Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

It is company policy for admins to login to the servers via a personal username, and then run sudo -i to become root. Upon running sudo -i, sudo will create an environmental variable called SUDO_USER, which contains the original user's username.

Is there a way to log ALL commands within syslog with something akin to the following syntax:

${TIME/DATE STAMP}: [${REAL_USER}|${SUDO_USER}]: ${CMD}

An example entry would be:

Sat Jan 19 22:28:46 CST 2013: [root|ksoviero]: yum install random-pkg

Obviously it doesn't have to be exactly the above syntax, it just has to include a minimum of the real user (eg. root), the sudo user (eg. ksoviero), and the full command that was run (eg. yum install random-pkg).

I've already tried snoopy, but it did not include the SUDO_USER variable.

share|improve this question
10  
You need auditd. –  Michael Hampton Jan 20 '13 at 4:56
1  
This is a brief intro to auditd –  Deer Hunter Jan 20 '13 at 7:37
1  
Could somebody please post this as an answer? Please include how I would strictly log all commands run by users. The "brief intro to auditd" was useful, but it didn't include anything related to logging actual commands (as far as I could tell anyway). –  Soviero Jan 20 '13 at 16:45
1  
Ok, I've started playing with auditd, and while I’ve gotten it to log all commands being run, it doesn't include the SUDO_USER variable (or equivalent information), and I can't find a way to include it. Any help would be greatly appreciated! –  Soviero Jan 21 '13 at 2:02
1  
And what will the company do with this record of all commands entered by administrators? –  ewwhite Feb 3 '13 at 19:03

6 Answers 6

up vote 13 down vote accepted
+50

As was suggested by Michael Hampton, auditd is the correct tool for the job here.

I tested this on an Ubuntu 12.10 installation, so your mileage may vary on other systems.

  • Install auditd:

    apt-get install auditd

  • Add these 2 lines to /etc/audit/audit.rules:

    -a exit,always -F arch=b64 -F euid=0 -S execve
    -a exit,always -F arch=b32 -F euid=0 -S execve

These will track all commands run by root (euid=0). Why two rules? The execve syscall must be tracked in both 32 and 64 bit code.

  • To get rid of auid=4294967295 messages in logs, add audit=1 to the kernel's cmdline (by editing /etc/default/grub)

  • Place the line

    session required pam_loginuid.so

in all PAM config files that are relevant to login (/etc/pam.d/{login,kdm,sshd}), but not in the files that are relevant to su or sudo. This will allow auditd to get the calling user's uid correctly when calling sudo or su.

  • Restart your system now.

  • Let's login and run some commands:

    $ id -u
    1000
    $ sudo ls /
    bin  boot  data  dev  etc  home  initrd.img  initrd.img.old  lib  lib32  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  scratch  selinux  srv  sys  tmp  usr  var  vmlinuz  vmlinuz.old
    $ sudo su -
    # ls /etc
    [...]

This will yield something like this in /var/log/audit/auditd.log:

----
time->Mon Feb  4 09:57:06 2013
type=PATH msg=audit(1359968226.239:576): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968226.239:576): item=0 name="/bin/ls" inode=2117 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968226.239:576):  cwd="/home/user"
type=EXECVE msg=audit(1359968226.239:576): argc=2 a0="ls" a1="/"
type=SYSCALL msg=audit(1359968226.239:576): arch=c000003e syscall=59 success=yes exit=0 a0=10cfc48 a1=10d07c8 a2=10d5750 a3=7fff2eb2d1f0 items=2 ppid=26569 pid=26570 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="ls" exe="/bin/ls" key=(null)
----
time->Mon Feb  4 09:57:06 2013
type=PATH msg=audit(1359968226.231:575): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968226.231:575): item=0 name="/usr/bin/sudo" inode=530900 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968226.231:575):  cwd="/home/user"
type=BPRM_FCAPS msg=audit(1359968226.231:575): fver=0 fp=0000000000000000 fi=0000000000000000 fe=0 old_pp=0000000000000000 old_pi=0000000000000000 old_pe=0000000000000000 new_pp=ffffffffffffffff new_pi=0000000000000000 new_pe=ffffffffffffffff
type=EXECVE msg=audit(1359968226.231:575): argc=3 a0="sudo" a1="ls" a2="/"
type=SYSCALL msg=audit(1359968226.231:575): arch=c000003e syscall=59 success=yes exit=0 a0=7fff327ecab0 a1=7fd330e1b958 a2=17cc8d0 a3=7fff327ec670 items=2 ppid=3933 pid=26569 auid=1000 uid=1000 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=1 comm="sudo" exe="/usr/bin/sudo" key=(null)
----
time->Mon Feb  4 09:57:09 2013
type=PATH msg=audit(1359968229.523:578): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968229.523:578): item=0 name="/bin/su" inode=44 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968229.523:578):  cwd="/home/user"
type=EXECVE msg=audit(1359968229.523:578): argc=2 a0="su" a1="-"
type=SYSCALL msg=audit(1359968229.523:578): arch=c000003e syscall=59 success=yes exit=0 a0=1ceec48 a1=1cef7c8 a2=1cf4750 a3=7fff083bd920 items=2 ppid=26611 pid=26612 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="su" exe="/bin/su" key=(null)
----
time->Mon Feb  4 09:57:09 2013
type=PATH msg=audit(1359968229.519:577): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968229.519:577): item=0 name="/usr/bin/sudo" inode=530900 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968229.519:577):  cwd="/home/user"
type=BPRM_FCAPS msg=audit(1359968229.519:577): fver=0 fp=0000000000000000 fi=0000000000000000 fe=0 old_pp=0000000000000000 old_pi=0000000000000000 old_pe=0000000000000000 new_pp=ffffffffffffffff new_pi=0000000000000000 new_pe=ffffffffffffffff
type=EXECVE msg=audit(1359968229.519:577): argc=3 a0="sudo" a1="su" a2="-"
type=SYSCALL msg=audit(1359968229.519:577): arch=c000003e syscall=59 success=yes exit=0 a0=7fff327ecab0 a1=7fd330e1b958 a2=17cc8d0 a3=7fff327ec670 items=2 ppid=3933 pid=26611 auid=1000 uid=1000 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=1 comm="sudo" exe="/usr/bin/sudo" key=(null)
----
time->Mon Feb  4 09:57:09 2013
type=PATH msg=audit(1359968229.543:585): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968229.543:585): item=0 name="/bin/bash" inode=6941 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968229.543:585):  cwd="/root"
type=EXECVE msg=audit(1359968229.543:585): argc=1 a0="-su"
type=SYSCALL msg=audit(1359968229.543:585): arch=c000003e syscall=59 success=yes exit=0 a0=13695a0 a1=7fffce08a3e0 a2=135a030 a3=7fffce08c200 items=2 ppid=26612 pid=26622 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="bash" exe="/bin/bash" key=(null)
----
time->Mon Feb  4 09:57:11 2013
type=PATH msg=audit(1359968231.663:594): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968231.663:594): item=0 name="/bin/ls" inode=2117 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968231.663:594):  cwd="/root"
type=EXECVE msg=audit(1359968231.663:594): argc=3 a0="ls" a1="--color=auto" a2="/etc"
type=SYSCALL msg=audit(1359968231.663:594): arch=c000003e syscall=59 success=yes exit=0 a0=7fff8c709950 a1=7f91a12149d8 a2=1194c50 a3=7fff8c709510 items=2 ppid=26622 pid=26661 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="ls" exe="/bin/ls" key=(null)

The auid column contains the calling user's uid, which allows you filter for commands run by this user with

 ausearch -ua 1000

This will even list commands the user ran as root.

Sources:

share|improve this answer
    
+50 This answer seems the most comprehensive, although I am a bit disappointed that it ended up to be rather complicated. Thank you for your contribution. –  grassroot Feb 7 '13 at 21:09

Remember that sudo itself logs all sudo commands in the syslog, so all priv'd users should be educated to not simply sudo to get a root shell but to :

sudo command p1 p2 ... pn

The problem with this or any approach I have thought of is that as the root user, it is quite difficult to prevent a user from evading any specific type of logging. Thus anything thing you try will be < 100% I am sorry to say.

Education, documentation, enforcement and above all trust is what is necessary.

share|improve this answer
1  
I understand that nothing will be perfect, but we will never be able to get everyone to work like you describe. These are sysadmins we're talking about ;) –  Soviero Jan 20 '13 at 16:46
1  
Not true....at least two very large companies I have personally worked with consisting of a large number of system administrators have this very policy in place! Again, with education and enforcement it works. –  mdpc Jan 20 '13 at 17:48
    
mdpc is 100% correct. This is exactly what the sudo command is for. I'm in a shop of ten sysadmins with hundreds of hosts, and we use individual sudo commands for everything - there's a specific policy that forbids becoming root via su -. It's the only reasonable way to ensure admin operations are properly audited. –  Jeff Albert Feb 2 '13 at 0:08
    
-1 Education will never do it. We live in outsourced world where you are just one of your sysadmins' many customers. –  grassroot Feb 7 '13 at 21:20

A number of establishments actually prohibit the use of auditd because it is resource intensive and may result in an opportunity for denial of service attacks.

One solution is to configure the latest Korn shell (ksh-93, see http://kornshell.com/ for details) to log all commands executed as root to a remote syslog server, and then to require by policy that, except in emergency situations, sysadmins log on with personal accounts and execute the enhanced Korn shell via sudo. Examination of the logs can detect when an admin launches another shell from the approved shell in order to cover their tracks, and the SA can then be educated as necessary.

share|improve this answer

Since version 2.0.0 snoopy is able to log arbitrary environmental variable.

However, recent contribution pointed out that logging owner of tty is fairly effective and elegant answer to the question of "Who executed that command, as root?".

Disclosure: I am snoopy maintainer.

share|improve this answer

Not that there's anything wrong with any of the other answers thus far, but if you decide that sudo's logging via syslog is satisfactory, may I suggest a wrinkle: log it via the network to a remote audit host.

That gets around the problem of "now I have become root, I can remove any trace of my malfeasance from the logs". You may now be root on the local box, but you can't call that log packet back from the network, and you (presumably) don't have root privileges on the remote audit host.

I've been doing this with some of the networks I manage for years, and it has two other signal advantages:

Firstly, there's one place on the network to go to check all syslogs, which allows much easier correlation of incidents, and so is a one-stop shop for investigations like "When juno was complaining that NFS server hera was not responding, was anyone else complaining about the same thing at the same time? If so, hera is likely to be the problem, let's see what she logged; if not, juno's network connection is more suspect, let's see what else juno logged at that time.".

Secondly, syslog log rotation becomes easier: you don't keep copies of logs on local hosts for more than a few days, but you make sure the audit server has huge amounts of disc space, and keep all syslogs there for several years. Plus if, say, you want to write them to WORM media for eg forensic audit purposes, you only have to buy one WORM drive.

share|improve this answer

I was once faced with the same problem and had to come up with a quick and dirty solution - each sudo user will have their own history file once they run the command sudo -i

In /root/.bashrc I added the following line -

 export HISTFILE=/root/.bash_history-$SUDO_USER
 export HISTTIMEFORMAT="%F %T "

So every user who sudos to root will have a history file .bash_history-username.

Another method -

Add the following code to /root/.bashrc and it will append the username, sudo-user, and the command to the log file, where ever the notice level is set, most likely /var/log/messages.

function log2syslog
{
   declare COMMAND
   COMMAND=$(fc -ln -0)
   logger -p local1.notice -t bash -i -- "${USER}:${SUDO_USER}:${COMMAND}"
}
trap log2syslog DEBUG

Credit to - http://backdrift.org/logging-bash-history-to-syslog-using-traps

share|improve this answer
    
Nice approach, although not quite what was asked. I'd like to see an auditd or similar solution. –  grassroot Jan 31 '13 at 20:13
    
ok i have updated it to rely on trap method. –  Daniel t. Jan 31 '13 at 20:24
1  
And for legitimate users, this works. But if that account was cracked, the cracker could quickly disable bash history by running /bin/sh, unset HISTFILE or /bin/bash --norc. –  Stefan Lasiewski Feb 1 '13 at 21:01

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.