82

It is company policy for admins to login to the servers via a personal username, and then run sudo -i to become root. Upon running sudo -i, sudo will create an environmental variable called SUDO_USER, which contains the original user's username.

Is there a way to log ALL commands within syslog with something akin to the following syntax:

${TIME/DATE STAMP}: [${REAL_USER}|${SUDO_USER}]: ${CMD}

An example entry would be:

Sat Jan 19 22:28:46 CST 2013: [root|ksoviero]: yum install random-pkg

Obviously it doesn't have to be exactly the above syntax, it just has to include a minimum of the real user (eg. root), the sudo user (eg. ksoviero), and the full command that was run (eg. yum install random-pkg).

I've already tried snoopy, but it did not include the SUDO_USER variable.

7
  • 14
    You need auditd. Jan 20, 2013 at 4:56
  • 1
    This is a brief intro to auditd Jan 20, 2013 at 7:37
  • 1
    Could somebody please post this as an answer? Please include how I would strictly log all commands run by users. The "brief intro to auditd" was useful, but it didn't include anything related to logging actual commands (as far as I could tell anyway).
    – Soviero
    Jan 20, 2013 at 16:45
  • 1
    Ok, I've started playing with auditd, and while I’ve gotten it to log all commands being run, it doesn't include the SUDO_USER variable (or equivalent information), and I can't find a way to include it. Any help would be greatly appreciated!
    – Soviero
    Jan 21, 2013 at 2:02
  • 3
    And what will the company do with this record of all commands entered by administrators?
    – ewwhite
    Feb 3, 2013 at 19:03

7 Answers 7

107
+50

Update: 2 more things that have popped up in the comments and in follow-up questions:

  • Using auditd this way will dramatically increase your log volume, especially if the system is heavily in use via commandline. Adjust your log retention policy.
  • Auditd logs on the host where they are created are just as secure as other files on the same box. Forward your logs to a remote log collection server like ELK or Graylog to preserve your logs' integrity. Plus, adding to the point above, it allows to more aggressively delete old logs.

As was suggested by Michael Hampton, auditd is the correct tool for the job here.

I tested this on an Ubuntu 12.10 installation, so your mileage may vary on other systems.

  • Install auditd:

    apt-get install auditd

  • Add these 2 lines to /etc/audit/audit.rules:

    -a exit,always -F arch=b64 -F euid=0 -S execve
    -a exit,always -F arch=b32 -F euid=0 -S execve

These will track all commands run by root (euid=0). Why two rules? The execve syscall must be tracked in both 32 and 64 bit code.

  • To get rid of auid=4294967295 messages in logs, add audit=1 to the kernel's cmdline (by editing /etc/default/grub)

  • Place the line

    session required pam_loginuid.so

in all PAM config files that are relevant to login (/etc/pam.d/{login,kdm,sshd}), but not in the files that are relevant to su or sudo. This will allow auditd to get the calling user's uid correctly when calling sudo or su.

  • Restart your system now.

  • Let's login and run some commands:

    $ id -u
    1000
    $ sudo ls /
    bin  boot  data  dev  etc  home  initrd.img  initrd.img.old  lib  lib32  lib64  lost+found  media  mnt  opt  proc  root  run  sbin  scratch  selinux  srv  sys  tmp  usr  var  vmlinuz  vmlinuz.old
    $ sudo su -
    # ls /etc
    [...]

This will yield something like this in /var/log/audit/auditd.log:

----
time->Mon Feb  4 09:57:06 2013
type=PATH msg=audit(1359968226.239:576): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968226.239:576): item=0 name="/bin/ls" inode=2117 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968226.239:576):  cwd="/home/user"
type=EXECVE msg=audit(1359968226.239:576): argc=2 a0="ls" a1="/"
type=SYSCALL msg=audit(1359968226.239:576): arch=c000003e syscall=59 success=yes exit=0 a0=10cfc48 a1=10d07c8 a2=10d5750 a3=7fff2eb2d1f0 items=2 ppid=26569 pid=26570 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="ls" exe="/bin/ls" key=(null)
----
time->Mon Feb  4 09:57:06 2013
type=PATH msg=audit(1359968226.231:575): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968226.231:575): item=0 name="/usr/bin/sudo" inode=530900 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968226.231:575):  cwd="/home/user"
type=BPRM_FCAPS msg=audit(1359968226.231:575): fver=0 fp=0000000000000000 fi=0000000000000000 fe=0 old_pp=0000000000000000 old_pi=0000000000000000 old_pe=0000000000000000 new_pp=ffffffffffffffff new_pi=0000000000000000 new_pe=ffffffffffffffff
type=EXECVE msg=audit(1359968226.231:575): argc=3 a0="sudo" a1="ls" a2="/"
type=SYSCALL msg=audit(1359968226.231:575): arch=c000003e syscall=59 success=yes exit=0 a0=7fff327ecab0 a1=7fd330e1b958 a2=17cc8d0 a3=7fff327ec670 items=2 ppid=3933 pid=26569 auid=1000 uid=1000 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=1 comm="sudo" exe="/usr/bin/sudo" key=(null)
----
time->Mon Feb  4 09:57:09 2013
type=PATH msg=audit(1359968229.523:578): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968229.523:578): item=0 name="/bin/su" inode=44 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968229.523:578):  cwd="/home/user"
type=EXECVE msg=audit(1359968229.523:578): argc=2 a0="su" a1="-"
type=SYSCALL msg=audit(1359968229.523:578): arch=c000003e syscall=59 success=yes exit=0 a0=1ceec48 a1=1cef7c8 a2=1cf4750 a3=7fff083bd920 items=2 ppid=26611 pid=26612 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="su" exe="/bin/su" key=(null)
----
time->Mon Feb  4 09:57:09 2013
type=PATH msg=audit(1359968229.519:577): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968229.519:577): item=0 name="/usr/bin/sudo" inode=530900 dev=08:01 mode=0104755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968229.519:577):  cwd="/home/user"
type=BPRM_FCAPS msg=audit(1359968229.519:577): fver=0 fp=0000000000000000 fi=0000000000000000 fe=0 old_pp=0000000000000000 old_pi=0000000000000000 old_pe=0000000000000000 new_pp=ffffffffffffffff new_pi=0000000000000000 new_pe=ffffffffffffffff
type=EXECVE msg=audit(1359968229.519:577): argc=3 a0="sudo" a1="su" a2="-"
type=SYSCALL msg=audit(1359968229.519:577): arch=c000003e syscall=59 success=yes exit=0 a0=7fff327ecab0 a1=7fd330e1b958 a2=17cc8d0 a3=7fff327ec670 items=2 ppid=3933 pid=26611 auid=1000 uid=1000 gid=1000 euid=0 suid=0 fsuid=0 egid=1000 sgid=1000 fsgid=1000 tty=pts0 ses=1 comm="sudo" exe="/usr/bin/sudo" key=(null)
----
time->Mon Feb  4 09:57:09 2013
type=PATH msg=audit(1359968229.543:585): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968229.543:585): item=0 name="/bin/bash" inode=6941 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968229.543:585):  cwd="/root"
type=EXECVE msg=audit(1359968229.543:585): argc=1 a0="-su"
type=SYSCALL msg=audit(1359968229.543:585): arch=c000003e syscall=59 success=yes exit=0 a0=13695a0 a1=7fffce08a3e0 a2=135a030 a3=7fffce08c200 items=2 ppid=26612 pid=26622 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="bash" exe="/bin/bash" key=(null)
----
time->Mon Feb  4 09:57:11 2013
type=PATH msg=audit(1359968231.663:594): item=1 name=(null) inode=668682 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=PATH msg=audit(1359968231.663:594): item=0 name="/bin/ls" inode=2117 dev=08:01 mode=0100755 ouid=0 ogid=0 rdev=00:00
type=CWD msg=audit(1359968231.663:594):  cwd="/root"
type=EXECVE msg=audit(1359968231.663:594): argc=3 a0="ls" a1="--color=auto" a2="/etc"
type=SYSCALL msg=audit(1359968231.663:594): arch=c000003e syscall=59 success=yes exit=0 a0=7fff8c709950 a1=7f91a12149d8 a2=1194c50 a3=7fff8c709510 items=2 ppid=26622 pid=26661 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="ls" exe="/bin/ls" key=(null)

The auid column contains the calling user's uid, which allows you filter for commands run by this user with

 ausearch -ua 1000

This will even list commands the user ran as root.

Sources:

4
  • +50 This answer seems the most comprehensive, although I am a bit disappointed that it ended up to be rather complicated. Thank you for your contribution.
    – grassroot
    Feb 7, 2013 at 21:09
  • Be forewarned that these changes can greatly increase one's log volume into /var/log/audit/audit.log. My log volume to this file more than doubled after adding the twoexecve lines to audit.rules
    – JDS
    Jan 18, 2016 at 17:17
  • 2
    Just a note: in some distros (e.g. Ubuntu 18.04+ at least) you should not modify /etc/audit/audit.rules but instead modify /etc/audit/rules.d/audit.rules. The /etc/audit/audit.rules file is re-created from the files contained in the subdirectory so changes to it will get lost. Nov 12, 2019 at 10:15
  • It's also a really good idea to forward the logging to remote audit server (e.g. syslog or graylog) that hopefully doesn't allow remote administration at all. If that server requires physical access to login and manipulate the logs, the collected logs should be pretty trustworthy. Any logs on the local system can be manipulated by root and the root can disable logging (to local or remote storage) at will in all cases but the command used to disable logging is logged. Nov 25, 2021 at 10:32
11

Remember that sudo itself logs all sudo commands in the syslog, so all priv'd users should be educated to not simply sudo to get a root shell but to :

sudo command p1 p2 ... pn

The problem with this or any approach I have thought of is that as the root user, it is quite difficult to prevent a user from evading any specific type of logging. Thus anything thing you try will be < 100% I am sorry to say.

Education, documentation, enforcement and above all trust is what is necessary.

4
  • 3
    I understand that nothing will be perfect, but we will never be able to get everyone to work like you describe. These are sysadmins we're talking about ;)
    – Soviero
    Jan 20, 2013 at 16:46
  • 4
    Not true....at least two very large companies I have personally worked with consisting of a large number of system administrators have this very policy in place! Again, with education and enforcement it works.
    – mdpc
    Jan 20, 2013 at 17:48
  • 3
    mdpc is 100% correct. This is exactly what the sudo command is for. I'm in a shop of ten sysadmins with hundreds of hosts, and we use individual sudo commands for everything - there's a specific policy that forbids becoming root via su -. It's the only reasonable way to ensure admin operations are properly audited. Feb 2, 2013 at 0:08
  • 4
    -1 Education will never do it. We live in outsourced world where you are just one of your sysadmins' many customers.
    – grassroot
    Feb 7, 2013 at 21:20
10

I was once faced with the same problem and had to come up with a quick and dirty solution - each sudo user will have their own history file once they run the command sudo -i

In /root/.bashrc I added the following line -

 export HISTFILE=/root/.bash_history-$SUDO_USER
 export HISTTIMEFORMAT="%F %T "

So every user who sudos to root will have a history file .bash_history-username.

Another method -

Add the following code to /root/.bashrc and it will append the username, sudo-user, and the command to the log file, where ever the notice level is set, most likely /var/log/messages.

function log2syslog
{
   declare COMMAND
   COMMAND=$(fc -ln -0)
   logger -p local1.notice -t bash -i -- "${USER}:${SUDO_USER}:${COMMAND}"
}
trap log2syslog DEBUG

Credit to - http://backdrift.org/logging-bash-history-to-syslog-using-traps

3
  • Nice approach, although not quite what was asked. I'd like to see an auditd or similar solution.
    – grassroot
    Jan 31, 2013 at 20:13
  • ok i have updated it to rely on trap method.
    – Daniel t.
    Jan 31, 2013 at 20:24
  • 3
    And for legitimate users, this works. But if that account was cracked, the cracker could quickly disable bash history by running /bin/sh, unset HISTFILE or /bin/bash --norc. Feb 1, 2013 at 21:01
3

A number of establishments actually prohibit the use of auditd because it is resource intensive and may result in an opportunity for denial of service attacks.

One solution is to configure the latest Korn shell (ksh-93, see http://kornshell.com/ for details) to log all commands executed as root to a remote syslog server, and then to require by policy that, except in emergency situations, sysadmins log on with personal accounts and execute the enhanced Korn shell via sudo. Examination of the logs can detect when an admin launches another shell from the approved shell in order to cover their tracks, and the SA can then be educated as necessary.

3

Sudo has something called sudoreplay when enabled sessions is logged and can be replayed later, works similar as the script command that make a typescript of terminal session that later can be replayed with the scriptreplay command.

2

Not that there's anything wrong with any of the other answers thus far, but if you decide that sudo's logging via syslog is satisfactory, may I suggest a wrinkle: log it via the network to a remote audit host.

That gets around the problem of "now I have become root, I can remove any trace of my malfeasance from the logs". You may now be root on the local box, but you can't call that log packet back from the network, and you (presumably) don't have root privileges on the remote audit host.

I've been doing this with some of the networks I manage for years, and it has two other signal advantages:

Firstly, there's one place on the network to go to check all syslogs, which allows much easier correlation of incidents, and so is a one-stop shop for investigations like "When juno was complaining that NFS server hera was not responding, was anyone else complaining about the same thing at the same time? If so, hera is likely to be the problem, let's see what she logged; if not, juno's network connection is more suspect, let's see what else juno logged at that time.".

Secondly, syslog log rotation becomes easier: you don't keep copies of logs on local hosts for more than a few days, but you make sure the audit server has huge amounts of disc space, and keep all syslogs there for several years. Plus if, say, you want to write them to WORM media for eg forensic audit purposes, you only have to buy one WORM drive.

2

Since version 2.0.0, Snoopy is able to log arbitrary environmental variables. Here is a link to Snoopy's example configuration file, where you can find all about the %{env:NAME_OF_MY_ENV_VAR_I_WANT_LOGGED} syntax.

However, recent contribution pointed out that logging the owner of the tty is a fairly effective and elegant answer to the question of "Who executed that command, as root?".

Disclosure: I am the maintainer of Snoopy.

Edit: Changed link to point directly to relevant documentation (apparently, certain people can't find it in the repo) + slight rewording.

2
  • Please provide instructions on how to set it up according to OP's requirements, rather than a simple link. Thanks. Aug 15, 2017 at 15:00
  • 2
    -1. This is just a plug for snoopy. You followed the disclosure, but you still didn't answer the question in your post; you just linked to your project. Jan 19, 2018 at 17:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .