I work at a firm with a certain amount of legacy batch-processes.
Some of that stuff is really leaky when it comes to database connections.
- I need to log every process in the system, which opens a database connection.
Strace is not a option, there are too many processes, they are too short lived, I need to audit multiple boxes.
IPtables is not an option, you can match on process path/user but you cannot log that information (to the best of my knowledge)
The usual tools such as lsof/netstat etc, only contain information about processes that are alive and what connections they are using.
These batch jobs on the other hand, are already dead - so I cannot correlate process with socket.
So this is as good a time as any to learn about systemtap. Or write a custom Kernel module. But I don't have the expertise/time to start hacking into the syscall table, and besides, I want to learn systemtap because it seems rather very useful tool.
My end goal is to write a probe which -
- Prints out as much information as possible, about every process that connects to MySQL locally.
But the devil is in the details.. I can't get it installed on Debian (wheezy) (at all), and I'm getting wierd errors on Ubuntu 12 (Precise).
On Ubuntu, a basic systemtap probe works just fine (these are basic copy and paste examples).
#! /usr/bin/env stap
probe begin { println("hello world") exit () }
It produces the expected output.
sudo stap -v stapinit.stp
Pass 1: parsed user script and 76 library script(s) using 22792virt/13512res/2224shr kb, in 90usr/0sys/93real ms.
Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) using 23056virt/14000res/2304shr kb, in 10usr/0sys/2real ms.
Pass 3: translated to C into "/tmp/stap2ntfcM/stap_32acf6b6a3f643d9444c9c8339e390d8_687.c" using 23056virt/14332res/2584shr kb, in 0usr/0sys/0real ms.
Pass 4: compiled C into "stap_32acf6b6a3f643d9444c9c8339e390d8_687.ko" in 1290usr/390sys/1902real ms.
Pass 5: starting run.
hello world
Pass 5: run completed in 10usr/40sys/596real ms.
But a more complex probe just dies, for example:
# Show sockets setting options
# Return enabled or disabled based on value of optval
function getstatus(optval)
{
if ( optval == 1 )
return "enabling"
else
return "disabling"
}
probe begin
{
print ("\nChecking for apps setting socket options\n")
}
# Set a socket option
probe tcp.setsockopt
{
status = getstatus(user_int($optval))
printf (" App '%s' (PID %d) is %s socket option %s... ", execname(), pid(), status, optstr)
}
# Check setting the socket option worked
probe tcp.setsockopt.return
{
if ( ret == 0 )
printf ("success")
else
printf ("failed")
printf ("\n")
}
probe end
{
print ("\nClosing down\n")
}
Produces -
Pass 1: parsed user script and 76 library script(s) using 22812virt/13660res/2224shr kb, in 90usr/10sys/120real ms.
semantic error: missing i386 kernel/module debuginfo under '/lib/modules/3.2.0-27-generic-pae/build' while resolving probe point kernel.function("tcp_setsockopt")
semantic error: no match while resolving probe point tcp.setsockopt
semantic error: missing i386 kernel/module debuginfo under '/lib/modules/3.2.0-27-generic-pae/build' while resolving probe point kernel.function("tcp_setsockopt").return
semantic error: no match while resolving probe point tcp.setsockopt.return
Pass 2: analyzed script: 2 probe(s), 1 function(s), 0 embed(s), 0 global(s) using 23136virt/14424res/2540shr kb, in 70usr/580sys/1255real ms.
Pass 2: analysis failed. Try again with another '--vp 01' option.
It seems I am missing a dependency, but I've pretty much installed the kitchen sink.
sudo apt-get install elfutils
sudo apt-get install linux-headers-generic gcc libcap-dev
sudo apt-get install systemtap-sdt-dev
sudo apt-get install systemtap systemtap-doc linux-image linux-headers-generic-pae
- Update 1
The packages suggested by apt-get install systemtap may be somewhat misleading.
linux-headers-virtual , linux-image , linux-source , and linux-tools are virtual packages so will track whatever kernel is currently upgraded to.
sudo apt-get install linux-headers-virtual linux-image linux-source linux-tools
For good measure, I've also installed fdutils , and kernel-package
sudo apt-get install fdutils kernel-package
Will try this and report back.
- Update 2
Still broken, same error, this actually the third system I've tried to install it upon.
I have also found the following Ubuntu bug, perhaps the package is bad?
https://bugs.launchpad.net/ubuntu/+source/systemtap/+bug/824105
- Update 3
It seems the problem on Ubuntu is that they don't build a kernel debug symbols package, or at least distribute it in a standard way. My conclusion is that the systemtap package is broken on Ubuntu, in so far as it doesn't install crucial dependencies.
https://bugs.launchpad.net/ubuntu/+source/systemtap/+bug/106957
On my Debian box, I have a different problem, testing is enabled - thus gcc has upgraded to 4.6, and linux-headers package requires gcc 4.3
root@datasift:~# apt-get install systemtap linux-image-`uname -r`-dbg linux-headers-`uname -r`
Reading package lists... Done
Building dependency tree
Reading state information... Done
systemtap is already the newest version.
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:
The following packages have unmet dependencies:
linux-headers-2.6.32-5-amd64 : Depends: gcc-4.3 but it is not going to be installed
E: Broken packages
root@datasift:~# uname -a
Linux datasift.sentimentmetrics.com 2.6.32-5-amd64 #1 SMP Sun May 6 04:00:17 UTC 2012 x86_64 GNU/Linux
- Conclusion
So, in conclusion - it is possible that systemtap works ok on Debian, highly unlikely that it works on Ubuntu, and I don't have time to continue. I wish I was running BSD/Solaris or something with a standard instrumentation interface, but like it or not Linux is the god of commodity.
Update 4
In the meantime, I have managed to use the Linux Auditing Framework, inspired by the following posting:
how i can identify which process is making UDP traffic on linux?
My auditd command:
auditctl -a exit,always -F arch=b64 -F a0=2 -S socket
Produces log messages in the following format:
type=SYSCALL msg=audit(1344346149.672:1670): arch=c000003e syscall=41 success=yes exit=5 a0=2 a1=1 a2=6 a3=1999999999999999 items=0 ppid=1 pid=29674 auid=0 uid=1003 gid=1003 euid=1003 suid=1003 fsuid=1003 egid=1003 sgid=1003 fsgid=1003 tty=(none) ses=124 comm="php5" exe="/usr/bin/php5" key=(null)
Which are nice, but ultimately useless as they don't contain the program arguments.
What I really need (at a minimum) is something that can capture the arguments, for example:
exe="/usr/bin/php5" args="/Administraitor/learn-to-code-hehe.php"
Ideally, I want to be able to filter by host and port as well.
