I work at a firm with a certain amount of legacy batch-processes.

Some of that stuff is really leaky when it comes to database connections.

  • I need to log every process in the system, which opens a database connection.

Strace is not a option, there are too many processes, they are too short lived, I need to audit multiple boxes.

IPtables is not an option, you can match on process path/user but you cannot log that information (to the best of my knowledge)

The usual tools such as lsof/netstat etc, only contain information about processes that are alive and what connections they are using.

These batch jobs on the other hand, are already dead - so I cannot correlate process with socket.

So this is as good a time as any to learn about systemtap. Or write a custom Kernel module. But I don't have the expertise/time to start hacking into the syscall table, and besides, I want to learn systemtap because it seems rather very useful tool.

My end goal is to write a probe which -

  • Prints out as much information as possible, about every process that connects to MySQL locally.

But the devil is in the details.. I can't get it installed on Debian (wheezy) (at all), and I'm getting wierd errors on Ubuntu 12 (Precise).

On Ubuntu, a basic systemtap probe works just fine (these are basic copy and paste examples).

 #! /usr/bin/env stap
 probe begin { println("hello world") exit () }

It produces the expected output.

sudo stap -v stapinit.stp
Pass 1: parsed user script and 76 library script(s) using 22792virt/13512res/2224shr kb, in 90usr/0sys/93real ms.
Pass 2: analyzed script: 1 probe(s), 1 function(s), 0 embed(s), 0 global(s) using 23056virt/14000res/2304shr kb, in 10usr/0sys/2real ms.
Pass 3: translated to C into "/tmp/stap2ntfcM/stap_32acf6b6a3f643d9444c9c8339e390d8_687.c" using 23056virt/14332res/2584shr kb, in 0usr/0sys/0real ms.
Pass 4: compiled C into "stap_32acf6b6a3f643d9444c9c8339e390d8_687.ko" in 1290usr/390sys/1902real ms.
Pass 5: starting run.
hello world
Pass 5: run completed in 10usr/40sys/596real ms.

But a more complex probe just dies, for example:

# Show sockets setting options

# Return enabled or disabled based on value of optval
function getstatus(optval)
    if ( optval == 1 )
        return "enabling"
        return "disabling"

probe begin
        print ("\nChecking for apps setting socket options\n")

# Set a socket option
probe tcp.setsockopt
    status = getstatus(user_int($optval))
        printf ("  App '%s' (PID %d) is %s socket option %s... ", execname(), pid(), status, optstr)

# Check setting the socket option worked
probe tcp.setsockopt.return
    if ( ret == 0 )
        printf ("success")
        printf ("failed")
    printf ("\n")

probe end
        print ("\nClosing down\n")

Produces -

Pass 1: parsed user script and 76 library script(s) using 22812virt/13660res/2224shr kb, in 90usr/10sys/120real ms.
semantic error: missing i386 kernel/module debuginfo under '/lib/modules/3.2.0-27-generic-pae/build' while resolving probe point kernel.function("tcp_setsockopt")
semantic error: no match while resolving probe point tcp.setsockopt
semantic error: missing i386 kernel/module debuginfo under '/lib/modules/3.2.0-27-generic-pae/build' while resolving probe point kernel.function("tcp_setsockopt").return
semantic error: no match while resolving probe point tcp.setsockopt.return
Pass 2: analyzed script: 2 probe(s), 1 function(s), 0 embed(s), 0 global(s) using 23136virt/14424res/2540shr kb, in 70usr/580sys/1255real ms.
Pass 2: analysis failed.  Try again with another '--vp 01' option.

It seems I am missing a dependency, but I've pretty much installed the kitchen sink.

sudo apt-get install elfutils
sudo apt-get install linux-headers-generic gcc libcap-dev
sudo apt-get install systemtap-sdt-dev
sudo apt-get install systemtap systemtap-doc linux-image linux-headers-generic-pae
  • Update 1

The packages suggested by apt-get install systemtap may be somewhat misleading.

linux-headers-virtual , linux-image , linux-source , and linux-tools are virtual packages so will track whatever kernel is currently upgraded to.

sudo apt-get install linux-headers-virtual linux-image linux-source linux-tools 

For good measure, I've also installed fdutils , and kernel-package

sudo apt-get install fdutils kernel-package

Will try this and report back.

  • Update 2

Still broken, same error, this actually the third system I've tried to install it upon.

I have also found the following Ubuntu bug, perhaps the package is bad?


  • Update 3

It seems the problem on Ubuntu is that they don't build a kernel debug symbols package, or at least distribute it in a standard way. My conclusion is that the systemtap package is broken on Ubuntu, in so far as it doesn't install crucial dependencies.


On my Debian box, I have a different problem, testing is enabled - thus gcc has upgraded to 4.6, and linux-headers package requires gcc 4.3

 root@datasift:~# apt-get install systemtap linux-image-`uname -r`-dbg linux-headers-`uname -r`
 Reading package lists... Done
 Building dependency tree       
 Reading state information... Done
 systemtap is already the newest version.
 Some packages could not be installed. This may mean that you have
 requested an impossible situation or if you are using the unstable
 distribution that some required packages have not yet been created
 or been moved out of Incoming.
 The following information may help to resolve the situation:

 The following packages have unmet dependencies:
  linux-headers-2.6.32-5-amd64 : Depends: gcc-4.3 but it is not going to be installed
 E: Broken packages
 root@datasift:~# uname -a
 Linux datasift.sentimentmetrics.com 2.6.32-5-amd64 #1 SMP Sun May 6 04:00:17 UTC 2012 x86_64 GNU/Linux
  • Conclusion

So, in conclusion - it is possible that systemtap works ok on Debian, highly unlikely that it works on Ubuntu, and I don't have time to continue. I wish I was running BSD/Solaris or something with a standard instrumentation interface, but like it or not Linux is the god of commodity.

Update 4

In the meantime, I have managed to use the Linux Auditing Framework, inspired by the following posting:

how i can identify which process is making UDP traffic on linux?

My auditd command:

 auditctl -a exit,always -F arch=b64 -F a0=2 -S socket

Produces log messages in the following format:

  type=SYSCALL msg=audit(1344346149.672:1670): arch=c000003e syscall=41 success=yes exit=5 a0=2 a1=1 a2=6 a3=1999999999999999 items=0 ppid=1 pid=29674 auid=0 uid=1003 gid=1003 euid=1003 suid=1003 fsuid=1003 egid=1003 sgid=1003 fsgid=1003 tty=(none) ses=124 comm="php5" exe="/usr/bin/php5" key=(null)

Which are nice, but ultimately useless as they don't contain the program arguments.

What I really need (at a minimum) is something that can capture the arguments, for example:

 exe="/usr/bin/php5" args="/Administraitor/learn-to-code-hehe.php"   

Ideally, I want to be able to filter by host and port as well.

2 Answers 2


I've managed to get it working on Ubuntu, Debian is another matter.

I've created repository with installation/monitoring scripts


The scripts also provide the program arguments, so for a naughty PHP script it is now possible to identify the wrongdoer.


Not so much an answer, but a word of warning: SystemTap can be confusing, sometimes even just plain buggy, other times non-intuitive. Still, it's a good tool to have, just be careful with it and cross check your results with other tools, too.

This blog entry by Brendan Gregg provides a very deep analysis to the weird world of SystemTap. Of course, since the blog writer usually uses dtrace, his point of view might be biased. Anyway, the post is very well done and made me think about the reliability of SystemTap again. The blog post also has some code examples which might help you to solve your problem(s).

Here's a quote from the blog post:

While using SystemTap, I’ve been keeping notes of what I’ve been doing, including what worked and what didn’t, and how I fixed problems. It’s proven a handy reference.

In some recent threads about DTrace, people have asked how it compares to SystemTap – with some commenting that they haven’t had time to study either. I’ve been encouraged to post about my experiences, which is easy to do from my notes. I could (and probably should) get some of these into the SystemTap bug database, too.

What I’m sharing are various experiences of using SystemTap, including times where I made mistakes because I didn’t know what I was doing. I’ll begin with a narrative about tracing disk I/O, which connects together various experiences. After that it gets a little discontiguous, clipped from various notes. I’ll try to keep this technical, positive, and constructive. Perhaps this will help the SystemTap project itself, to see what a user (with a strong Dynamic Tracing and kernel engineering background) struggles with.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .