Sign up ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Running tcpdump on local connections to an apache server, I found TCP connections being established and closed immediately every 2 seconds. How do I find which process is responsible for these? netstat -ctp did not help, the connections were too fast and the process identifier is not displayed for TIME_WAIT ones.

They turned out to be haproxy probes, which I could verify with strace, but I still do not know any way to pinpoint haproxy in the first place.

share|improve this question

2 Answers 2

up vote 7 down vote accepted

You can use the auditd framework for these kind of things. They're not very "user friendly" or intuitive, so requires a little bit of digging around on your part.

First make sure you have auditd installed, running and that your kernel supports it.
For Ubuntu you can install it with apt-get install auditd for example.

Then you add a policy for audit to monitor all connect syscalls like this:

auditctl -a exit,always -F arch=b64 -S connect -k MYCONNECT

If you are using a 32-bit installation of Linux you have to change b64 to b32.

This command will insert a policy to the audit framework, and any connect() syscalls will now be logged to your audit logfiles (usually /var/log/audit/audit.log) for you to look at.

For example, a connection with netcat to port 80 will result in something like this:

type=SYSCALL msg=audit(1326872512.453:12752): arch=c000003e syscall=42 success=no exit=-115 a0=3 a1=24e8fa0 a2=10 a3=7fff07a44cd0 items=0 ppid=5675 pid=7270 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts4 ses=4294967295 comm="nc" exe="/bin/nc.openbsd" key="MYCONNECT"
type=SOCKADDR msg=audit(1326872512.453:12752): saddr=02000050AE84E16A0000000000000000

Here you can see that the /bin/nc.openbsd application initiated a connect() call, if you get lots of connect calls and only want to grep out a certain ip or port you have to do some conversion. The SOCKADDR line contains a saddr argument, it begins with 0200 followed by the port number in hexadecimal (0050) which means 80, and then the IP in hex (AE84E16A) which is's IP of

The audit framework can generate a lot of logs, so remember to disable it when you've accomplished your mission. To disable the above policy, simply replace -a with -d as such:

auditctl -d exit,always -F arch=b64 -S connect -k MYCONNECT

Good documentation on the auditd framework:

Convert IP adresses to/from hex, dec, binary, etc at:

General hex/dec converter:

A Brief Introduction to auditd, from the IT Security Stack Exchange.

Edit 1:
Another quick'n'dirty (swedish: fulhack) way to do it is to create a fast loop that dumps the connection data to you, like this:

while true;do
  ss -ntap -o state established '( dport = :80 )'
  sleep 1

This command uses the ss command (socket statistics) to dump current established connections to port 80 including what process initiated it. If its a lot of data you can add | tee /tmp/output after done to both show the output on the screen aswell as write it to /tmp/output for later processing/digging. If it doesn't catch the quick haproxy connection, please try removing sleep 1 but be cautious of extensive logging if its a heavily utilized machine. Modify as needed!

share|improve this answer
Thanks you for the detailed response. I will take your word for the auditd solution as the host kernel does not support it and I do not have the time now to find one suitable for experimentation, but I will keep that in mind. As for the polling solution, I started doing something similar with lsof but stopped pretty quickly as it was not... satisfying. – pmezard Jan 22 '12 at 10:53
You can also use ausearch -i to have those saddr hex strings decoded automatically for your. – sch Sep 2 '14 at 11:37
ss is more satisfying than lsof because it's faster and it has good filtering rules - no need for grep. I can appreciate the problems with support: Systemtap is another tool that is superb but getting it to run on a production server can be ... not satisfying. – Max Murphy Nov 21 at 19:31

You can also grep the huge logs you get from "ausearch -i" to see only those sockets that successfully connected to another host on the internet. I wrote a simplistic script to get each process and command that created a socket to connect to a host on the internet along with the connection address of that target host and the current time that the socket was "created". Here it is:


if [[ $EUID -ne 0 ]]; then

    echo "You must run this script as root boy!"
    exit 1  


> proccessConnections.dat

connections=`ausearch -i | grep host: | awk -F "msg=audit" '{print $2}' | awk -F ": saddr" '{print $1}'`

connectionsNumber=`echo "$connections" | wc -l`

echo "Number of connections: $connectionsNumber"

echo "$connections" > conTemp.dat

let counter=1
while read connectInfo; do

    success=`ausearch -i | grep "$connectInfo" | grep "type=SYSCALL" | grep success=yes`    
    addressInfo=`ausearch -i | grep "$connectInfo" | grep type=SOCKADDR | awk -F ': ' '{print $2}'`
    processInfo=`ausearch -i | grep "$connectInfo" | grep "type=SYSCALL" | awk -F 'comm=' '{print $2}' | awk -F 'key' '{print $1}'` 

    if [[ $success != "" ]]
        echo "[$counter - $connectionsNumber] (success)     comm=$processInfo - $addressInfo - $connectInfo"
        echo "[$counter - $connectionsNumber] (success)     comm=$processInfo - $addressInfo - $connectInfo" >> proccessConnections.dat
        echo "[$counter - $connectionsNumber] (no success)  comm=$processInfo - $addressInfo - $connectInfo"
        echo "[$counter - $connectionsNumber] (no success)  comm=$processInfo - $addressInfo - $connectInfo" >> proccessConnections.dat

    let counter++

done < conTemp.dat
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.