Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have a process (dbus-daemon) which has many open connection over UNIX sockets. One of these connections is fd #36:

=$ ps uw -p 23284
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
depesz   23284  0.0  0.0  24680  1772 ?        Ss   15:25   0:00 /bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session

=$ ls -l /proc/23284/fd/36 
lrwx------ 1 depesz depesz 64 2011-03-28 15:32 /proc/23284/fd/36 -> socket:[1013410]

=$ netstat -nxp | grep 1013410
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
unix  3      [ ]         STREAM     CONNECTED     1013410  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD

=$ netstat -nxp | grep dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1013953  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1013825  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1013726  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1013471  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1013410  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1012325  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1012302  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1012289  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1012151  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011957  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011937  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011900  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011775  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011771  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011769  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011766  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011663  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011635  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011627  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011540  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011480  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011349  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011312  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011284  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011250  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011231  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011155  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011061  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011049  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011035  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1011013  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1010961  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD
unix  3      [ ]         STREAM     CONNECTED     1010945  23284/dbus-daemon   @/tmp/dbus-3XDU4PYEzD

Based on number connections, I assume that dbus-daemon is actually server. Which is OK. But how can I find which process is connected to it - using the connection that is 36th file handle in dbus-launcher? Tried lsof and even greps on /proc/net/unix but I can't figure out a way to find the client process.

share|improve this question
add comment

4 Answers

up vote 10 down vote accepted

Quite recently I stumbled upon a similar problem. I was shocked to find out that there are cases when this might not be possible. I dug up a comment from the creator of lsof (Vic Abell) where he pointed out that this depends heavily on unix socket implementation. Sometimes so called "endpoint" information for socket is available and sometimes not. Unfortunatelly it is impossible in Linux as he points out.

On Linux, for example, where lsof must use /proc/net/unix, all UNIX domain sockets have a bound path, but no endpoint information. Often there is no bound path. That often makes it impossible to determine the other endpoint, but it is a result of the Linux /proc file system implementation.

If you look at /proc/net/unix you can see for yourself, that (at least on my system) he is absolutelly right. I'm still shocked, because I find such feature essential while tracking server problems.

share|improve this answer
add comment

This answer is for Linux only. Based on an answer from the Unix & Linux Stack Exchange, I successfully identified the other end of a unix domain socket using in-kernel data structures, accessed using gdb and /proc/kcore. You need to enable the CONFIG_DEBUG_INFO and CONFIG_PROC_KCORE kernel options.

You can use lsof to get the kernel address of the socket, which takes the form of a pointer, e.g. 0xffff8803e256d9c0. That number is actually the address of the relevant in-kernel memory structure or type struct unix_sock. That structure has a field called peer which points at the other end of the socket. So the commands

# gdb /usr/src/linux/vmlinux /proc/kcore
(gdb) p ((struct unix_sock*)0xffff8803e256d9c0)->peer

will print the address of the other end of the connection. You can grep the output of lsof -U for that number to identify the process and file descriptor number of that other end.

Some distributions seem to provide kernel debug symbols as a separate package, which would take the place of the vmlinux file in the above command.

share|improve this answer
    
This looks interesting, but requirement to recompile kernel seems to be an overkill. I'm thinking that perhaps it would be possible to do it, without hand-made kernel, and without using gdb, just by peeking at values in kcore and doing some "manual" decoding of values. –  user13185 Aug 16 '12 at 9:24
1  
@depesz, all you need to know is the offset of the peer member in the unix_sock structure. On my x86_64 system, that offset is 656 bytes, so I could obtain that other end using p ((void**)0xffff8803e256d9c0)[0x52]. You still need CONFIG_PROC_KCORE, obviously. –  MvG Sep 2 '12 at 9:11
add comment

Unix sockets usually are assigned numbers in pairs, and are usually consecutive. So the pair for you would likely be 1013410+/-1. See which of those two exists and guess at the culprit.

share|improve this answer
add comment

Edit your system.conf

In this file you could add more stuff for debugging purpose.

File location: /etc/dbus-1/system.conf

For debugging purpose, you can edit your system.conf to allow eavesdropping:

  1. replace the policy section by:

    <policy context="default">

    <!-- Allow everything to be sent -->

    <allow send_destination="*" eavesdrop="true"/>

    <!-- Allow everything to be received -->

    <allow eavesdrop="true"/>

    <!-- Allow anyone to own anything -->

    <allow own="*"/>

    <!-- XXX: Allow all users to connect -->

    <allow user="*"/> </policy>

  2. Remove the includedir line: system.d

    <includedir>system.d</includedir>

Source: http://old.nabble.com/dbus-send-error-td29893862.html


Some other useful stuff regarding unix sockets

The simplest way to figure out what's happening on the bus is to run the dbus-monitor program, which comes with the D-Bus package

Also you can try to use dbus-cleanup-sockets to clean up leftover sockets.

Following command will show you which process is connected how many times to dbus sockets based on netstat output:

sudo netstat -nap | grep dbus | grep CONNECTED | awk '{print $8}' | sort | uniq -c

(tested on Ubuntu)

Hardcore way: This command will find manually the processes from /proc and show which are using the most connections (all type of sockets):

ls -lR */fd/* | grep socket | sed -r "s@([0-9{1}]+)/fd/@_\1_@g" | awk -F_ '{print $2}' | uniq -c | sort -n | awk '{print $1" "$2; print system("ps "$2"|tail -n1")}'

Example output:

(count, PID and the next line contains details about the process)

25 3732
 3732 ?        Ss     0:38 /usr/bin/wineserver
89 1970
 1970 ?        Ss     0:02 //bin/dbus-daemon --fork --print-pid 5 --print-address 7 --session

(tested on Ubuntu)

Have fun.


See also related articles for the reference:

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.