find owner and spawn process of ssh connection

Question

There is this one server I manage which regularly (few times a minute) tries a connection to another machine on tcp port 22. (ssh)

My question: how can I find the process that spawned this connection?

What did I check already:

• tcpdump capture on ethernet interface + feed to wireshark => result was not so interesting except that it appears to be a standard encrypted SSH connection, the reassembly of the stream did not give me usefull info as it was (ofcourse) encrypted.

• lsof script: I dumped all file access known to lsof to a file about 60 times with 1 second between each run. I can see the connection setup in the logfile but around it, I recognize no useful info:

  ...
  ssh        8478     root    3u     IPv4             945966                 TCP servername:randomport->destinationserver:ssh (ESTABLISHED)
  ...
  

It looks like it's owned by the root user. Repeated the process a few times but nothing interesting recognized.

• netstat shows this: (netstat -ntap)

  tcp        0      0 servername:randomport  destinationserver:22   TIME_WAIT   -
  tcp        0      0 servername:randomport  destinationserver:22   TIME_WAIT   -
  

when I log enough iterations of this, I see it goes down to one time_wait cnx and back to 2 a few times per minute.

• crontabs:

no crontabs to be found for any user that starts some ssh cnx nothing about ssh in /etc/cron* files and directories

• started auditd

added this rule:

    auditctl -a exit,always -F arch=b64 -S connect -k MYCONNECT

Learned nothing except that it looks indeed like an ssh connection with same source and dest ip etc.

• destination server is not under my control but someone who manages it showed me a log that makes it look like an ssh cnx wich fails public-key authentication, there is no key placed whatsoever on the destination machine. This destination server is a storage linux-like (but not really) device.

I'm at a loss where this one comes from.

If I can't find the source of this useless connections, I can at least use iptables to block them but that's not so pretty ofcourse.

thanks for any hints.

Answer 1

I'd be tempted to use auditd to (briefly) gather information on task creation and see if I could work back from that

auditctl -D
auditctl -a task,always 

Then trawl through the results

ausearch -i -sc execve

Also lsof provides a PID so you could try

ps -p <PID> -o ppid=

Answer 2

One thing to try is to put a wrapper script around the standard ssh binary and have that wrapper script log id and time. This is just off the top of my head so it may need some work.

mv /usr/bin/ssh /usr/bin/ssh.real 

Replace

/usr/bin/ssh

with this script:

#!/usr/bin/perl
use Sys::Syslog;                        # all except setlogsock()
use Sys::Syslog qw(:standard :macros);  # standard functions & macros
openlog('ssh', "ndelay,pid", "auth");   # or whatever is appropriate other than auth
syslog('info',"started by $ENV{'USER'}"); 
exec '/usr/bin/ssh.real',  @ARGV;