Two weeks ago, I was notified by my VPS provider that my server (CentOS 5.5, yum is up to date) had originated "NULL byte/Directory Traversal" attacks agains some servers at DreamHost. I spent a few hours going over the server with a fine toothed comb and didn't find anything. Before logging in, I retrieved a copy of the sshd binary and confirmed that it hadn't been modified. I installed a rootkit checker (chkrootkit-0.49) that found nothing. I checked the web logs of the websites I host, looking for a hit that may have triggered a script on my server to initiate the attacks but found nothing. Checked /var/log/secure and /var/log/messages around the times of the attack but found nothing. Checked
last, but found nothing. Did a
find on key directories looking for files modified in the past 3 days, but nothing.
What else can I do to to find the cause of the attacks? I wrote a script to check for outbound TCP connections on port 80, but only came up with legitimate connections (SpamAssassin and ClamAV downloading updates, Joomla checking its site for updates, etc.). Even if I did see an active outbound connection, would I even be able to dump data from the process (in the
/proc directory) to show me the account originating the attack?
After watching the server for a few days, I gave up. Now I've received another complaint from DreamHost, so it's happened again. I've requested detailed logs from DreamHost, but then what? Where else can I look? If I can't find the source, is there something I can install to monitor the server and log data when it starts making outbound connections to tcp/80 in the DreamHost IP space? What would I log? Just get a
tcpdump of all traffic in that timeframe and try to sift through it manually?
See my accepted answer for the solution I came up with.
I'm still interested in options for logging the source of all outbound port 80 traffic -- a way to know what the source process is and perhaps its parent process (and the parent's parent).