Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I need to monitor a single process (e.g. be warned when there are more than 3000 connections established) and collect statistics on it (e.g. to determine how many connections were established today 01:20 AM, when the server worked too slow, as client said). What tools should I use?

share|improve this question
    
What process is this? Something that can handle 3000 connection would come with some statistics functionality, I recon. –  wzzrd Jul 24 '09 at 7:59
    
It is a daemon that I wrote by myself. I don't want to add statistics functionality to it. –  exquisitor Jul 24 '09 at 8:10
add comment

5 Answers 5

up vote 0 down vote accepted

If you don't want to go for a full Nagios (or whatever) install to monitor a single process, why not just write a script to do it yourself? I've done something similar to keep track of DB connections from one of our boxes, using the output of netstat to do the count and logging the results to a file. Adding an extra few lines to send an email if the count is >3000 should be trivial.

share|improve this answer
add comment

This maybe is not the most sophisticated solution, but - especially if you do not have other processes opening so many sockets - you could check the output of

netstat -nutp

(n: no name resolution, t: TCP, u: UDP, p:show PID and program - you might want to only provide only one of u or t based on whether your process opens UDP or TCP connections).

You can grep for the pid from the output:

netstat -nutp | grep -c ' 12345/progname$'

where '12345' should be replaced with your PID and 'progname' with the name of your process. The option -c for grep counts the matches. You might want to refine the search to more accurately match your needs (e.g. include only ESTABLISHED connections).

Also 'lsof' might be your friend. You can try

lsof -p 12345 -a -i4

and check the output and do some grepping. Have a look into lsof manual page to see if you can modify the output format to better suit scripted parsing.

You can write a simple script to run the command periodically. For huge number of connections, you better experiment how much resources running netstat or lsof takes and adjust the interval. E.g. once per minute (by default):

#!/bin/sh

prog=progname

if [ -z "$1" ]; then
     interval=60
else
     interval="$1"
fi

pid=$(pidof $prog)
while :; do
    n=$(netstat -nutp | grep -c " ${pid}/${prog}$')
    date +"Number of connections [%Y-%m-%d %H:%M:%S]: $n" > connection.log
    if [ "$n" -gt $TRESHOLD ]; then
       # warn the admin
    fi
    sleep "${interval}"
done

(quite useless, just provide to give idea).

share|improve this answer
add comment

If you want alerts and monitoring then I would look at Nagios if you want pure graphs then I would look at Munin or Cacti. If you just want to know how many connections a process has open at any time then use lsof.

share|improve this answer
add comment

You can use a ready made solution, ps-watcher

your config can be like this:

[processname$]
    trigger = $count > 3000
    action  = <<EOF
    mail -s "processname treshold exceeded" <<< "You have $count processes" 
    /root/bin/run_some_cleanup
EOF

[[p]rocessname$]
    action = echo "$count processes are running"

This will mail you when process count exceeds treshold. The second part has a different regular expression that matches the same process name, it logs the count of processes. As it is not limited by any trigger, the action runs at every ps-watcher check. You can change checking interval with "--sleep 150" option to ps-watcher.

share|improve this answer
add comment

I would intall munin and write a plugin to monitor a specific process or a specific behavior of a service.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.