Explore inherited server

Question

We have inherited a series of servers with a number of systems on them, all running CentOS. A bunch of these are under-utilized and may not be doing much more than idling. None-the-less, there is the possibility that a script or database is being accessed occasionally. I would like to see exactly what these servers are doing--which scripts are being run and which databases are being accessed, etc. How would you go about exploring an inherited system to see exactly what is being run/accessed?

Answer 1

Your best bang for your buck will be to look at two things:

• What services are being provided?
• What jobs are running on a schedule?

What Services Are Bring Provided
You should be able to find most everything in two ways

• See what gets started by init with chkconfig --list. Look for anything labeled "on" and add to your list to be profiled.
• See what's actually listening on a port with `netstat -anut | grep 'LISTEN '. This will give you a list of all TCP/UDP ports in the listening state along with which binary opened them. Add those to your list to be profiled.

What Scheduled Jobs Are Running This will involve looking at all the various and sundry cron jobs for anything non-standard. This one might take a bit longer since there are a lot more locations to look through.

• Look at all the executable files in `/etc/cron.(daily|weekly|hourly|monthly).
• Look at all, if any, files in /etc/cron.d
• Inspect all of the user crontabs in /var/spool/cron

Depending on what these two find, you may need to start digging more. For instance, if the service httpd is running, you will need to look in /var/www and /etc/httpd to figure out exactly what is being served up.

This kind of discovery can take a while, and be a lot of work. In the end, it's a great exercise. You will come out with a well exercised set of skills, and a pretty good understanding of the environment.

Answer 2

Running ps -ef would show current processes.

In a few terminals you could run these for a while: top, iftop and iotop these, respectively speaking, offer CPU, network and disk usage amongst other things.

Checking scheduling around cron's files and binaries/events mentioned in /var/log/messages, /var/log/secure and /var/log/syslog should help too.