43

I've been handed 3 Linux boxes, 1 front facing with apache on it and another 2 which, as far as I can tell, don't do an awful lot. All running on Redhat.

The question is simple: How can I tell what the server is actually doing? Zero documentation is available from the creator.

Improve this question
18
  • 7
    The processlist, network listeners (maybe compare to would should have been running based on init scripts) ...
    – HBruijn
    Jun 9, 2015 at 13:27
  • 5
    damn! surely someone knows what they are for? so they want you to support them but dont know what they do?!
    – Digital Lightcraft
    Jun 9, 2015 at 14:25
  • 11
    Turn them off. Someone will let you know, almost immediately, what's not working.
    – jscott
    Jun 9, 2015 at 17:15
  • 58
    DO NOT TURN IT OFF. Unplug the Ethernet cable, if you want to scream-test. If you've never had a box with a two-year uptime fail to reboot, you will at some point. This isn't the time to add that frustration to the mix.
    – Aaron Copley
    Jun 9, 2015 at 18:16
  • 7
    What everyone else said, but also: run nmap against them.
    – Katherine Villyard
    Jun 9, 2015 at 21:11

6 Answers 6

Reset to default
43

Unplug the ethernet cable and see who gets upset.

Seriously though, mystery machines like this create a lot of mental overhead for a team and often provide absolutely no business value. Talk to your boss, if no one knows what it does maybe no one cares what it does.

Improve this answer
6
  • 44
    DO NOT TURN IT OFF. Unplug the Ethernet cable, if you want to scream-test. If you've never had a box with a two-year uptime fail to reboot, you will at some point. This isn't the time to add that frustration to the mix. (Copying this here because it needs to be read.)
    – Aaron Copley
    Jun 9, 2015 at 18:17
  • 3
    You're 100% right, I've edited the post to reflect that. I was being a little facetious but I can still be facetious without promoting disaster.
    – Josh Rumbut
    Jun 9, 2015 at 18:44
  • 8
    Before the network is unplugged it is a good idea to save a list of running processes and open sockets on each server - just in case something is relying on a TCP connection between the servers which happens to have remained ESTABLISHED for many months without anybody thinking about automating whatever steps were needed to open it in the first case. (For example if somebody needed an ssh port forwarding temporarily and then forgot about it.)
    – kasperd
    Jun 9, 2015 at 21:52
  • 4
    Don't do this. What ridiculous advice. Stupid thoughtless IT people doing this cost me so much time and work. ASK FIRST. If you don't know who to ask, ASK EVERYONE.
    – Lightness Races in Orbit
    Jun 10, 2015 at 20:16
  • 4
    Make sure you leave sufficient time for someone to scream - the one time I did this with a dusty or desktop machine sitting on a server room rack - no one knew what it did, it took a full month for someone to notice when we unplugged it. Turns out it was part of the payroll system, without that server, Accounting couldn't generate monthly payroll. It had been running unattended for at least 3 years with no one paying any attention to it -- so the "scream test" was a success, if we hadn't done it, the server would have eventually died on its own. We ended up p2v'ing it into our vmware cluster.
    – Johnny
    Jun 10, 2015 at 23:32
31

This is a pretty broad question for the Serverfault format, but here is a good start:

  • Check for running processes and those scheduled to run at system startup.
    • Review the running configuration of each.
    • Look into any defined data directories. (Maybe someone installed MySQL and turned it on, but there are no databases.)
  • Check for scheduled tasks.
  • Check the logs to see;
    • who has logged in recently (and ask them)
    • and to get an idea of what's been running.

You didn't mention the version, so I've omitted the specifics.

Improve this answer
4
  • 8
    There is something more important than which services are configured to start when the system boots. Which services are running right now? Starting a service and forgetting to configure it to start at boot is not a difficult mistake to make. On a related note, it is a good idea to look at other system state such as: mount points, routing table, iptables rules. All of those are things which could easily be changed while the system is running without remembering to update the config files used during boot.
    – kasperd
    Jun 9, 2015 at 21:58
  • In addition I would use a port scanner to see which ports are open, then try to connect to it using the usual tools. For (a simple) example if port 443 is open you could try using a web browser to connect to it. I have had to explore such seemingly abandoned servers frequently and one of my favourite tools to quickly browse through the configuration files in /etc and other places is to use "lynx" or "links" if you prefer. These are character based web browsers which do a good enough job as a file browser as well, with convenient cursor key navigation.
    – aseq
    Jun 9, 2015 at 22:43
  • 1
    @kasperd Fair play on running vs. persistent services. But, I thought about firewall rules, mount points, etc. Those seemed to me as ancillary components that will already be tied to one of the existing bullet points. YMMV.
    – Aaron Copley
    Jun 9, 2015 at 23:18
  • Please add - Check for active network connections, and write down the names of the services and port numbers. The best method to do this varies by operating system. E.G. net stat. Also, put some kind of trace on the computer so you can see what it does throughout the day. You may also want to add consideration for scenarios when things running on the servers COULD be malicious.
    – IceMage
    Jun 12, 2015 at 20:28
19

There are a few things you could do to try and ascertain what's running on your system.

You can check which ports your server is listening on to get an idea of what's on there. A good command to use would be:

 [root@server ~]# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address               Foreign Address             Stat    e       PID/Program name
tcp        0      0 0.0.0.0:139                 0.0.0.0:*                   LIST    EN      1880/smbd
tcp        0      0 0.0.0.0:5666                0.0.0.0:*                   LIST    EN      1911/nrpe
tcp        0      0 0.0.0.0:22                  0.0.0.0:*                   LIST    EN      1759/sshd

As you can see from the example output above, it presents you with the protocol version (tcp or udp), the address that's being listened on, the port that's open and the program that's listening.

In the above truncated example (a server machine) you can see tcp ports 139, 5666, and 22 are listening. These resolve to samba, nrpe (Nagios agent) and ssh respectively, and is confirmed when you check the program that's listening on that port.

Additionally, you can check the list of daemons which are configured to start at boot, in order to do that, run: chkconfig --list | grep "3:on"

Example:

[root@server ~]# chkconfig --list | grep "3:on"
NetworkManager  0:off   1:off   2:on    3:on    4:on    5:on    6:off
acpid           0:off   1:off   2:on    3:on    4:on    5:on    6:off
sshd            0:off   1:off   2:on    3:on    4:on    5:on    6:off
sysstat         0:off   1:on    2:on    3:on    4:on    5:on    6:off
udev-post       0:off   1:on    2:on    3:on    4:on    5:on    6:off
vncserver       0:off   1:off   2:on    3:on    4:on    5:on    6:off
webmin          0:off   1:off   2:on    3:on    4:off   5:on    6:off
x2gocleansessions       0:off   1:off   2:on    3:on    4:on    5:on    6:off
.
.
.

or :

service --status-all

Improve this answer
2
  • 5
    I find netstat -plunt easier to remember.
    – abligh
    Jun 9, 2015 at 19:32
  • 2
    Also, tcpdump may be useful to determine who is actually using each service.
    – abligh
    Jun 9, 2015 at 19:34
18

Another method involves checking the /etc directory and looking at the modification dates. After a fresh install all the files in this directory should have roughly the same date/time. And since an install usually installs a lot of things people usually do not use, only the files that have a later modification date reflect the actual purpose of the server. If this is ext4 you also should be able to extract the birth date of directories, so the task could be quite easy.

Yet another method would involve checking the .bash_history files to see what the admins were up to. This file can provide a wealth of knowledge.

Improve this answer
7

Check the firewall rules. With a bit of luck, it's configured for default-deny. That means there's an explicit rule for each allowed service.

This is better then netstat because it can also show ports that are open for e.g for nightly backups.

Improve this answer
6

One answer I've not seen yet: Check the most recently modified files. Logs, database files, other output files etc. may get written to still that may provide clues:

find . -mtime -3

That would find modified files in the current directory and deeper, changed in the last 3 days. Increase the number 3 to an educated guess until you get some output you can investigate.

Not fool-proof, as the boxes may just process some web service calls, returning some data without ever writing anything. But added to the great mix mentioned above, it may just yield some clues.

Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.