Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have just inherited 6 web servers from previous server guy who was fired, I am not a sysadmin I am more a DevOps.

Could anyone point me to some sort of standard checklist one would follow when inheriting existing servers? Things I need to know are:

  1. What software is on the servers
  2. What are the standard things I should do to check they are secure?
  3. what is connecting to them and what are they connected too?
  4. What else should I know?

Any advise is welcome, I was hoping there was a standard kind of checklist that one would follow as a start, but I could not find anything.

All servers are Ubuntu (various versions)

share|improve this question
3  
You are not a DevOps. DevOps is not a title, it's a culture. –  gWaldo Mar 13 at 0:11
    
What would be a better term to use for a "Developer forced into having to do a sysadmins job without the expertise of a sysadmin but a good understanding of all concepts involved and enough hands on experience to 'get by'" –  user3408844 Mar 13 at 23:16
    
Agreed having read DevOps on Wikipedia I see I used it incorrectly, but it seems to have become slang in the daily life of developers having to become semi syadmins –  user3408844 Mar 13 at 23:22
    
The term for what you describe is "Developer" or "Developer forced to take on some Systems / Operations duties". –  gWaldo Mar 14 at 12:34
    
Wow! - lot of effort put into that one –  user3408844 Mar 16 at 6:31

3 Answers 3

up vote 24 down vote accepted
  1. To determine what software has been installed, you can review /var/log/dpkg.log However, this may not be a complete record. There may be binaries and code that was compiled manually or copied directly to the system pre-compiled. You could compare a default install of the same Ubuntu version and type to the server(s) and look for what files are different, but that can be quiet tedious. A file monitor solution would be ideal (tripewire, inotifywatch, etc.) http://linuxcommando.blogspot.com/2008/08/how-to-show-apt-log-history.html

  2. You need to check EVERYTHING on the server. Every user account in /etc/passwd, every application user account (such as users in Apache/PHP, database accounts, etc.) should be accounted for, and you should change all the passwords. You should check to see what services are launched on boot, what the default runlevel is and what starts with it and with other runlevels. I would use a vulnerability scanner and a baseline configuration tool to audit the current state. The Center for Internet Security offers a free configuration assessment tool, but it may be limited. They have more advanced tools for member organizations ($). http://benchmarks.cisecurity.org/ OpenVAS is a FOSS scanner, not unlike Nessus, which may have similar capabilities. There are many, many more things to check, but this answer is already getting a bit long... (Code review for webapps and web pages is a good example.)

  3. You can see review the state of ports available for connections to the servers with a variety of flags for netstat. http://www.thegeekstuff.com/2010/03/netstat-command-examples/ To identify who has been connecting to the server you will have to resort to the sexiest of Internet Security activities, reviewing system logs. The info can be in any one of a number of logs depending on what applications and servers are on the system. You may also have some luck with external network logs, if they exist.

  4. You have a lot of follow up to do. You indicated that the previous admin was fired; if you suspect malicious intent from that person (i.e. they may have left backdoors, boobie traps, logic bombs, etc.) your almost certain to be better off rebuilding the servers from clean media and reimplement the webapps on them. If this previous admin had full access and control to those system and was not subjected to diligent auditing and overwatch, you should probably assume there are backdoors.

This is based on a pessimistic assumption about the previous admin. Unfortunately that is the way the cookie crumbles for operational network security. There is a lot more to consider, as I said...way more than can be covered here. These points should give you some things to start doing so you can report to management that you are making some progress; but to be brutally honest, if you are not a security professional and you have reason to suspect this person did act with malice, you are probably in over your head.

It is an unpopular answer with management because it requires a lot of effort (which means more $), but the general security minded answer is when in doubt, wipe and rebuild from clean sources. That is how most important gov't systems work with malware; if an alert comes up from AV, the system is segregated, wiped, and rebuilt. Hope you made a backup cuz that data is GONE.

Good luck, and I hope this was helpful and not just depressing.

share|improve this answer
    
This question may also be well suited to the StackExchange site www.AskUbuntu.com –  0xSheepdog Mar 12 at 5:07
2  
Excellent answer. –  EEAA Mar 12 at 5:13
3  
Wipe and rebuild is probably made a lot easier if you have good declarative configuration management. If you don't, you probably should consider working toward that goal. –  zigg Mar 12 at 14:12
2  
/var/log/dpkg.log is well suited to review the installation process itself (and look for errors), but to get a list of installed packages, the output of dpkg -l or even simpler dpkg --get-selections would be easier to digest. –  Dubu Mar 12 at 15:36

The man pages are your friend:

 man <command> 

Check out these commonly used commands and their usage. Find more help in the man pages for each or in some cases by running

  <command> --help 

Software:

  • dpkg -l (list installed software)
  • ps -ef |more (get a list of running processes and pause for reading)

Security:

  • iptables (what ports are open and are they necessary)
  • check updates with apt-get (are the servers up to date?)
  • cat /etc/passwd (who has an account on the box)
  • sshd (check if sshd is running and who can log in on that)

Connections:

  • netstat (what services are listening and on what ports)

Good luck. It's tough inheriting a batch of servers without the person running them having the opportunity to train you. If the guy was fired it's even more worrysome because I assume there was a reason and if I also assume that it was job related there might be some strange setups in the batch.

share|improve this answer
  1. which applications are running: do a "ps -ef " or "ps -auxw" to get the process list. weed out everything that's not kernel related, look for stuff that's running, do man pages on each one to figure out what it is. most of the running processes you can safely ignore because they aren't user-applications

  2. for security: do a "netstat -pan" to see which ports are open, and close any that are not necessary. In other words, the only ports that should be open are the ones that correspond to the network services provided by these servers. If the server is a web server, then obviously it needs to be listening on port 80/443/etc. But if the server is listening on port 21 and nobody is using it, then you should turn off the process that has that port open.

  3. for connections, again "netstat -pan" gives you the answer. It tells you which hosts are connected and which ports they are connected on.

  4. look through logs in /var/log to get an idea of what they system is doing and to see if there are any obvious errors or red flags coming from different applications.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.