8

So our Linux admin left our project and suddenly I (the web coder with basic linux/server knowledge) am responsible for our dedicated server (Ubuntu Server) mainly running a web site (apache/mysql/php) and mail (Postfix). Our admin wasn't really a pro Linux admin, but rather some guy with basic Linux knowledge who figured out stuff as he went along. So I expect funky configurations, insecure services, etc.

My questions are:

  1. How do I perform an "audit" of the server to figure out it's current state, to make sure things are properly configured, that there are no unnecessary user accounts, that we aren't running unnecessary services, etc, etc.

  2. I'm unsure about how to back up our production web site. Besides the actual CMS files and db, there are apache configurations, mail databases and more that need back up. Any suggestion on how to automate this?

  3. What are the most important everyday duties of a linux admin that I absolutely have to perform? Huge question, I know.

1
  • What do you mean by a "funky" configuration? When my generation used the word it meant something was good. Mar 7, 2010 at 8:40

5 Answers 5

11

Wow. Where to start.

This is what I would do but hopefully others will jump in with more/better suggestions.

First don't panic. I am assuming you are now root. You are currently the most dangerous threat to the server right now as you have a lot of power and not much idea of what to do with it.

Write down what services the server should be running. You know apache, mysql and postfix are required. I guess you may have an ftp server on there and you are able to ssh in so you need sshd running. Write down what services are installed. The quickest way to find out is probably by listing /etc/init.d/*. You then need to find out what are running. I don't know what the equivalent to the Red Hat chkconfig is, but failing an alternative ps -ef will list what processes are currently running. Also find out if a firewall is installed (such as iptables) and how it is configured.

Next get a list of all the cronjobs that are run. You probably don't need to worry too much about what they do right away but you should have a general idea of what the server is doing at different times.

Again I would write all this down.

Now write down who should have access to the server and who is allowed to have root access. Get a list of users who do have accounts from /etc/passwd.

Do a similar thing for FTP access, and other services if relevant like Subversion or remote MySQL connections.

Now you know a bit more about what your server is doing and who can access it, you should move on to how well it is doing. Check the log files in /var/log, especially /var/log/messages and spend some time looking for any errors.

Check if there are any outstanding updates to do using apt-get update && apt-get upgrade

When prompted to update choose no for now.

So far you should have made no changes.

You now need to review the information you have collected and decide what (if anything) needs fixing. The priorities are attempted cracking attempts in /var/log/auth.log, shutting down uneccessary services and tightening the firewall.

Make copies of all files before you edit them and test changes often so you can easily back out if something breaks.

Backups

You will need to decide what needs to be backed up. Obvious candidates are databases, /home/ /etc/ /var/log/ /var/spool/cron/ /var/www/ and any custom scripts called by crontabs. Then most people write a shell script to back this up locally and then use something like rsync to copy the files to another machine USB drive.

Day to day duties will include, checking the log files for any problems (check out logwatch to help you), performing security updates, checking backups and goinf forward setting up monitoring like MRTG and Nagios to eventually take the heavy lifting out of being an admin.

I would not worry too much though. It may seem daunting but that is because you are asking for it all in one go. The server is probably fine as it is, keep an eye on the logs and apply updates as they are released, plan what you want to do and work towards it, take little steps and try to enjoy it.

5
  • 3
    Before doing anything else, I'd image and/or P2V that sucker.
    – Orangutech
    Mar 5, 2010 at 22:01
  • +1 for getting a current snapshot. Tarball the server over SSH or rsync at the very least. Mar 5, 2010 at 22:36
  • 1
    When grabbing what services are installed, also run "dpkg -l > ~/installed_software". This will write a list the packages that were installed via apt-get to a file named "installed_software" in your home directory. Check for any services that are running that were not installed via apt-get. The old admin may not have installed the servers via apt-get, and those services will have to be updated through means other than "apt-get upgrade". One would think this would be rare, but it is not.
    – Dru
    Mar 7, 2010 at 6:35
  • Another suitable suggestion would be to get the configuration files under some form of version control. Then as you make adjustments or change things to suit your way of thinking you can track those changes and see exactly where you screwed up. Or maybe you're lucky, but it's still good advice.
    – OldTroll
    Jun 3, 2011 at 21:17
  • All of this is great advice. In addition to the above I'd run netstat -tanp and/or -anp to show details on active and listening connections and their associated processes.
    – nedm
    Jun 3, 2011 at 21:23
5

Some things to get you started:

  1. get somebody who has already done such an audit - or at least has some knowledge about the common pitfalls maintaining a server. Seriously - it pays off.

  2. Back up as good as you can and try to reconstruct a spare server - might be a virtual instance somewhere - until you're confident that a) you have backed up all important stuff and b) you're able to reconstruct the spare server from your backup in as little time as you desire. To add karma: exchange the current production server with your spare server. As long as you've not demonstrated that you can reconstruct from your backup, act as if you have none.

  3. Update, read security notices, keep an eye on the logfiles and automate the heck out of this once you know what to look for.

3
  • 3
    +1 for finding someone who knows what they are doing. It will be cheaper in the long run. Mar 6, 2010 at 3:13
  • 1
    + 1 for the same but I'd go even further and suggest getting someone who is demonstrably more expert to make sure the whole system is in an appropriate state. A live system is no place to be learning pretty much from scratch. Mar 7, 2010 at 8:45
  • fully agreed, but as the system is already live, I guess there's no way around learning on a live system.
    – Olaf
    Mar 7, 2010 at 15:48
1

To more easily get a handle on the log files, you might consider installing (or activating..I dunno if ubuntu has this in the default install) LOGWATCH. It's very nice for giving you a summary sent to email every day. It usually picks up some funky stuff that a glance @ configs may not uncover.

1

If you value your server and it's data get help. Get someone to audit it.

If you don't know what "right" looks like then it can be hard to spot where something is "wrong" (or as you put it "funky"). Once someone the server is in a known good state.

Using something like VMWare Converter to make a staging VM of the server is a GREAT idea, you should look into that.

You then can poke around on the staging VM (copy of the server) and try to do each thing you are asked to do on the VM staging server first before the Production server.

0

After doing what Richard Holloway says; then do a network scan of the system to check what services are being provided by the server and check this against the data you have so far. It is possible to do really interesting things with linux that are hard to find just looking at logs.

I suggest using Zenmap from another system on the same network and getting whatever clearances you need from your bosses first. Zenmap is simple to install, is/has a GUI and doesn't try to exploit anything that is found.