Toolkit & Habits for Linux Network & System Administration [closed]

Question

I am tasked with the administration of a small office network as well as several workstations running mostly Debian and Ubuntu. There are two servers: one database & print-server, and one backup & file server.

Being relatively new to this side of things, knowing enough to help myself to some degree on Linux, I would like to know what software tools and tasks/habits I can use/acquire to learn this field and be effective while doings so.

I don't need to know what is the best, just what a newbie sys-admin can use as a starter-pack to learn and use as a base to grow into proper system administration.

What I need is those few basic tools to start with, and the kind of things I need to do regularly, e.g.: which logs to check, when & what to monitor, the kind of 'right' place to start and to which I can ad as I need.

Answer 1

1

monitor business critical processes [ eg - is database running and responding to simple queries ], basic parameters of operating system [ free disk space, load average ]. you can use nagios or zabbix for instance.

2

gather statistics to establish some baselines. in the future this will be useful for capacity planning. you can use munin, zabbix, cacti etc.

3

run scheduled automatic backups, store some of them offline, offsite. monitor if they succeeded, from time to time check manually if you can recover critical data. you might want to use backupninja to orchestrate them or Zmanda, but there are much more useful tools..

4

document. for your own good. dont think that lack of documentation increases your job security. first one who will suffer because of lack of it is you, you'll probably forget things quite quickly.

5

from other random things:

learn some scripting language - maybe you know one aldready? perl/python/php can be used for automating tasks and in many cases are better suited than bash for more complicated tasks

learn your tools - it's endless list... ssh comes into mind probably first. check out this and that.

Answer 2

The Practice of System and Network Administration, by Limoncelli, et al, is where you should start. Technologies will come and go (and can be easily Googled as required), but the information in that book is timeless (and priceless).

Answer 3

A Few Basic Tools To Start With

Google. No seriously. Google is a system administrator's dream come true. There is no finer way to harness the vast amount of information on the internet. And don't limit yourself to just searching for "linux for beginners"; if you have a specific task you're trying to accomplish, Google for that specific thing. You'll often find way more info than you need, and as a beginner a lot of it will seem like another language, but learning through doing is a good way to pick things up quickly.

Tab-completion. The Linux command line allows for tab-completion of all commands, directory trees, directory names, and file names.

Man pages. Every command, and many system config files (/etc/fstab, /etc resolv.conf, etc.) have Man pages. Just type "man command_name" or "man file_name" to see if what you're looking for has one. Oh and "q" exits a man page.

SSH. One of the best ways to access a Linux system. Probably the best way if you don't have physical access to the system.

Screen. Screen is a fantastic little application that lets you turn one terminal into many, lets you get things out of your way/into the background, and lets you leaves things running so that you can come back to them later.

Nano. You mentioned above that you already use this, but I just thought I'd throw in my two cents and say that I agree. Vi and Vim and all those are fine and all, but it's the simplicity of Nano that I love. It's like the Notepad of the Linux world.

Find and Grep. Find is great for searching for files, Grep is great for searching in files. Both of them can be used in very simple ways, and both of them can be used in very complex ways, but both are quite useful either way.

Sudo. Lets you act like root, without being root. Very useful.

Plus a few other tools that I will mention in the context of the next section...

The Kinds Of Things You Need To Do Regularly

Monitor your system. Monitor your disk usage (df is a useful command, and du as well for specific directories), monitor your running processes and tasks (via the ps command and top commands), monitor the users logged into your systems (the users and who commands will tell you this), and monitor your network usage (apps like cacti are good for that). If you happen to have access to an X Windows environment, I always found GKrellM to be a very useful all-in-one system monitoring tool.

Backups. For the love of Tux, backups. Backup config files, backup home directories, backup application data. Backups. Even if all you're doing is straightup copying the data from the server to a CIFS/NFS share on another box and an external harddrive. And yes, you should keep two copies of each backup, and never on the same media/system. Think of it as backups of your backups.

Check your backups. Routinely check to make sure you can restore data from your backups onto your systems. Empty/corrupt/incomplete backups are as useless as the day is long.

Use your log files. Dmesg, /var/log/messages, and really pretty much anything in /var/log period. If something isn't working right and you don't know why, the logs may not have the answer but they can definitely help you find it. And the logs and directories in /var/log are sensibly named so finding the right log shouldn't be hard. You won't need to constantly monitor every log file, but keeping an eye on them will help you keep your system healthy and secure.

Keep your system updated. Don't let your software go for months and months without being updated, because it can result in a lot of headaches and breaking things when config file syntax or dependencies change. Different distros have different update programs (apt-get, yum, etc.) but whichever one you use, learn it and use it regularly.

Keep your system secure. Use things like iptables, PAM, hosts.allow/hosts.deny and similar to prevent unwanted access to and use of your system.

Never stop learning. To continue with something I said earlier (learning through doing), something you should look into is virtual machines. Download VirtualBox (or if you have VMWare licenses even better) and make yourself a Linux virtual machine. You can pick any distro you want really but obviously it probably makes the most sense to go with one you're using in your environment. Play around in the VM.. use it like a sandbox. Set stuff up, break things, investigate, learn. The beauty of a sandbox VM is that it doesn't matter what happens to it. If you totally hose it, just make a new one. Or keep a backup copy of the original after you get it setup, and reuse that whenever you need to.

As some of the other posts in this thread have mentioned and alluded to, these lists could really be nearly infinite, but hopefully this gets you off to a good start.

Answer 4

If you're just getting started, and especially since you have some Debian (Ubuntu is Debian at its core) systems, I highly recommend the Debian Reference. It's a great overview of nearly every aspect of system administration and should cover almost everything you need to know about maintaining a small set of systems like these.

I also agree with all the points pQd made, and more specifically I think it would be a good idea for you to set up a wiki to document all your processes and configurations. At my organization we use Trac, but any wiki engine should do, just make sure it has a nice way to display source code since that's useful for tiny scripts and command listings.

Answer 5

Honestly, Linux System Administration isn't a field you can just 'jump into'.

If you have to, though, there are a few good books on the subject. O'Reilly has two books (Linux Network Administration and Linux System Administration) which should get you started.

Personally, if I were you I'd just spend a few days messing around with different distributions, installing software, setting up Nagios/Cacti/Apache2/SSH/NFS type things, and perhaps learning some sort of scripting language (I use Perl, myself; but a lot of my fellow admins prefer to use Python. It's really up to you what you want to learn, though).

And definitely learn the command line. Don't fallback on graphical tools as a crutch.

Learn vi. Even if you only learn it well enough to do basic edits then fine -- but it's important to learn vi because sometimes you're stuck on a system without Vim/Nano/Emacs. When in that situation, you'll be happy you spent a day or two learning vi.

If you need any help feel free to e-mail me (my email is [removed for security reasons]) -- I'd be glad to help you outside of ServerFault.

Answer 6

If you're comfortable installing an application on the servers, consider webmin as it provides a "one stop" shop for most of the logging and configuration. Set it up to run over a high order port using SSL and it pays for itself in ease of checkup.

Answer 7

I'll tell you the greatest non-secret of system and network administration. You ready? Ok, here it is:

Learn the fundamentals. Let me elaborate.

Anyone (or just about) can learn what this or that particular software does, and how to push this button on that tool to make x, y, or z work. That's nothing special.

If you want to be a good sys/net admin, learn the under-the-hood stuff. What's the sequence of events in a typical network connection? What's the difference between a frame and a packet? What does load average really mean on a Unix system? What's the typical boot-up process for a machine (that one alone, if you follow it from beginning to end, will provide a wealth of knowledge).

Once you understand the fundamentals, and understand them really well, laying knowledge on top of a good foundation is much easier. But if you start at the top, and try to learn specific bits of software without knowing what goes on underneath, that will make you... just another high-tech janitor, basically.

Answer 8

First off, find your logs.  Most Linux distros log to /var/log/messages, although I've seen a couple log to /var/log/syslog.  If something is wrong, most likely there will be some relevant information in the logs.  Also, if you are dealing with email at all, don't forget /var/log/mail.  Double-check your applications, find out if any of them log somewhere ridiculous, outside of syslog.

Brush up on your vi skills.  Nano might be what all the cool kids are using these days, but experience has taught me that vi is the only text editor that is guaranteed to be on the system.  Once you get used to the keyboard shortcuts, and start creating your own triggers, vi will be like second nature to you.

Read the man page, and then run the following commands on each machine, and copy the results into your documentation:

hostname
cat /etc/*release*
cat /etc/hosts
cat /etc/resolv.conf
cat /etc/nsswitch
df -h
ifconfig -a
free -m
crontab -l
ls /etc/cron.d
echo $SHELL

That will serve as the beginnings of your documentation.  Those commands let you know your environment, and can help narrow down problems later on.

Grep through your logs and search for "error" or "failed".  That will give you an idea of what's not working as it should.  Your users will give you their opinion on whats wrong, listen closely to what they have to say.  They don't understand the system, but they see it in a different way than you do.

When you have a problem, check things in this order:

1. Disk Space (df -h):  Linux, and some apps that run on Linux, do some very strange things when disk space runs out.  It may seem unrelated, until you check and find a filesystem 100% full.

2. Top:  Top will let you know if you've got some process that's stuck out there eating up all of your available CPU cycles.  Nothing should consume 99% CPU for any extended period of time.  If its a legitimate process, it should probably fluctuate up and down.  While you are in top, check...

3. System Load:  The system load should normally be below 3 on a standard server or workstation.  The system load is based on CPU, memory, and I/O.

4. Memory (free -m): RAM use in Linux is a little different.  It's not uncommon to see a server with nearly all of its RAM used up.  Don't Panic, if you see this, it's mostly just cache, and will be cleared out as needed.  However, pay close attention to the amount of swap in use.  If possible, keep this as close to zero as you can.  Insufficient memory can lead to all kinds of performance problems.

5.  Logs: Go back to your logs, run tail -500 /var/log/messages | more and start reading through and seeing what's been going on.  Hopefully, the logs will be able to point you in the direction you need to go next.

A well maintained Linux server can run for years without problems.  We just shut one down that had been running for 748 days, and we only shut it down because we had migrated the application over to new hardware.  Hopefully, this will help you get your feet wet, and get you off to a good start.

One last thing, always make a copy of a config file you intend to change, and always copy the line you are changing, and comment out the original, adding your reason for changing it.  This will get you into the habit of documenting as you go, and may save your hide 9 months down the road.

Answer 9

good question.

My Advice. Learn to use your shell.

Standard is bash. You can just type help, to get in the documentaion.

learn pipes "|" to get the output from one command to the input of a second command.

One last thing, helped me much long time ago: Linux One Page Manual

work hard, never give up.

In 3-4 years you will have enough knowledge and many things come from itself :)