What programmer things should every sysadmin know?

Question

Inspired by this question, I ask the reverse: how much about programming do system administrators need to know? More specifically, what programming tools are useful for a sysadmin to have?

Answer 1

• Version Control. Be able to generate, read and apply patches. Know how to use a version control system that presents repository wide versions and why you want one. Know how to write descriptive changelogs and why you want them. Know how to search a repository's logs for keywords and time frames.
• Scripting. Do something once and be on your way. Do it twice or more, do it once then write a script.
• Debugging. Know how to read a stack trace and how to report relevant errors to your software support contact. Finding the error is nice and helpful, but knowing how to fix it can take a lot of investment in reading the code. Do the part that's easy for you, and let them do the part that's easy for them.
• Testing. Monitor continuously and log errors. Used in conjunction with version control and testing, you have a strong idea of what may have gone wrong when and what changed around then. Monitor both production and preproduction.
• Peer Review. Propose and review changes to production systems. Test on preproduction, determine exactly what needs to be done and record what services may be affected for how long. Don't let Change Management degrade into political battles of bureaucratic power.
• Study Cryptography. A modern system administrator is in charge of network resources; adding security as a final step is somewhere between impossible and a very expensive proposition. Understanding public key cryptography, password handling practices, and encryption in general will be extremely valuable.

Answer 2

I'd start with:

1. How to write more than the most basic scripts no matter the language. Automation is your friend.
2. How to run a debugger/know how to read crash reports of some kind: it makes figuring out a crash report and who to ask what the deuce to much easier.
3. Version Control - not just for code but for anything worth keeping a history or change control over.

Answer 3

System administration is just programming. A configuration management system lets you see your entire infrastructure as a distributed machine. As a sysadmin, your job is to program this machine.

Know one text editor. Usually vi for sysadmins, emacs for programmers.

Know how to script. Pick a scripting language. Perl is a perennial favourite, as is the shell/sed/awk combo. Ruby and Python can work, but for a lot of things objects are the wrong paradigm.

Know how to write readable documentation.

Version control. Crucial for maintaining configuration files and audit trails.

Learn to think in programming terms. Understand procedural, object oriented and functional programming paradigms. These may never be used, but knowing them makes you infinitely more productive.

Learn to use a debugger (or more) and logging.

Learn to use a RDBMS. There are plenty of data processing requirements which can be simplified by using a RDBMS correctly. Even being able to switch to a DBA role on demand can do wonders.

Test before deployment. See the test-driven development philosophy.

Pair programming is good. Have someone else watch over your shoulder to learn/advise/correct.

Learn to at least have a passing familiarity with a bunch of languages. Learn a new language every year.

Answer 4

• Know how to program. Yes, that's important: as one commenter said, if you do it more than once, then write a script - and write it well.
• Know the tools included in a standard operating system installation. For UNIX administrators, this means tools like Korn Shell (ksh) and Perl and vi. Don't rely on Emacs or Ruby or Tcl or C shell being present.
• Know the network. How are Ethernet packets put together? What does a packet look like? What are the differing packet types? Get to know tools like tcpdump, wireshark, snoop, and others.
• Write portable code. If the code is going to run under Linux - and Tru64 - and Solaris - and OpenVMS - then write portable code. Even if it will run under two versions of UNIX only - make it portable. But then, if it doesn't need to be portable, don't spend extra time on it.
• Know where the documentation is. For Perl, this means perldoc, Perl Mongers, perl.org, etc. For ksh, this means the man page and your favorite Korn shell books.
• Document, document, document, document! Document your code, create man pages, create inline Perl documentation, and whatever else is appropriate. Explain how to use the program and why you coded it that way.
• Know your system's packaging tools. Know how to create RPMs for Red Hat, or HP-UX depots, or Solaris packages: this will allow you to build packages for your systems, and thus to integrate them into the installation process.

Answer 5

I am a programmer and senior admin/integrator. To work where I do, you need to know the following things:

• How to write shell scripts (POSIX, no Bashisms), a little Perl, a little Python
• Basic C (Single UNIX specification)
• How to install a compiler tool chain
• How to build libraries and other OSS projects from source
• What Valgrind is and how to use it to spot memory leaks
• How to roll .rpm and .deb packages
• While we don't require it, a little PHP and a little LISP won't hurt

Most of the junior admins can look at one of our programmers and say (with authority), yes, I installed it correctly, here's where you have a leak which is why the code you just pushed is breaking. Or "No, its not our version of MySQL, look at your query here ..."

I think we're kind of different, when we hire a gifted administrator .. we half way expect for them to (eventually) find their way to programming, even if its just hacking something we use to work the way they need it to work.

In summary, being well rounded is never going to hurt you.

Answer 6

Scripting is essential, but I'd second the advice that knowing some "real" languages is a definite bonus. You can do some very useful things with the .NET System.DirectoryServices namespace, for example.

Answer 7

Many, many years ago (Windows NT 3.1 in fact) I started out as a programmer specialising in writing services and even the occasional device driver. This means you get to know the Windows kernel very well. In time I got a bit bored with the programming and switched to network management. I've found my background in programming has been immensley valuable over and over again.

It's not just that writing VBScript scripts is relatively painless. Knowing how the guts of Windows works, and particularly how IP works, helps greatly with detangling odd server and network problems. Also it's a mindset. Programmers are used to documentation and version control, and it horrifies me me to see how many Windows admins will just try something to see if it works and worry about untangling the mess afterwards.

All very well, but it isn't very useful advice to say go away and work as a programmer for ten years! However I think it would help most sysadmins to have some experience of programming in a "hard" language like C++ or C#. And if you have tame programmers in your organisation go drinking with them!

John Rennie

Answer 8

Bad things happen, your code will be around longer than you think it will. Fail gracefully, with useful error messages. You will run out of disk, ram, cpu, time, etc, deal with it.

Don't be so optimistic, that weird once in a million issue will happen far too often in production.

Answer 9

• SQL - Everything works with databases
• Programming - Language doesn't matter, just as long as you understand the logic and process you can learn and adapt to any language.
• How to DOCUMENT CODE and CHANGES - this one is huge. How many times have you been stuck looking at a configuration trying to figure out why the heck did somebody do this and what is it going to affect if we change it back. This one goes beyond anything you code or script, this one covers when you make changes in the firewall settings to handle a fringe case when user X recieves email from domain Y on a tuesday afternoon when its raining out. And it may not be you, it might be the PFY who decides to go ahead and do itand then has to call you up at 2am on your vacation because everything broke in a hideous manner.
• System languages - On Windows this is things like Powershell and WMI that let your scripting languages tie into some pretty powerful resources and do anything on a PC. Unix/Linux has their own set of tools that I'm not qualified to speak on.

Answer 10

Knowing how to bind to your LDAP and retrieve needed info in your scripting language of choice is key for everything from troubleshooting to really stepping up various types of automation (account and resource provisioning, more intelligent checks on user or machine accounts).

May seem basic but I have seen plenty of bread-n-butter shops where folks don't think they have a need for this kind of system administration. Plenty of places spend big $$$ on big sets of administrative tools and, as far as I can tell, 95% of what they use the tools for could have been done with a few hours of scripting if anybody knew that or knew how to do it.

Answer 11

I've found that knowing bash, dos batch, and powershell scripting provide a system admin with the framework to do just about anything, on any popular system.