Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm wondering about tools that are powerful and that most system administrators either don't know about or don't use (but should).

For one thing, I like the possibility of finding out about a tool that is good and that I should be using - or at least, trying out. I also find that giving these tools their time in the sun (again) can be a positive, letting others know about the wonderful tools that are out there.

Thus, things like sudo, vi, emacs, dtrace, ps, and top are out. I have some ideas but I just hate to skew the statistics...

I'll just wait and see if anyone mentions my favorites.

share

locked by EEAA Dec 6 '15 at 21:35

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

closed as not a real question by Chris S Feb 26 '12 at 20:07

It's difficult to tell what is being asked here. This question is ambiguous, vague, incomplete, overly broad, or rhetorical and cannot be reasonably answered in its current form. For help clarifying this question so that it can be reopened, visit the help center.If this question can be reworded to fit the rules in the help center, please edit the question.

7  
There are so many 'what tools' questions, do we really need another? – Zoredache Jun 6 '09 at 9:49
4  
I just have to ask: if no one's heard of a tool, then how will it get upvotes? – quux Jun 7 '09 at 2:52
5  
+1 for camera phone. They are also incredibly useful when working with someone over the phone to troubleshoot a problem and there is a long error message on the screen. I wish all remote hands people had/made use of them. – Mark Jun 17 '09 at 18:40
1  
I've used it to document lights on the front panel and ports on the back panel (in this case, of a DEC Alpha system). – Mei Jun 17 '09 at 23:18
2  
Please mention one tool per answer. – Cristian Ciupitu Aug 4 '09 at 14:18

74 Answers 74

iPerf

Iperf helps you run tests that measure maximum TCP and UDP bandwidth performance. It allows the tuning of various parameters and UDP characteristics reporting bandwidth, delay jitter, datagram loss: http://openmaniak.com/iperf.php


,And

MTR

MTR (My Trace Route) is also a pretty good tool. It combines the functionality of the 'traceroute' and 'ping' programs in a single network diagnostic tool. Like traceroute except it gives more network quality and network diagnostic info. Leave running to get real time stats. Reports best and worst round trip times in milliseconds... It shows latency, jitter (average/best/worst): http://www.bitwizard.nl/mtr/

use:

mtr server.domain.com (or IP)

  • Add more fields with "O", type "LDRS NBAW V JMXI" and hit ENTER
  • Type "n" to toggle DNS Off/On

alt text

share
2  
+1 for mtr. Beats traceroute every time. – David Pashley Jun 6 '09 at 9:15
    
Except for when they've blocked ICMP and allowed UDP. Then it's back to traceroute -PUDP. – sh-beta Oct 30 '09 at 18:58
1  
@sh-beta, or mtr -u ;) – l0c0b0x Oct 30 '09 at 21:10
    
o.O I really, really hope that's a recent development. The hours I've wasted! Alas! – sh-beta Nov 2 '09 at 19:40

Paperclips: I've had to use paperclips to pick the locks on the front of a sun and dell server We've all used them to open a cd rom

HotPlug This thing is awesome. Want to move a server to the next rack over without turning it off? http://www.wiebetech.com/products/HotPlug.php

Blocks of wood The idiot before you not mount the server properly? The server too old to have rack mounts? Blocks of wood are your friend. It's ghetto, but works.

share
1  
wow, blocks of wood...you're right, that's pretty ghetto. You're still getting +1 for the hotplug, though – Matt Simmons Jun 17 '09 at 19:11
3  
I've used blocks of wood to keep a UPS from crushing me while I mounted it in the rack. – steve.lippert Jun 17 '09 at 20:03
    
I'm with Matt - hotplug is today's best link. – Kara Marfia Jun 17 '09 at 20:22
    
Hotplug link - wow. – Dan Carley Jun 17 '09 at 21:56
    
Another vote for HotPlug. That looks awesome. – Nic Nov 7 '09 at 20:46

A bent paper clip to pop open a CD drive.

share
5  
I started hanging them on the doors of all of our server racks - they're also perfect for getting at reset buttons that are recessed. People started asking what on earth paperclips were doing hanging from the doors until I explained. Now they agree it's a pretty good idea. – Mark Jun 17 '09 at 18:47
1  
I keep one in my wallet at all times. Something no one should ever be without! – Matt Simmons Jun 17 '09 at 19:09
1  
And today I learned how much more useful they are! Time to see if it'll pick a poweredge face panel lock... – Kara Marfia Jun 17 '09 at 19:36
    
I used a few paper clips and some electrical tape to make a nice one with a handle that big handed people like me can use. – steve.lippert Jun 17 '09 at 20:01
    
I forgot! You can open plastic locking zip ties with them: standalone-sysadmin.blogspot.com/2008/10/… – Matt Simmons Jun 17 '09 at 20:31

Sysstat / Sar - superb historic monitoring for servers.

Network/CPU/Memory/IO/etc/etc brilliant if you need to examine why a machine went down.

For example load averages between 6am and 7am this morning:

hcooper@localhost:~$ sar -q -s 06:00:00 -e 07:30:00
Linux 2.4.27-3-686 (localhost)  06/06/09

06:05:01      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
06:15:01            2       200      0.00      0.05      0.04
06:25:01            4       199      0.01      0.05      0.04
06:35:02            0       208      1.74      1.39      0.79
06:45:01            1       201      0.12      0.52      0.70
06:55:02            1       197      0.04      0.14      0.40
07:05:01            2       203      0.13      0.10      0.24
07:15:01            2       200      0.00      0.05      0.15
07:25:01            2       200      0.01      0.04      0.08
Average:            2       201      0.26      0.29      0.30
share
    
I'd be surprised if an admin hasn't heard of this one: I'm continually surprised at Linux distributions that don't contain these venerable tools, and at other UNIXes that don't have these enabled by default. – Mei Jun 8 '09 at 15:26

A Paper Bag: To breathe into when you discover that the SQL script you wrote wasn't actually mashing up the test database after all. Oh, and your backup system crashes and now you have to wait on the phone with tech support because you don't have a support contract.

Origami Paper: To pass the time when you're on hold with tier 1 tech support at said backup software vendor.

A Stress Ball: To help when discussing the problem with Tier 1 support after they finally answer the phone.

An Ice Pack: To soothe your aching fist after pounding it into the desk when you realize it took 90 minutes just to get handed off to Tier 2 support.

A Spare Cell Phone: After you throw yours into the ground when you get disconnected just as Tier 3 support answers your call.

An Account at Monster.com and Experience in Laying Carpet: =(

--

On a slightly more serious note, having a small refridgerator can be handy especially for putting crashed hard drives in to try and recover data.

Also, it's even better if your camera phone can take video. I take video when I'm messing with BIOSs and other things that I don't have time to simultaneously document while editing.

Cough drops and nose spray for extended stays in the dry server room.

A P-Touch Labeler to label EVERYTHING even if you're certain you'll remember which patch cables connect to the security cameras.

Fingernail clippers to remove the sheath from the 4 pairs of wires in a Cat5 cable.

Silver Sharpies for when you have to mark on something that has a dark finish.

share
    
words of wisdom! – pQd Jun 17 '09 at 20:03
    
Silver Sharpies ... I forgot all about those, I used to carry them everywhere. – Joseph Kern Mar 10 '10 at 11:26
    
They're also handy to have around when you want someone to autograph the bottom of your laptop. I keep a few in my backpack at all times. – Wesley Mar 10 '10 at 15:41

I use 'ss' which is part of the iproute package (which also includes the great ip and tc commands) to get information about tcp/udp sockets. It supplies more information that the regular 'netstat' and can be used with filters, for instance:

$ ss -o state established '( dport = :www or sport = :www )'
Recv-Q Send-Q Local  Address:Port                  Peer Address:Port   
0      0      192.168.22.100:52430                 209.85.173.118:www     
0      0      192.168.22.100:51115                 209.85.229.101:www     
0      0      192.168.22.100:49105                 209.85.227.104:www     
0      0      192.168.22.100:37792                 209.85.227.101:www     
0      0      192.168.22.100:60337                 69.59.196.212:www     
0      0      192.168.22.100:37794                 209.85.227.101:www
share
    
For that info, use netstat -nt|grep :80. It's more succinct, and if you make a habit of using the standard toolkit where it costs you nothing then you can hit the ground running on a new server, rather than mess about installing stuff. – mc0e Jun 18 '13 at 16:44

I already mentioned this tool in another answer on SF.
sysv-rc-conf

DESCRIPTION:
sysv-rc-conf gives an easy to use interface for manag‐ ing "/etc/rc{runlevel}.d/" symlinks.

alt text

If you have ubuntu : sudo apt-get install sysv-rc-conf

share
    
Ooh, and it's written in Perl! Awesome. – Brad Beyenhof Jun 6 '09 at 22:02
    
"On SF"? Go see who owns "sf.net" :) – Mei Jun 12 '09 at 20:26

"arping" to determine if a machine in the LAN is really up. Bypasses all blocking efforts of ICMP echo (ping), by using ARP packets.

share
    
There were two different arping programs out there: the only one I liked let you ping a host by MAC address - the other was an arping produced by Red Hat I believe. – Mei Jun 8 '09 at 15:27
1  
I should mention that the two arping programs are available in most distributions and that they conflict. The one is a part of iproute2, and the other is independent. Wikipedia describes both; the arping I like is by Thomas Habets. – Mei Sep 16 '11 at 14:40

couple of network tools:

tshark - real time text version of graphical network traffic analyser - wireshark. when raw output from tcpdump is just not enough for you tshark can do the trick. sample :

1041488.938623 192.168.10.14 -> 123.13.42.143 TCP 2525 > 25 [SYN] Seq=0 Len=0 MSS=1460
1041488.964593 192.168.10.14 -> 123.13.42.143 TCP 2525 > 25 [ACK] Seq=1 Ack=0 Win=65535 Len=0
1041488.997561 192.168.10.14 -> 123.13.42.143 SMTP Command: EHLO s72f30c9a2c784
1041489.034541 192.168.10.14 -> 123.13.42.143 SMTP Command: AUTH LOGIN
1041489.064026 192.168.10.14 -> 123.13.42.143 SMTP Message Body
1041489.095757 192.168.10.14 -> 123.13.42.143 SMTP Message Body
1041489.304390 192.168.10.14 -> 123.13.42.143 SMTP Command: MAIL FROM: <someone@somewhere.cx>
1041489.375849 192.168.10.14 -> 123.13.42.143 SMTP Command: RCPT TO: <anotherperson@wherever.se>
1041489.409579 192.168.10.14 -> 123.13.42.143 SMTP Command: DATA
1041489.470060 192.168.10.14 -> 123.13.42.143 SMTP Message Body
1041489.503278 192.168.10.14 -> 123.13.42.143 SMTP Message Body
1041489.529797 192.168.10.14 -> 123.13.42.143 SMTP EOM:
1041492.660752 192.168.10.14 -> 123.13.42.143 SMTP Command: QUIT
1041492.726452 192.168.10.14 -> 123.13.42.143 TCP 2525 > 25 [FIN, ACK] Seq=2362 Ack=281 Win=65254 Len=0
1041492.734770 192.168.10.14 -> 123.13.42.143 TCP 2525 > 25 [ACK] Seq=2363 Ack=282 Win=65254 Len=0

httpry real time passive http analyser. sample output:

06/06/2009 13:36:16     74.125.77.104   192.168.10.138  <       -       -       -       HTTP/1.1        204     No Content
06/06/2009 13:37:16     192.168.2.255   212.77.100.101  >       GET     wp.pl   /       HTTP/1.0        -       -
06/06/2009 13:37:16     212.77.100.101  192.168.2.255   <       -       -       -       HTTP/1.0        302     Found
06/06/2009 13:37:16     192.168.2.255   212.77.100.101  >       GET     www.wp.pl       /       HTTP/1.0        -       -
06/06/2009 13:37:16     212.77.100.101  192.168.2.255   <       -       -       -       HTTP/1.0        200     OK

both tools are very useful for troubleshooting some connectivity issues reported by users in remote offices / networks where you have access only to edge gateway.

and usual monitoring favorites:

  • nagios for alerting, bash or any other scripting language to write your own checks of anything you want to monitor
  • munin for plotting trend charts
share
    
Thanks again for the reminder of Wireshark's text component: I always forget about it (my loss). Next time I reach for tcpdump I'll have to switch and try it. – Mei Jun 12 '09 at 19:02

pv pipe viewer. Insert it between commands in a pipeline to get an indication how fast data is moving, how long till it finishes, etc. It can also act as a rate limiter.

share

strace is useful in a surprisingly large number of places.

share
    
Good call - I love this tool as well. – MikeyB Jun 12 '09 at 21:02
1  
mmm, What does it do? – Alex. S. Jun 12 '09 at 22:11
1  
strace man page - linux.die.net/man/1/strace – Cristian Ciupitu Aug 4 '09 at 14:22
    
Also, look up "perf trace". It can give similar information as strace but it is newer and uses a different subsystem in the kernel to gather its trace information (Linux Perf Counters). I haven't used it enough to get a feel for why you would want to use one of the over, but it's worth a look. – Noah Spurrier Apr 11 '14 at 2:12

Windows: Nbtstat (has a variety of tricks, but -A to get the machine name of a remote IP can be really handy assuming you're in an environment where permissions will let this work)

All of the sysinternals tools, (process explorer, debug view, etc...). Particularly debug view, you'd be surprised how many server programs write interesting/useful things to the debug log and that's the only way I know to view it. Those are must-haves though.

Unix: strace/ktrace/truss Are my favorite tools (best non-obvious use, if a there's a process running that has it's stderr redirected to /dev/null and you don't want to stop it, monitor the calls to write on fd2.

lsof: (list of open files) I always forget about this one, but when you need it, you need it.

share
6  
+1 for lsof - it's great – Dennis Williamson Jun 6 '09 at 16:54
    
Nbtstat is super-handy at getting a MAC address of a remote workstation too when you only know its name or IP. – Matt Hanson Jun 7 '09 at 5:55
    
nbtstat has a UNIX equivalent in open source but I can never find it. – Mei Jun 8 '09 at 15:38

I would say the oddest tool I've had to use was a Putty knife to open a Mac-Mini (as directed by Apple).

Think Different!

alt text

share
    
Upvoted for the pic. – Joseph Kern Jun 17 '09 at 18:33
1  
Been there, done that ^-^ – Berzemus Jun 17 '09 at 19:01
    
lol...That's high tech right thar! – Cube_Zombie Jun 17 '09 at 20:44
    
Ditto, been there doing that one myself. – Please Delete Me Jun 23 '09 at 12:40

Earplugs for longer actions in a noisy data center.

share
    
+1 because I recently had to work on a box where the RAID controller made an ungodly and painful screeching each time I booted up, until the drives were inserted in the correct order. – Nic Feb 27 '10 at 7:13

A lot of the suggestion above are Network related. But for a SysAdmnin:

htop - an interactive process viewer for Linux

This is htop, an interactive process viewer for Linux. It is a text-mode application (for console or X terminals) and requires ncurses. Tested with Linux 2.4 and 2.6.

Or Human Readable Top

http://htop.sourceforge.net/

share
    
I also like htop: very nice. With a name like htop, I'm also constantly reminded of atop: atcomputing.nl/Tools/atop. I usually install both on any Linux box I use. Htop is colorful, understandable, and well presented; atop is comprehensive and saves historical data as well. – Mei Jun 16 '09 at 22:31

A magentized paperclip on a string. I dropped a screw in a full rack once. It was w-a-y down there. I had a strong magnet stuck to my desk but it didn't have any way to attach a string to it. So I found a thicker paperclip and magnetized it after tying some pull-string to it. I g-e-n-t-l-y fed it down the left side of the rack in the space between the server rails and the cabinet door. It took some poking, but it found the screw I needed. It took a few tries before I could pull it all the way back to the top without dropping it again.

Delicate work. At the time I was swearing vociferously that I clearly hadn't played enough 'Operation' as a kid.

share
    
My dad has one of these in his toolbox; maplin.co.uk/free_uk_delivery/… . I keep meaning to steal it. – Murali Suriar Jun 17 '09 at 19:44
1  
+1 for ingenuity! – Kara Marfia Jun 17 '09 at 19:55

Pipe Cleaners -- They are better than twist ties for tying off cables. They are longer, softer on the fingers, and color coded.

share
    
Love the creativity =D – Antoine Benkemoun Apr 6 '10 at 14:49

Unix centric answer (simple tools) :

  • ntop (can't believe I haven't seen this yet)
  • tcpdump/snoop
  • double recommendation for sar
  • rsync (life would suck more than anything without it)
share

Multitail is a must. Monitor multiple files on one screen. Can also follow file names instead of descriptors.

alt text

share
2  
Nice, but how is that different than running the standard tail -f in two windows using the GNU screen utility? – Mei Jun 8 '09 at 15:32
    
Tail doesn't color on patterns. You also can't scroll back and view the backlog. – Marcin Aug 4 '09 at 12:39
    
Actually, you can scroll back: GNU screen has a scrollback buffer with search and copy built in. – Mei Sep 16 '11 at 14:44

I second the flashlight comments, though I prefer a headlamp over the minimag or such. Leaves your hands free.

I also find a click-style ball point pen to come in handy. Useful for straightening pins on VGA cables (when the pen is retracted), pressing reset buttons, and jabbing in your own throat when you find your file server has crashed and everything is lost.

share
3  
+1 for the headlamp...Found out how useful they were camping and just transferred use over to work.. – Cube_Zombie Jun 17 '09 at 20:46

vnstat comes in handy - a little traffic monitor that gives output in the same way as vmstat, e.g:

> eth1
> 
>            received:       1.40 TiB   (48.4%)
>         transmitted:       1.50 TiB   (51.6%)
>               total:       2.90 TiB   since 20.11.08
> 
>                         rx      |     tx      |   total
>         ------------------------+-------------+------------
>         yesterday      6.15 GiB |    8.50 GiB |   14.65 GiB
>             today      6.98 GiB |    4.77 GiB |   11.75 GiB
>         ------------------------+-------------+------------
>         estimated      7.82 GiB |    5.35 GiB |   13.17 GiB
share

Systemtap rocks. Systemtap provides you with very simple-to-grasp tools to write a kernel module that inspects various parts of the kernel. Pretty deep tech, but on occasion very, very useful.

Apart from that I'd say: a combination of sysstat and rrdtool for long term trend analysis.

share
    
Unfortunately, Ubuntu does not support systemtap out of the box; it requires hackery to get working. – Mei Sep 16 '11 at 14:41

Looks like your post was slanted towards Linux but here's one for windows even I didn't know about until recently:

getmac

C:\Admin>getmac

Physical Address    Transport Name
=================== ==========================================================
42-35-7C-4F-85-1b   \Device\Tcpip_{72338DC1-13A4-8514-2C1B-60FC3B4559DB}
00-11-05-86-D2-C0   \Device\Tcpip_{CCD25CFB-7765-1BE2-C59B-57C05FD32B67}

C:\Admin>
share
10  
getmac is a bit hopeless IMHO. Does anyone know which interface "\Device\Tcpip_{72338DC1-13A4-8514-2C1B-60FC3B4559DB}" is off the top of their head? I prefer "ipconfig /all" or even a PowerShell one-liner like "get-wmiobject win32_networkadapter | format-table name,netconnectionid,macaddress". – ThatGraemeGuy Jun 6 '09 at 16:47
    
The problem (and benefit) of Linux is that there's practically nothing you can't do. Even so, lots of these tools are available for other environments. – Mei Jun 8 '09 at 15:30

od - octal dump

strings - find printable strings in files

share
    
I once spent a fun afternoon running "strings" against the Linux kernel, and grepping the output for various rude words. Those kernel developers can be pretty foul-mouthed! – RainyRat Dec 31 '11 at 12:06

iftop, this gives a much better output than ntop in my opinion.

share

Programmable Bash completion functions. Make life less error-prone. They're not that hard to write.

Some of the less obvious options to GNU grep, especially --color, -P, and -o.

My standard quick reporting one-liner:

grep something /some/logfile | sort | uniq -c | sort -n

Simple frequency analysis from a logfile:

grep something /some/log | grep -o '^... .. ..:..' | uniq -c

(adjust the second bit for how granular you want the frequency breakdown to be).

share
    
No doubt I'm the little fish against the tide, but I think bash is an abomination..... (duck and run!) Seriously, every time I try and use it it makes my life miserable - and it's not ksh-compatible. One of the first commands I do in bash is "exec ksh -o vi" and that (usually) fixes it. – Mei Jun 12 '09 at 18:55
    
Not necessarily. It's no particular bash love, but more the good-enough-and-ubiquitous factor that keeps me on bash. Personally, I think of ksh as a good scripting shell but not a great interactive shell. I know you can technically do programmable completion in ksh, but it's icky compared to bash or zsh completion, and there's no ksh equivalent to the bash-completion project that I know of. (Oh, and you can set -o vi in bash too, you know). – Rob Chanter Jun 15 '09 at 0:37
    
The frequency analysis is great, do that all the time, even simpler with cut -c1-12 instead of grep if all lines have a strict data format - customarily true for syslog files. – Felix Frank Aug 13 '14 at 15:11

A good Minimag flashlight with fresh batteries is good to have when you need to peek inside a cabinet/server case/under a raised floor/etc.

share
1  
Or a cheap LED one that doesn't break the bulb when you drop it and has a nice long run time. Or a $65 fancy LED one with variable brightness from dim (8 lumens, and a run time in days) to blinding (215 lumens). But that's probably overkill. – Ronald Pottol Jun 18 '09 at 3:30
    
Maglite now makes their own LED lights, and they're much brighter than their standard light. – Joe Internet Mar 10 '10 at 18:51

Facter is a particular favourite.

It's frequently just considered as part of the larger Puppet configuration management framework, but extremely useful in its own right; giving a consistent cross-platform way of finding out core system information. Great in shell scripts, essentially.

For example:

  facter operatingsystem => ubuntu
  facter lsbdistcodename => hardy
  facter domain => mydnsdomain.com

It's also very easy to extend with your own (or other people's) 'facts', eg:

  facter local_postgresql_port => 5434
  facter has_hardware_raid => LSI
share

I'm going to go with a hacker tool, that might as well be useful to diagnose troubles: Cain. It can sniff a network and do a lot of attacks(arp poisoning, man in the middle,etc) on a network as well as breaking(or recovering) passwords.

What better way to know your network is secure.

share

Windows

Although all SysInterals tools are great, the tools which helped me the most, were the monitoring tools:

Nothing's better to quickly determine, why something is not working.

(* Regmon and Filemon are integrated into Process Monitor for newer Windows versions)

share

Not the answer you're looking for? Browse other questions tagged or ask your own question.