41
votes

I'm wondering about tools that are powerful and that most system administrators either don't know about or don't use (but should).

For one thing, I like the possibility of finding out about a tool that is good and that I should be using - or at least, trying out. I also find that giving these tools their time in the sun (again) can be a positive, letting others know about the wonderful tools that are out there.

Thus, things like sudo, vi, emacs, dtrace, ps, and top are out. I have some ideas but I just hate to skew the statistics...

I'll just wait and see if anyone mentions my favorites.

7
  • 7
    There are so many 'what tools' questions, do we really need another?
    – Zoredache
    Jun 6, 2009 at 9:49
  • 4
    I just have to ask: if no one's heard of a tool, then how will it get upvotes?
    – quux
    Jun 7, 2009 at 2:52
  • 5
    +1 for camera phone. They are also incredibly useful when working with someone over the phone to troubleshoot a problem and there is a long error message on the screen. I wish all remote hands people had/made use of them.
    – Mark
    Jun 17, 2009 at 18:40
  • 1
    I've used it to document lights on the front panel and ports on the back panel (in this case, of a DEC Alpha system).
    – Mei
    Jun 17, 2009 at 23:18
  • 2
    Please mention one tool per answer. Aug 4, 2009 at 14:18

74 Answers 74

33
votes

iPerf

Iperf helps you run tests that measure maximum TCP and UDP bandwidth performance. It allows the tuning of various parameters and UDP characteristics reporting bandwidth, delay jitter, datagram loss: http://openmaniak.com/iperf.php


,And

MTR

MTR (My Trace Route) is also a pretty good tool. It combines the functionality of the 'traceroute' and 'ping' programs in a single network diagnostic tool. Like traceroute except it gives more network quality and network diagnostic info. Leave running to get real time stats. Reports best and worst round trip times in milliseconds... It shows latency, jitter (average/best/worst): http://www.bitwizard.nl/mtr/

use:

mtr server.domain.com (or IP)

  • Add more fields with "O", type "LDRS NBAW V JMXI" and hit ENTER
  • Type "n" to toggle DNS Off/On

alt text http://locobox.googlepages.com/mtrsavedw.jpg

3
  • Except for when they've blocked ICMP and allowed UDP. Then it's back to traceroute -PUDP.
    – sh-beta
    Oct 30, 2009 at 18:58
  • 1
    @sh-beta, or mtr -u ;)
    – l0c0b0x
    Oct 30, 2009 at 21:10
  • o.O I really, really hope that's a recent development. The hours I've wasted! Alas!
    – sh-beta
    Nov 2, 2009 at 19:40
24
votes

Paperclips: I've had to use paperclips to pick the locks on the front of a sun and dell server We've all used them to open a cd rom

HotPlug This thing is awesome. Want to move a server to the next rack over without turning it off? http://www.wiebetech.com/products/HotPlug.php

Blocks of wood The idiot before you not mount the server properly? The server too old to have rack mounts? Blocks of wood are your friend. It's ghetto, but works.

5
  • 1
    wow, blocks of wood...you're right, that's pretty ghetto. You're still getting +1 for the hotplug, though Jun 17, 2009 at 19:11
  • 3
    I've used blocks of wood to keep a UPS from crushing me while I mounted it in the rack. Jun 17, 2009 at 20:03
  • I'm with Matt - hotplug is today's best link. Jun 17, 2009 at 20:22
  • Hotplug link - wow.
    – Dan Carley
    Jun 17, 2009 at 21:56
  • Another vote for HotPlug. That looks awesome.
    – Nic
    Nov 7, 2009 at 20:46
19
votes

A bent paper clip to pop open a CD drive.

6
  • 5
    I started hanging them on the doors of all of our server racks - they're also perfect for getting at reset buttons that are recessed. People started asking what on earth paperclips were doing hanging from the doors until I explained. Now they agree it's a pretty good idea.
    – Mark
    Jun 17, 2009 at 18:47
  • 1
    I keep one in my wallet at all times. Something no one should ever be without! Jun 17, 2009 at 19:09
  • 1
    And today I learned how much more useful they are! Time to see if it'll pick a poweredge face panel lock... Jun 17, 2009 at 19:36
  • I used a few paper clips and some electrical tape to make a nice one with a handle that big handed people like me can use. Jun 17, 2009 at 20:01
  • I forgot! You can open plastic locking zip ties with them: standalone-sysadmin.blogspot.com/2008/10/… Jun 17, 2009 at 20:31
18
votes

Sysstat / Sar - superb historic monitoring for servers.

Network/CPU/Memory/IO/etc/etc brilliant if you need to examine why a machine went down.

For example load averages between 6am and 7am this morning:

hcooper@localhost:~$ sar -q -s 06:00:00 -e 07:30:00
Linux 2.4.27-3-686 (localhost)  06/06/09

06:05:01      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
06:15:01            2       200      0.00      0.05      0.04
06:25:01            4       199      0.01      0.05      0.04
06:35:02            0       208      1.74      1.39      0.79
06:45:01            1       201      0.12      0.52      0.70
06:55:02            1       197      0.04      0.14      0.40
07:05:01            2       203      0.13      0.10      0.24
07:15:01            2       200      0.00      0.05      0.15
07:25:01            2       200      0.01      0.04      0.08
Average:            2       201      0.26      0.29      0.30
1
  • I'd be surprised if an admin hasn't heard of this one: I'm continually surprised at Linux distributions that don't contain these venerable tools, and at other UNIXes that don't have these enabled by default.
    – Mei
    Jun 8, 2009 at 15:26
17
votes

A Paper Bag: To breathe into when you discover that the SQL script you wrote wasn't actually mashing up the test database after all. Oh, and your backup system crashes and now you have to wait on the phone with tech support because you don't have a support contract.

Origami Paper: To pass the time when you're on hold with tier 1 tech support at said backup software vendor.

A Stress Ball: To help when discussing the problem with Tier 1 support after they finally answer the phone.

An Ice Pack: To soothe your aching fist after pounding it into the desk when you realize it took 90 minutes just to get handed off to Tier 2 support.

A Spare Cell Phone: After you throw yours into the ground when you get disconnected just as Tier 3 support answers your call.

An Account at Monster.com and Experience in Laying Carpet: =(

--

On a slightly more serious note, having a small refridgerator can be handy especially for putting crashed hard drives in to try and recover data.

Also, it's even better if your camera phone can take video. I take video when I'm messing with BIOSs and other things that I don't have time to simultaneously document while editing.

Cough drops and nose spray for extended stays in the dry server room.

A P-Touch Labeler to label EVERYTHING even if you're certain you'll remember which patch cables connect to the security cameras.

Fingernail clippers to remove the sheath from the 4 pairs of wires in a Cat5 cable.

Silver Sharpies for when you have to mark on something that has a dark finish.

3
  • words of wisdom!
    – pQd
    Jun 17, 2009 at 20:03
  • Silver Sharpies ... I forgot all about those, I used to carry them everywhere. Mar 10, 2010 at 11:26
  • They're also handy to have around when you want someone to autograph the bottom of your laptop. I keep a few in my backpack at all times.
    – Wesley
    Mar 10, 2010 at 15:41
16
votes

I use 'ss' which is part of the iproute package (which also includes the great ip and tc commands) to get information about tcp/udp sockets. It supplies more information that the regular 'netstat' and can be used with filters, for instance:

$ ss -o state established '( dport = :www or sport = :www )'
Recv-Q Send-Q Local  Address:Port                  Peer Address:Port   
0      0      192.168.22.100:52430                 209.85.173.118:www     
0      0      192.168.22.100:51115                 209.85.229.101:www     
0      0      192.168.22.100:49105                 209.85.227.104:www     
0      0      192.168.22.100:37792                 209.85.227.101:www     
0      0      192.168.22.100:60337                 69.59.196.212:www     
0      0      192.168.22.100:37794                 209.85.227.101:www   
1
  • For that info, use netstat -nt|grep :80. It's more succinct, and if you make a habit of using the standard toolkit where it costs you nothing then you can hit the ground running on a new server, rather than mess about installing stuff.
    – mc0e
    Jun 18, 2013 at 16:44
16
votes

I already mentioned this tool in another answer on SF.
sysv-rc-conf

DESCRIPTION:
sysv-rc-conf gives an easy to use interface for manag‐ ing "/etc/rc{runlevel}.d/" symlinks.

alt text http://blog.ubuntu-tweak.com/wp-content/uploads/2007/09/sysv-rc-conf.png

If you have ubuntu : sudo apt-get install sysv-rc-conf

2
  • Ooh, and it's written in Perl! Awesome. Jun 6, 2009 at 22:02
  • "On SF"? Go see who owns "sf.net" :)
    – Mei
    Jun 12, 2009 at 20:26
15
votes

"arping" to determine if a machine in the LAN is really up. Bypasses all blocking efforts of ICMP echo (ping), by using ARP packets.

2
  • There were two different arping programs out there: the only one I liked let you ping a host by MAC address - the other was an arping produced by Red Hat I believe.
    – Mei
    Jun 8, 2009 at 15:27
  • 1
    I should mention that the two arping programs are available in most distributions and that they conflict. The one is a part of iproute2, and the other is independent. Wikipedia describes both; the arping I like is by Thomas Habets.
    – Mei
    Sep 16, 2011 at 14:40
13
votes

couple of network tools:

tshark - real time text version of graphical network traffic analyser - wireshark. when raw output from tcpdump is just not enough for you tshark can do the trick. sample :

1041488.938623 192.168.10.14 -> 123.13.42.143 TCP 2525 > 25 [SYN] Seq=0 Len=0 MSS=1460
1041488.964593 192.168.10.14 -> 123.13.42.143 TCP 2525 > 25 [ACK] Seq=1 Ack=0 Win=65535 Len=0
1041488.997561 192.168.10.14 -> 123.13.42.143 SMTP Command: EHLO s72f30c9a2c784
1041489.034541 192.168.10.14 -> 123.13.42.143 SMTP Command: AUTH LOGIN
1041489.064026 192.168.10.14 -> 123.13.42.143 SMTP Message Body
1041489.095757 192.168.10.14 -> 123.13.42.143 SMTP Message Body
1041489.304390 192.168.10.14 -> 123.13.42.143 SMTP Command: MAIL FROM: <[email protected]>
1041489.375849 192.168.10.14 -> 123.13.42.143 SMTP Command: RCPT TO: <[email protected]>
1041489.409579 192.168.10.14 -> 123.13.42.143 SMTP Command: DATA
1041489.470060 192.168.10.14 -> 123.13.42.143 SMTP Message Body
1041489.503278 192.168.10.14 -> 123.13.42.143 SMTP Message Body
1041489.529797 192.168.10.14 -> 123.13.42.143 SMTP EOM:
1041492.660752 192.168.10.14 -> 123.13.42.143 SMTP Command: QUIT
1041492.726452 192.168.10.14 -> 123.13.42.143 TCP 2525 > 25 [FIN, ACK] Seq=2362 Ack=281 Win=65254 Len=0
1041492.734770 192.168.10.14 -> 123.13.42.143 TCP 2525 > 25 [ACK] Seq=2363 Ack=282 Win=65254 Len=0

httpry real time passive http analyser. sample output:

06/06/2009 13:36:16     74.125.77.104   192.168.10.138  <       -       -       -       HTTP/1.1        204     No Content
06/06/2009 13:37:16     192.168.2.255   212.77.100.101  >       GET     wp.pl   /       HTTP/1.0        -       -
06/06/2009 13:37:16     212.77.100.101  192.168.2.255   <       -       -       -       HTTP/1.0        302     Found
06/06/2009 13:37:16     192.168.2.255   212.77.100.101  >       GET     www.wp.pl       /       HTTP/1.0        -       -
06/06/2009 13:37:16     212.77.100.101  192.168.2.255   <       -       -       -       HTTP/1.0        200     OK

both tools are very useful for troubleshooting some connectivity issues reported by users in remote offices / networks where you have access only to edge gateway.

and usual monitoring favorites:

  • nagios for alerting, bash or any other scripting language to write your own checks of anything you want to monitor
  • munin for plotting trend charts
1
  • Thanks again for the reminder of Wireshark's text component: I always forget about it (my loss). Next time I reach for tcpdump I'll have to switch and try it.
    – Mei
    Jun 12, 2009 at 19:02
13
votes

pv pipe viewer. Insert it between commands in a pipeline to get an indication how fast data is moving, how long till it finishes, etc. It can also act as a rate limiter.

11
votes

strace is useful in a surprisingly large number of places.

4
  • Good call - I love this tool as well.
    – MikeyB
    Jun 12, 2009 at 21:02
  • 1
    mmm, What does it do?
    – Alex. S.
    Jun 12, 2009 at 22:11
  • 1
    strace man page - linux.die.net/man/1/strace Aug 4, 2009 at 14:22
  • Also, look up "perf trace". It can give similar information as strace but it is newer and uses a different subsystem in the kernel to gather its trace information (Linux Perf Counters). I haven't used it enough to get a feel for why you would want to use one of the over, but it's worth a look. Apr 11, 2014 at 2:12
10
votes

Windows: Nbtstat (has a variety of tricks, but -A to get the machine name of a remote IP can be really handy assuming you're in an environment where permissions will let this work)

All of the sysinternals tools, (process explorer, debug view, etc...). Particularly debug view, you'd be surprised how many server programs write interesting/useful things to the debug log and that's the only way I know to view it. Those are must-haves though.

Unix: strace/ktrace/truss Are my favorite tools (best non-obvious use, if a there's a process running that has it's stderr redirected to /dev/null and you don't want to stop it, monitor the calls to write on fd2.

lsof: (list of open files) I always forget about this one, but when you need it, you need it.

2
  • Nbtstat is super-handy at getting a MAC address of a remote workstation too when you only know its name or IP. Jun 7, 2009 at 5:55
  • nbtstat has a UNIX equivalent in open source but I can never find it.
    – Mei
    Jun 8, 2009 at 15:38
9
votes

I would say the oddest tool I've had to use was a Putty knife to open a Mac-Mini (as directed by Apple).

Think Different!

alt text http://locobox.googlepages.com/putty.jpg

4
  • Upvoted for the pic. Jun 17, 2009 at 18:33
  • 1
    Been there, done that ^-^
    – Berzemus
    Jun 17, 2009 at 19:01
  • lol...That's high tech right thar! Jun 17, 2009 at 20:44
  • Ditto, been there doing that one myself.
    – Please Delete Me
    Jun 23, 2009 at 12:40
9
votes

Earplugs for longer actions in a noisy data center.

1
  • +1 because I recently had to work on a box where the RAID controller made an ungodly and painful screeching each time I booted up, until the drives were inserted in the correct order.
    – Nic
    Feb 27, 2010 at 7:13
8
votes

A lot of the suggestion above are Network related. But for a SysAdmnin:

htop - an interactive process viewer for Linux

This is htop, an interactive process viewer for Linux. It is a text-mode application (for console or X terminals) and requires ncurses. Tested with Linux 2.4 and 2.6.

Or Human Readable Top

http://htop.sourceforge.net/

1
  • I also like htop: very nice. With a name like htop, I'm also constantly reminded of atop: atcomputing.nl/Tools/atop. I usually install both on any Linux box I use. Htop is colorful, understandable, and well presented; atop is comprehensive and saves historical data as well.
    – Mei
    Jun 16, 2009 at 22:31
8
votes

A magentized paperclip on a string. I dropped a screw in a full rack once. It was w-a-y down there. I had a strong magnet stuck to my desk but it didn't have any way to attach a string to it. So I found a thicker paperclip and magnetized it after tying some pull-string to it. I g-e-n-t-l-y fed it down the left side of the rack in the space between the server rails and the cabinet door. It took some poking, but it found the screw I needed. It took a few tries before I could pull it all the way back to the top without dropping it again.

Delicate work. At the time I was swearing vociferously that I clearly hadn't played enough 'Operation' as a kid.

1
8
votes

Pipe Cleaners -- They are better than twist ties for tying off cables. They are longer, softer on the fingers, and color coded.

1
7
votes

Unix centric answer (simple tools) :

  • ntop (can't believe I haven't seen this yet)
  • tcpdump/snoop
  • double recommendation for sar
  • rsync (life would suck more than anything without it)
7
votes

Multitail is a must. Monitor multiple files on one screen. Can also follow file names instead of descriptors.

alt text

3
  • 2
    Nice, but how is that different than running the standard tail -f in two windows using the GNU screen utility?
    – Mei
    Jun 8, 2009 at 15:32
  • Tail doesn't color on patterns. You also can't scroll back and view the backlog.
    – Marcin
    Aug 4, 2009 at 12:39
  • Actually, you can scroll back: GNU screen has a scrollback buffer with search and copy built in.
    – Mei
    Sep 16, 2011 at 14:44
6
votes

I second the flashlight comments, though I prefer a headlamp over the minimag or such. Leaves your hands free.

I also find a click-style ball point pen to come in handy. Useful for straightening pins on VGA cables (when the pen is retracted), pressing reset buttons, and jabbing in your own throat when you find your file server has crashed and everything is lost.

1
  • 3
    +1 for the headlamp...Found out how useful they were camping and just transferred use over to work.. Jun 17, 2009 at 20:46
6
votes

vnstat comes in handy - a little traffic monitor that gives output in the same way as vmstat, e.g:

> eth1
> 
>            received:       1.40 TiB   (48.4%)
>         transmitted:       1.50 TiB   (51.6%)
>               total:       2.90 TiB   since 20.11.08
> 
>                         rx      |     tx      |   total
>         ------------------------+-------------+------------
>         yesterday      6.15 GiB |    8.50 GiB |   14.65 GiB
>             today      6.98 GiB |    4.77 GiB |   11.75 GiB
>         ------------------------+-------------+------------
>         estimated      7.82 GiB |    5.35 GiB |   13.17 GiB
5
votes

Systemtap rocks. Systemtap provides you with very simple-to-grasp tools to write a kernel module that inspects various parts of the kernel. Pretty deep tech, but on occasion very, very useful.

Apart from that I'd say: a combination of sysstat and rrdtool for long term trend analysis.

1
  • Unfortunately, Ubuntu does not support systemtap out of the box; it requires hackery to get working.
    – Mei
    Sep 16, 2011 at 14:41
5
votes

Looks like your post was slanted towards Linux but here's one for windows even I didn't know about until recently:

getmac

C:\Admin>getmac

Physical Address    Transport Name
=================== ==========================================================
42-35-7C-4F-85-1b   \Device\Tcpip_{72338DC1-13A4-8514-2C1B-60FC3B4559DB}
00-11-05-86-D2-C0   \Device\Tcpip_{CCD25CFB-7765-1BE2-C59B-57C05FD32B67}

C:\Admin>
2
  • 10
    getmac is a bit hopeless IMHO. Does anyone know which interface "\Device\Tcpip_{72338DC1-13A4-8514-2C1B-60FC3B4559DB}" is off the top of their head? I prefer "ipconfig /all" or even a PowerShell one-liner like "get-wmiobject win32_networkadapter | format-table name,netconnectionid,macaddress". Jun 6, 2009 at 16:47
  • The problem (and benefit) of Linux is that there's practically nothing you can't do. Even so, lots of these tools are available for other environments.
    – Mei
    Jun 8, 2009 at 15:30
5
votes

od - octal dump

strings - find printable strings in files

1
  • I once spent a fun afternoon running "strings" against the Linux kernel, and grepping the output for various rude words. Those kernel developers can be pretty foul-mouthed!
    – RainyRat
    Dec 31, 2011 at 12:06
5
votes

iftop, this gives a much better output than ntop in my opinion.

5
votes

Programmable Bash completion functions. Make life less error-prone. They're not that hard to write.

Some of the less obvious options to GNU grep, especially --color, -P, and -o.

My standard quick reporting one-liner:

grep something /some/logfile | sort | uniq -c | sort -n

Simple frequency analysis from a logfile:

grep something /some/log | grep -o '^... .. ..:..' | uniq -c

(adjust the second bit for how granular you want the frequency breakdown to be).

3
  • No doubt I'm the little fish against the tide, but I think bash is an abomination..... (duck and run!) Seriously, every time I try and use it it makes my life miserable - and it's not ksh-compatible. One of the first commands I do in bash is "exec ksh -o vi" and that (usually) fixes it.
    – Mei
    Jun 12, 2009 at 18:55
  • Not necessarily. It's no particular bash love, but more the good-enough-and-ubiquitous factor that keeps me on bash. Personally, I think of ksh as a good scripting shell but not a great interactive shell. I know you can technically do programmable completion in ksh, but it's icky compared to bash or zsh completion, and there's no ksh equivalent to the bash-completion project that I know of. (Oh, and you can set -o vi in bash too, you know). Jun 15, 2009 at 0:37
  • The frequency analysis is great, do that all the time, even simpler with cut -c1-12 instead of grep if all lines have a strict data format - customarily true for syslog files. Aug 13, 2014 at 15:11
5
votes

A good Minimag flashlight with fresh batteries is good to have when you need to peek inside a cabinet/server case/under a raised floor/etc.

2
  • 1
    Or a cheap LED one that doesn't break the bulb when you drop it and has a nice long run time. Or a $65 fancy LED one with variable brightness from dim (8 lumens, and a run time in days) to blinding (215 lumens). But that's probably overkill. Jun 18, 2009 at 3:30
  • Maglite now makes their own LED lights, and they're much brighter than their standard light. Mar 10, 2010 at 18:51
4
votes

Facter is a particular favourite.

It's frequently just considered as part of the larger Puppet configuration management framework, but extremely useful in its own right; giving a consistent cross-platform way of finding out core system information. Great in shell scripts, essentially.

For example:

  facter operatingsystem => ubuntu
  facter lsbdistcodename => hardy
  facter domain => mydnsdomain.com

It's also very easy to extend with your own (or other people's) 'facts', eg:

  facter local_postgresql_port => 5434
  facter has_hardware_raid => LSI
4
votes

I'm going to go with a hacker tool, that might as well be useful to diagnose troubles: Cain. It can sniff a network and do a lot of attacks(arp poisoning, man in the middle,etc) on a network as well as breaking(or recovering) passwords.

What better way to know your network is secure.

4
votes

Windows

Although all SysInterals tools are great, the tools which helped me the most, were the monitoring tools:

Nothing's better to quickly determine, why something is not working.

(* Regmon and Filemon are integrated into Process Monitor for newer Windows versions)

Not the answer you're looking for? Browse other questions tagged .