What tools should every sysadmin use that no one's heard of? [closed]

Question

I'm wondering about tools that are powerful and that most system administrators either don't know about or don't use (but should).

Thus, things like sudo, vi, emacs, dtrace, ps, and top are out. I have some ideas but I just hate to skew the statistics...

I'll just wait and see if anyone mentions my favorites.

Answer 1

iPerf

Iperf helps you run tests that measure maximum TCP and UDP bandwidth performance. It allows the tuning of various parameters and UDP characteristics reporting bandwidth, delay jitter, datagram loss: http://openmaniak.com/iperf.php

---

,And

MTR

MTR (My Trace Route) is also a pretty good tool. It combines the functionality of the 'traceroute' and 'ping' programs in a single network diagnostic tool. Like traceroute except it gives more network quality and network diagnostic info. Leave running to get real time stats. Reports best and worst round trip times in milliseconds... It shows latency, jitter (average/best/worst): http://www.bitwizard.nl/mtr/

use:

mtr server.domain.com (or IP)

• Add more fields with "O", type "LDRS NBAW V JMXI" and hit ENTER
• Type "n" to toggle DNS Off/On

Answer 2

Sysstat / Sar - superb historic monitoring for servers.

Network/CPU/Memory/IO/etc/etc brilliant if you need to examine why a machine went down.

For example load averages between 6am and 7am this morning:

hcooper@localhost:~$ sar -q -s 06:00:00 -e 07:30:00
Linux 2.4.27-3-686 (localhost)  06/06/09

06:05:01      runq-sz  plist-sz   ldavg-1   ldavg-5  ldavg-15
06:15:01            2       200      0.00      0.05      0.04
06:25:01            4       199      0.01      0.05      0.04
06:35:02            0       208      1.74      1.39      0.79
06:45:01            1       201      0.12      0.52      0.70
06:55:02            1       197      0.04      0.14      0.40
07:05:01            2       203      0.13      0.10      0.24
07:15:01            2       200      0.00      0.05      0.15
07:25:01            2       200      0.01      0.04      0.08
Average:            2       201      0.26      0.29      0.30

Answer 3

I use 'ss' which is part of the iproute package (which also includes the great ip and tc commands) to get information about tcp/udp sockets. It supplies more information that the regular 'netstat' and can be used with filters, for instance:

$ ss -o state established '( dport = :www or sport = :www )'
Recv-Q Send-Q Local  Address:Port                  Peer Address:Port   
0      0      192.168.22.100:52430                 209.85.173.118:www     
0      0      192.168.22.100:51115                 209.85.229.101:www     
0      0      192.168.22.100:49105                 209.85.227.104:www     
0      0      192.168.22.100:37792                 209.85.227.101:www     
0      0      192.168.22.100:60337                 69.59.196.212:www     
0      0      192.168.22.100:37794                 209.85.227.101:www

Answer 4

I already mentioned this tool in another answer on SF.
sysv-rc-conf

DESCRIPTION:
sysv-rc-conf gives an easy to use interface for manag‐ ing "/etc/rc{runlevel}.d/" symlinks.

If you have ubuntu : sudo apt-get install sysv-rc-conf

Answer 5

"arping" to determine if a machine in the LAN is really up. Bypasses all blocking efforts of ICMP echo (ping), by using ARP packets.

Answer 6

couple of network tools:

tshark - real time text version of graphical network traffic analyser - wireshark. when raw output from tcpdump is just not enough for you tshark can do the trick. sample :

1041488.938623 192.168.10.14 -> 123.13.42.143 TCP 2525 > 25 [SYN] Seq=0 Len=0 MSS=1460
1041488.964593 192.168.10.14 -> 123.13.42.143 TCP 2525 > 25 [ACK] Seq=1 Ack=0 Win=65535 Len=0
1041488.997561 192.168.10.14 -> 123.13.42.143 SMTP Command: EHLO s72f30c9a2c784
1041489.034541 192.168.10.14 -> 123.13.42.143 SMTP Command: AUTH LOGIN
1041489.064026 192.168.10.14 -> 123.13.42.143 SMTP Message Body
1041489.095757 192.168.10.14 -> 123.13.42.143 SMTP Message Body
1041489.304390 192.168.10.14 -> 123.13.42.143 SMTP Command: MAIL FROM: <someone@somewhere.cx>
1041489.375849 192.168.10.14 -> 123.13.42.143 SMTP Command: RCPT TO: <anotherperson@wherever.se>
1041489.409579 192.168.10.14 -> 123.13.42.143 SMTP Command: DATA
1041489.470060 192.168.10.14 -> 123.13.42.143 SMTP Message Body
1041489.503278 192.168.10.14 -> 123.13.42.143 SMTP Message Body
1041489.529797 192.168.10.14 -> 123.13.42.143 SMTP EOM:
1041492.660752 192.168.10.14 -> 123.13.42.143 SMTP Command: QUIT
1041492.726452 192.168.10.14 -> 123.13.42.143 TCP 2525 > 25 [FIN, ACK] Seq=2362 Ack=281 Win=65254 Len=0
1041492.734770 192.168.10.14 -> 123.13.42.143 TCP 2525 > 25 [ACK] Seq=2363 Ack=282 Win=65254 Len=0

httpry real time passive http analyser. sample output:

06/06/2009 13:36:16     74.125.77.104   192.168.10.138  <       -       -       -       HTTP/1.1        204     No Content
06/06/2009 13:37:16     192.168.2.255   212.77.100.101  >       GET     wp.pl   /       HTTP/1.0        -       -
06/06/2009 13:37:16     212.77.100.101  192.168.2.255   <       -       -       -       HTTP/1.0        302     Found
06/06/2009 13:37:16     192.168.2.255   212.77.100.101  >       GET     www.wp.pl       /       HTTP/1.0        -       -
06/06/2009 13:37:16     212.77.100.101  192.168.2.255   <       -       -       -       HTTP/1.0        200     OK

both tools are very useful for troubleshooting some connectivity issues reported by users in remote offices / networks where you have access only to edge gateway.

and usual monitoring favorites:

• nagios for alerting, bash or any other scripting language to write your own checks of anything you want to monitor
• munin for plotting trend charts

Answer 7

strace is useful in a surprisingly large number of places.

Answer 8

pv pipe viewer. Insert it between commands in a pipeline to get an indication how fast data is moving, how long till it finishes, etc. It can also act as a rate limiter.

Answer 9

Windows: Nbtstat (has a variety of tricks, but -A to get the machine name of a remote IP can be really handy assuming you're in an environment where permissions will let this work)

All of the sysinternals tools, (process explorer, debug view, etc...). Particularly debug view, you'd be surprised how many server programs write interesting/useful things to the debug log and that's the only way I know to view it. Those are must-haves though.

Unix: strace/ktrace/truss Are my favorite tools (best non-obvious use, if a there's a process running that has it's stderr redirected to /dev/null and you don't want to stop it, monitor the calls to write on fd2.

lsof: (list of open files) I always forget about this one, but when you need it, you need it.

Answer 10

A lot of the suggestion above are Network related. But for a SysAdmnin:

htop - an interactive process viewer for Linux

This is htop, an interactive process viewer for Linux. It is a text-mode application (for console or X terminals) and requires ncurses. Tested with Linux 2.4 and 2.6.

Or Human Readable Top

http://htop.sourceforge.net/

Answer 11

Unix centric answer (simple tools) :

• ntop (can't believe I haven't seen this yet)
• tcpdump/snoop
• double recommendation for sar
• rsync (life would suck more than anything without it)

Answer 12

Multitail is a must. Monitor multiple files on one screen. Can also follow file names instead of descriptors.

Answer 13

vnstat comes in handy - a little traffic monitor that gives output in the same way as vmstat, e.g:

> eth1
> 
>            received:       1.40 TiB   (48.4%)
>         transmitted:       1.50 TiB   (51.6%)
>               total:       2.90 TiB   since 20.11.08
> 
>                         rx      |     tx      |   total
>         ------------------------+-------------+------------
>         yesterday      6.15 GiB |    8.50 GiB |   14.65 GiB
>             today      6.98 GiB |    4.77 GiB |   11.75 GiB
>         ------------------------+-------------+------------
>         estimated      7.82 GiB |    5.35 GiB |   13.17 GiB

Answer 14

Systemtap rocks. Systemtap provides you with very simple-to-grasp tools to write a kernel module that inspects various parts of the kernel. Pretty deep tech, but on occasion very, very useful.

Apart from that I'd say: a combination of sysstat and rrdtool for long term trend analysis.

Answer 15

Looks like your post was slanted towards Linux but here's one for windows even I didn't know about until recently:

getmac

C:\Admin>getmac

Physical Address    Transport Name
=================== ==========================================================
42-35-7C-4F-85-1b   \Device\Tcpip_{72338DC1-13A4-8514-2C1B-60FC3B4559DB}
00-11-05-86-D2-C0   \Device\Tcpip_{CCD25CFB-7765-1BE2-C59B-57C05FD32B67}

C:\Admin>

Answer 16

od - octal dump

strings - find printable strings in files

Answer 17

Facter is a particular favourite.

It's frequently just considered as part of the larger Puppet configuration management framework, but extremely useful in its own right; giving a consistent cross-platform way of finding out core system information. Great in shell scripts, essentially.

For example:

  facter operatingsystem => ubuntu
  facter lsbdistcodename => hardy
  facter domain => mydnsdomain.com

It's also very easy to extend with your own (or other people's) 'facts', eg:

  facter local_postgresql_port => 5434
  facter has_hardware_raid => LSI

Answer 18

Programmable Bash completion functions. Make life less error-prone. They're not that hard to write.

Some of the less obvious options to GNU grep, especially --color, -P, and -o.

My standard quick reporting one-liner:

grep something /some/logfile | sort | uniq -c | sort -n

Simple frequency analysis from a logfile:

grep something /some/log | grep -o '^... .. ..:..' | uniq -c

(adjust the second bit for how granular you want the frequency breakdown to be).

Answer 19

I'm going to go with a hacker tool, that might as well be useful to diagnose troubles: Cain. It can sniff a network and do a lot of attacks(arp poisoning, man in the middle,etc) on a network as well as breaking(or recovering) passwords.

What better way to know your network is secure.

Answer 20

Windows

Although all SysInterals tools are great, the tools which helped me the most, were the monitoring tools:

• Regmon *
• Filemon *
• Process Monitor
• Autoruns

Nothing's better to quickly determine, why something is not working.

(* Regmon and Filemon are integrated into Process Monitor for newer Windows versions)

Answer 21

iftop, this gives a much better output than ntop in my opinion.

Answer 22

For Windows:

PATHPING is nice (and built in) for some advanced ping stats

I also like QCHECK : http://www.ixchariot.com/products/datasheets/qcheck.html It's a great GUI based simple network analyzer.

File Unlocker is a life saver at times even on servers: http://ccollomb.free.fr/unlocker/

Answer 23

fldiff is a graphical diff program

Answer 24

All our apps are clustered, so cssh is pretty handy.

scp - only mentioned because so few people use it, relying on FTP instead. I've always loved the ability to grab files instantly from remote hosts. Secure, quick, and doesn't need a dedicated daemon running.

Answer 25

One tool that I absolutely find indispensible is tcptraceroute [1] - this is a traceroute that does not use ICMP packets to perform timings, but uses TCP instead. This allows you to traceroute without regard to the typical blocking of ICMP: it works well.

Answer 26

Here is a few tools I have come across:

SYDI

Auto network documentation tool.

Orca

Messing with .msi files.

RichCopy

An advanced alternative to Robocopy.

ImgBurn

Best free burning software hands down.

PowerGUI

A nice PowerShell editor.

Your Brain!

A lot of people seem to forget they have one.

Answer 27

grep, awk, netcat - can't live without them

For monitoring I use NetMRG and Nagios.

Uptimed is also a nice tool to monitor uptime and avalibility.

Answer 28

I try not to use too many "nonstandard" tools. That is, tools that aren't easily available between operating systems or out of a particular distribution's software repository. I work on a Macbook, and I do testing on a variety of different Linux/Unix platforms.

That said, I like ohai(1). It is the node data collection tool used by Chef. It outputs data in JSON, so it can be manipulated with a variety of JSON parsing libraries. Since I work on Chef :-), it's a "nonstandard" tool available on every system I work with.

(1) Disclosure. I work for the company that wrote Chef and Ohai.

Answer 29

Odd answer in a way as it's not a tech tool directly - but I couldn't live without EverNote. It's a centralised note taking system, it has Windows, Mac, iPhone clients and lets you clip pits of web pages (with links to the original), has catergories, records images & voice notes.

It's what I use when my brain breaks :)

Answer 30

It's not exactly a small cli tool as such, but I can heartily recommend Zenoss Core for network and system monitoring. It has restored my faith in monitoring-and-alerting software.

Where tools like Nagios, Cacti, etc. all have steep learning curves and seem to be good at only one thing, most often you have a need for an all-in-one solution, and Zenoss Core provides that for free (and Free).

It's hard to summarize, but basically it's an application with web-interface that autodiscovers devices on your network, then monitors tons of parameters, logfiles, ports, services, software, hardware, amount-of-pages-printed, etc. If anything fails or crosses a threshold you get an alert. It's easy to get started: enable/install snmp on each device you want to monitor, install Zenoss on a server, and open the web interface.

The free Core version is good enough for small and medium businesses, while the commercial Professional and Enterprise versions provide some advanced features.

If you ever thought about Nagios, Cacti or the like, be sure to evaluate Zenoss as well.