Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

locked by voretaq7 Feb 27 '12 at 22:37

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

54 Answers 54

To get the ball going, I find screen to be essential:

When screen is called, it creates a single window with a shell in it (or the specified command) and then gets out of your way so that you can use the program as you normally would. Then, at any time, you can create new (full-screen) windows with other programs in them (including more shells), kill the current window, view a list of the active windows, turn output logging on and off, copy text between windows, view the scrollback history, switch between windows, etc. All windows run their programs completely independent of each other. Programs continue to run when their window is currently not visible and even when the whole screen session is detached from the users terminal.

OMG! screen again? – setatakahashi May 13 '09 at 16:07

Maybe I don't use these every day, but I use them frequently:

  • strace Check out what files are loaded by the process.
  • htop A nicer top.
  • mtr ping + traceroute combined
  • lynx/links/w3m In case you need console browsing
  • ettercap Great network sniffer (i prefer it over wireshark)
  • scripting bash Every *nix admin should know this.
  • A programming language. For the more complex things, stay away from bash scripting and use something like python/perl/ruby/tcl/... (I use Lisp)
  • Midnight Commander can be great for people who liked norton commander.
  • irssi You never know when you just want to go ask something on IRC.
  • wget / curl Download stuff from the command line.
  • scp Copy stuff over ssh
  • lftp / ncftp Good (scriptable) console FTP clients.
  • iotop Check what's stressing your disks
  • nmap good port scanner
Thanks for the htop and iotop tips! – Oddmund May 15 '09 at 13:14

I was surprised to find that you can run remote GUI applications over SSH, using the "-X" parameter. For example:

# on my machine
$ ssh -X linuxserver
# on remote machine
$ gedit /etc/my.cnf &

The gedit window appears on my local machine, editing the "my.cnf" file on the server.

I'm assuming this only works if your client machine has an X environment -- in other words, not on Windows. But it works great on my Mac!

using xming you can have it work on windows, too! – xkcd150 May 2 '09 at 4:04
SSH forwarding does encryption which slows everything down. If you're on a secure LAN you can just send programs from one Linux desktop to another Linux computers X session. Very handy. I used to run movies via mplayer like that on my main desktop which was plugged into the stero at one end of the lounge room. The mplayer audio would then go out the stereo but the image would got to my laptop (via ethernet) at the other end of the lounge room which was plugged into the data projector. Home movie awesomeness. – gyaresu May 3 '09 at 7:13
Instead of adding the -X each time you could update your ssh_config with a host line that matches your internal hosts and address space 'Host 192.168..*' and enable the option 'ForwardX11 yes' – Zoredache May 4 '09 at 7:52

lsof often gets ignored, its a very useful tool. lsof lets you view a list of every open file on the system, who / what is using it, etc.

For instance:

root@tower:~ # umount /mnt/hardy
umount: /mnt/hardy: device is busy
umount: /mnt/hardy: device is busy
root@tower:~ # lsof | grep /mnt/hardy
bash       5966       root  cwd       DIR      253,2     1024          2 /mnt/hardy
root@tower:~ #

Now I see that I'm logged into a shell in another terminal, and /mnt/hardy is my current working directory. So I can either kill that shell, or go to the other terminal and get out of that directory so it can be unmounted.

That's really a trivial example, its very handy for cleaning up the occasional 'bot' infestation too. The options are extensive, see man lsof for more.

fuser is another nice trick, if you're only interested in one file or one mountpoint. – bdonlan May 5 '09 at 6:21
Hidden for sure. I've been looking for this feature for ages. (I'm a noob though) – altCognito May 8 '09 at 13:29

To change to the last directory you were in:

cd -

Network stack can be left running after a system halt. I don't know if this is current with the 2.6.x series of kernels, but on older versions, you could configure the firewall/routing, then halt the system without a shutdown, leaving just the network stack running. This would allow you to make a cheap (although static) firewall that "can't be hacked" - because there's nothing to hack, as there are no programs or services, just the network stack portion of the kernel passing packets back and forth...

"Kernel"? What is this "Kernel" you speak of? When it goes into this mode, the only code running is the /network stack/. No task scheduling, no APIs, code, services, user space, nothing, zip, nada...just the faint sound of packets echoing through ethernet... Think of it as more of an embedded device with two functions in life - accept, and send, packets. I suppose you could "hack" this in the sense that you can try and sneak packets through, but there's nothing you can /directly/ attack in the stack itself. – Avery Payne May 29 '09 at 19:51

I like "locate" - a much easier way to find files than the gnarly "find . -name xxxx -print". Note that you have to use the updatedb command with it to a your file index up to date; see the man pages for details.

Luckily, many Linux distros include updatedb in cron.daily :-). – Matt Solnit May 2 '09 at 22:26
It's also much, much faster than find. – altCognito May 8 '09 at 13:27

I find "ngrep" really useful for debugging network code on remote servers without having to punt tcpdump files around:

ngrep -d any -W byline port 80

for example, will show you live HTTP requests and responses.

One other thing I've found useful frequently is the "-e" switch to strace:

strace -p <pid> -e trace=open

will show all open() syscalls for a given pid, and

strace -p <pid> -e trace=\!rt_sigprocmask

will exclude all calls to rt_sigprocmask() from the output (useful for debugging RoR code, which seems to make an awful lot of those calls when built to use pthreads).


DESCRIPTION Each manual page has a short description available within it. apropos searches the descriptions for instances of keyword.

gyaresu@debian:~/bin$ apropos ettercap
etter.conf (5)       - Ettercap configuration file
ettercap (8)         - (unknown subject)
ettercap_curses (8)  - (unknown subject)
ettercap_plugins (8) - (unknown subject)

Thankfully I've only needed a couple of times, but the Magic SysRq key still remains one of my all time favorite hidden features.


+1 Magic SysRQ is golden. Remember 'Raising Elephants Is So Utterly Boring' as a hint in which order to use the keys, though the order is not set in stone. Which is why RSEIUB is used above: it syncs data to disk before giving the 'terminate' and 'kill' commands instead of the other way around. Correct order is debatable. – wzzrd May 27 '09 at 10:48
 units -- conversion program

 The units program converts quantities expressed in various scales to their 
equivalents in other scales.  The units program can only handle multiplicative
scale changes.  It cannot convert Celsius to Fahrenheit, for example.
It works interactively by prompting the user for input:

     You have: meters
     You want: feet
             * 3.2808399
             / 0.3048

     You have: cm^3
     You want: gallons
             * 0.00026417205
             / 3785.4118

     You have: meters/s
     You want: furlongs/fortnight
             * 6012.8848
             / 0.00016630952

     You have: 1|2 inch
     You want: cm
             * 1.27
             / 0.78740157

It's not really that hidden, but may be so for those with no experience, but I really like being able to give lists and let bash expand them like in:

cp arq{,.bak}

which is the same as typing

cp arq arq.bak

I also use the history shortcuts (I don't think that's the right term, but...) like


to repeat the last command, or


to replace foo by bar in the last command


ssh-copy-id for transferring ssh keys. The old way was to cat the key over ssh and even older was to scp the key and then cat the key. If you are using a non-standard ssh port then this will do the trick for you ..

ssh-copy-id -i /path/to/key '-p nonstandardport hostname'

Other wise..

ssh-copy-id -i /path/to/key hostname


Virtual consoles. Most people know how to use (Ctrl-)Alt-F1 to get to the first console, etc., but what if you have more than 12? You can use Alt-Left and Alt-Right to cycle through the rest of them. :-D


I like the debian-goodies package:

Description: Small toolbox-style utilities for Debian systems
 These programs are designed to integrate with standard shell tools,
 extending them to operate on the Debian packaging system.
  dgrep  - Search all files in specified packages for a regex
  dglob  - Generate a list of package names which match a pattern
 These are also included, because they are useful and don't justify
 their own packages:
  debget       - Fetch a .deb for a package in APT's database
  dpigs        - Show which installed packages occupy the most space
  debman       - Easily view man pages from a binary .deb without extracting
  debmany      - Select manpages of installed or uninstalled packages
  checkrestart - Help to find and restart processes which are using old
                 versions of upgraded files (such as libraries)
  popbugs      - Display a customized release-critical bug list based on
                 packages you use (using popularity-contest data)
and also moreutils, which is basically awesome pipe tools on wheels:
Description: additional unix utilities
 This is a growing collection of the unix tools that nobody thought
 to write thirty years ago.
 So far, it includes the following utilities:
  - sponge: soak up standard input and write to a file
  - ifdata: get network interface info without parsing ifconfig output
  - ifne: run a program if the standard input is not empty
  - vidir: edit a directory in your text editor
  - vipe: insert a text editor into a pipe
  - ts: timestamp standard input
  - combine: combine the lines in two files using boolean operations
  - pee: tee standard input to pipes
  - zrun: automatically uncompress arguments to command
  - mispipe: pipe two commands, returning the exit status of the first
  - isutf8: check if a file or standard input is utf-8
  - lckdo: execute a program with a lock held

A great "feature" I use every day at work: The ability to have SSH listen on port 443 so I can create a tunnel which bypasses my work firewall, allowing me to run a local SOCKS proxy tunneled through SSH to my internet facing Linux server.

I can completely ignore my corporate firewall.

It always make me chuckle to think of the people who believe that you can "lock down" a network. If leave a single port for traffic to get out on, you're screwed. – baudtack May 22 '09 at 23:29

I've always liked "man hier" for helping people that are new to the system.


Perhaps one of the great features of Linux, as opposed to Windows, is that there are next to no hidden features by design. Hopefully if any undocumented features are important enough to know about the community does its job.


Here are a few I've used on a regular basis:

  • sar - shows system activity
  • vmstat - virtual memory stats
  • iostat - io stats
  • pkill - like pgrep but allows you to kill the returned process ID
  • xargs -I<string> - allows for replacing strings with piped data
  • at - schedule a task
  • tkdiff - graphical diff utility

These are probably not "hidden", but I find them extremely useful:

  • df -hk - show disk usage in human readable format
  • ls -ltr - list files sorted by date
  • while :; do...done - (Bash) replacement for watch if unavailable
  • perl -e - run a Perl snippet on the command-line
  • free -kt - show memory information (kilobytes w/total)

Start with this article with some great Linux tricks. My favorite is pstree, which displays your processes in a tree format so you can see which process spawned which.

curl beats the hell out of wget for pretty much anything you would use wget for.

zgrep and zless are great for searching gzipped log files so you don't have to pipe them through gunzip or leave uncompressed stuff around in /var/log.


The concept is that everything in Linux is a file.

All configurations are in text files, and everything in Linux is treated as a file. This is a much simpler approach which makes it very easy to change things in Linux. In Linux even your filesystem itself can be viewed as a file.


ESC + .

'Pastes' the last parameter from the previous line into the current prompt


ls -l /home/someuser/somedir/somefile

followed by

rm ESC + .

translates to

rm /home/someuser/somedir/somefile

I see the yes command very nice:

yes | do_you_agree

From man page:

       yes - output a string repeatedly until killed

       yes [STRING]...
       yes OPTION

It's open source. Nothing is "hidden" if you bother to look.

I'd have argued the opposite - if a feature is hidden in a closed source app, it's up to the dev shop to disclose it. In an open source app, the feature is there for everyone to see. Furthermore, the number of people actively developing is generally much larger (more people know and talk about 'hidden' features). Lastly, open source development is largely community oriented, so people generally tell each other about hidden features. – Tom Wright May 2 '09 at 15:35

The hidden feature is that there are no hidden features. The system provides a tremendous amount of power that you have complete access to. You have to understand every piece of it and every tool available to know what power is at your fingertips.

You should start by understanding every command in /bin, /sbin, /usr/bin, /usr/sbin, and every virtual file in /proc. Read the manpages, other documentation, and source as necessary.

If you aren't comfortable reading a man page or reading the source, that is your hidden feature.


grep, awk and sed


Those aren't very hidden – jayrdub May 1 '09 at 21:58
  • od - dump files in octal and other formats. Useful to check for example if there's some BOM crap in the beginning of file
  • file - what might be the filetype of given file?
  • lshw, lsusb, lspci - list hardware
  • tracepath - are we dealing with MTU problem here?
  • netwox contains 200+ network related tools
  • ip - for network config/info
  • sysctl - filesystem/network/kernel info
  • ebtables - iptables for bridge
  • vconfig - VLAN configuration
  • brctl - Bridge configuration
  • socat - netcat on steroids
  • ipgrab - tcpdump-like utility that prints detailed header information
  • dig - What's up with DNS server now?

tee is awesome. Output to screen and logfile? Check.


Just found this one today:

Reports information about your system's hardware as described in your system BIOS according to the SMBIOS/DMI standard. This information typically includes system manufacturer, model name, serial number, BIOS version, asset tag as well as a lot of other details of varying level of interest and reliability depending on the manufacturer. This will often include usage status for the CPU sockets, expansion slots (e.g. AGP, PCI, ISA) and memory module slots, and the list of I/O ports (e.g. serial, parallel, USB).

It definitely makes answering questions about somerandomserver27 at the colo facility a lot easier!


Not really a Linux thing, more of a Bash thing: process substitution,

diff some_local_file <(ssh somehost "cat some_remote_file")

This diffs a local and a remote file retrieved via SSH in one line.


Not the answer you're looking for? Browse other questions tagged or ask your own question.