Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

locked by voretaq7 Feb 27 '12 at 22:37

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

54 Answers 54

To get the ball going, I find screen to be essential:

When screen is called, it creates a single window with a shell in it (or the specified command) and then gets out of your way so that you can use the program as you normally would. Then, at any time, you can create new (full-screen) windows with other programs in them (including more shells), kill the current window, view a list of the active windows, turn output logging on and off, copy text between windows, view the scrollback history, switch between windows, etc. All windows run their programs completely independent of each other. Programs continue to run when their window is currently not visible and even when the whole screen session is detached from the users terminal.

share
    
has anyone got anything to say about the screen profiles rolling out with ubuntu 9.04? i looked at them for a bit but didn't give them enough time to be able to say if they're actually useful or just look nice – xkcd150 May 2 '09 at 4:02
12  
OMG! screen again? – setatakahashi May 13 '09 at 16:07
    
@xkcd150, they're pretty similar, but the biggest win is the status line. It shows all your windows plus a heap of stuff about the machine. – staticsan Feb 5 '10 at 3:08
    
And if you want a good version of screen use tmux ;) – Good Person Jan 17 '12 at 15:11

Maybe I don't use these every day, but I use them frequently:

  • strace Check out what files are loaded by the process.
  • htop A nicer top.
  • mtr ping + traceroute combined
  • lynx/links/w3m In case you need console browsing
  • ettercap Great network sniffer (i prefer it over wireshark)
  • scripting bash Every *nix admin should know this.
  • A programming language. For the more complex things, stay away from bash scripting and use something like python/perl/ruby/tcl/... (I use Lisp)
  • Midnight Commander can be great for people who liked norton commander.
  • irssi You never know when you just want to go ask something on IRC.
  • wget / curl Download stuff from the command line.
  • scp Copy stuff over ssh
  • lftp / ncftp Good (scriptable) console FTP clients.
  • iotop Check what's stressing your disks
  • nmap good port scanner
share
    
+1 for strace -- absolutely fantastic – bedwyr May 14 '09 at 1:56
1  
Thanks for the htop and iotop tips! – Oddmund May 15 '09 at 13:14
    
htop is awesome, many thanks. – The Pixel Developer Jun 10 '09 at 21:16
    
I would add atop too – Pier May 23 '10 at 17:08
    
Both the TLDP bash documents are very bad. – adaptr Dec 1 '11 at 14:59

I was surprised to find that you can run remote GUI applications over SSH, using the "-X" parameter. For example:

# on my machine
$ ssh -X linuxserver
# on remote machine
$ gedit /etc/my.cnf &

The gedit window appears on my local machine, editing the "my.cnf" file on the server.

I'm assuming this only works if your client machine has an X environment -- in other words, not on Windows. But it works great on my Mac!

share
9  
using xming sourceforge.net/projects/xming you can have it work on windows, too! – xkcd150 May 2 '09 at 4:04
3  
SSH forwarding does encryption which slows everything down. If you're on a secure LAN you can just send programs from one Linux desktop to another Linux computers X session. Very handy. I used to run movies via mplayer like that on my main desktop which was plugged into the stero at one end of the lounge room. The mplayer audio would then go out the stereo but the image would got to my laptop (via ethernet) at the other end of the lounge room which was plugged into the data projector. Home movie awesomeness. – gyaresu May 3 '09 at 7:13
5  
Instead of adding the -X each time you could update your ssh_config with a host line that matches your internal hosts and address space 'Host .domain.org 192.168..*' and enable the option 'ForwardX11 yes' – Zoredache May 4 '09 at 7:52
    
I've grown accustomed to -Y, never learned the difference though. – Notmyfault Jun 25 '09 at 5:52
    
-Y is trusted X11 forwarding. It's way more secure than -X (simple X11 forwarding). BTW many default sshd setups won't allow -X now, only -Y. If using -X you encounter strange errors, apps closing with "MIT MAGIC COOKIE" errors, then use -Y. Conclusion : don't use -X, only -Y – wazoox Jun 26 '09 at 9:42

lsof often gets ignored, its a very useful tool. lsof lets you view a list of every open file on the system, who / what is using it, etc.

For instance:

root@tower:~ # umount /mnt/hardy
umount: /mnt/hardy: device is busy
umount: /mnt/hardy: device is busy
root@tower:~ # lsof | grep /mnt/hardy
bash       5966       root  cwd       DIR      253,2     1024          2 /mnt/hardy
root@tower:~ #

Now I see that I'm logged into a shell in another terminal, and /mnt/hardy is my current working directory. So I can either kill that shell, or go to the other terminal and get out of that directory so it can be unmounted.

That's really a trivial example, its very handy for cleaning up the occasional 'bot' infestation too. The options are extensive, see man lsof for more.

share
2  
fuser is another nice trick, if you're only interested in one file or one mountpoint. – bdonlan May 5 '09 at 6:21
1  
Hidden for sure. I've been looking for this feature for ages. (I'm a noob though) – altCognito May 8 '09 at 13:29
    
lsof is truly wonderful. Combined with the UNIX idea of everything (well, almost) being a file, it can do amazing things. Many things that at first seem impossible or hard to do turn out to be solvable with a oneliner thanks to some of lsof's options. – ptman May 23 '10 at 17:21

To change to the last directory you were in:

cd -
share
    
I use this all the time. – MathewC Jun 26 '09 at 12:21

Network stack can be left running after a system halt. I don't know if this is current with the 2.6.x series of kernels, but on older versions, you could configure the firewall/routing, then halt the system without a shutdown, leaving just the network stack running. This would allow you to make a cheap (although static) firewall that "can't be hacked" - because there's nothing to hack, as there are no programs or services, just the network stack portion of the kernel passing packets back and forth...

share
    
Wow. Really cool! +1 I am certainly going to try this out on a 2.6 kernel when I have some spare time. – wzzrd May 27 '09 at 10:43
    
That's awesome, but how is it unhackable? Doesn't the kernel still have to be running? Or is it a minimal kernel? – cdeszaq May 29 '09 at 18:42
7  
"Kernel"? What is this "Kernel" you speak of? When it goes into this mode, the only code running is the /network stack/. No task scheduling, no APIs, code, services, user space, nothing, zip, nada...just the faint sound of packets echoing through ethernet... Think of it as more of an embedded device with two functions in life - accept, and send, packets. I suppose you could "hack" this in the sense that you can try and sneak packets through, but there's nothing you can /directly/ attack in the stack itself. – Avery Payne May 29 '09 at 19:51

I like "locate" - a much easier way to find files than the gnarly "find . -name xxxx -print". Note that you have to use the updatedb command with it to a your file index up to date; see the man pages for details.

share
4  
Luckily, many Linux distros include updatedb in cron.daily :-). – Matt Solnit May 2 '09 at 22:26
1  
It's also much, much faster than find. – altCognito May 8 '09 at 13:27
    
It should be pointed out that some people see locate as a security risk. That being said, I use it all the time :-) – baudtack May 22 '09 at 23:26
    
Recent verions of locate have the -i switch for ignoring case, very handy. – niXar Sep 23 '09 at 10:00
    
And as to security, there are secure locate versions, which let you "find" only those files the logged-in user can actually see. – Jürgen A. Erhard Jan 8 '10 at 12:54

I find "ngrep" really useful for debugging network code on remote servers without having to punt tcpdump files around:

ngrep -d any -W byline port 80

for example, will show you live HTTP requests and responses.

One other thing I've found useful frequently is the "-e" switch to strace:

strace -p <pid> -e trace=open

will show all open() syscalls for a given pid, and

strace -p <pid> -e trace=\!rt_sigprocmask

will exclude all calls to rt_sigprocmask() from the output (useful for debugging RoR code, which seems to make an awful lot of those calls when built to use pthreads).

share
    
Now I really starting to love linux. :) – THEn May 13 '10 at 14:35
apropos

DESCRIPTION Each manual page has a short description available within it. apropos searches the descriptions for instances of keyword.

gyaresu@debian:~/bin$ apropos ettercap
etter.conf (5)       - Ettercap configuration file
ettercap (8)         - (unknown subject)
ettercap_curses (8)  - (unknown subject)
ettercap_plugins (8) - (unknown subject)
share
    
I always useman -k instead but I'm spelling impaired. – mctylr Mar 11 '10 at 19:18

Thankfully I've only needed a couple of times, but the Magic SysRq key still remains one of my all time favorite hidden features.

Alt+SysRq+RSEIUB

share
1  
+1 Magic SysRQ is golden. Remember 'Raising Elephants Is So Utterly Boring' as a hint in which order to use the keys, though the order is not set in stone. Which is why RSEIUB is used above: it syncs data to disk before giving the 'terminate' and 'kill' commands instead of the other way around. Correct order is debatable. – wzzrd May 27 '09 at 10:48
    
+1 Certainly a brilliant one to remember. – Coops Jun 6 '09 at 14:07
    
Another memory aid: BUSIER (spelled backwards). – Mei May 18 '11 at 16:53
NAME
 units -- conversion program

DESCRIPTION
 The units program converts quantities expressed in various scales to their 
equivalents in other scales.  The units program can only handle multiplicative
scale changes.  It cannot convert Celsius to Fahrenheit, for example.
It works interactively by prompting the user for input:

     You have: meters
     You want: feet
             * 3.2808399
             / 0.3048

     You have: cm^3
     You want: gallons
             * 0.00026417205
             / 3785.4118

     You have: meters/s
     You want: furlongs/fortnight
             * 6012.8848
             / 0.00016630952

     You have: 1|2 inch
     You want: cm
             * 1.27
             / 0.78740157
share

It's not really that hidden, but may be so for those with no experience, but I really like being able to give lists and let bash expand them like in:

cp arq{,.bak}

which is the same as typing

cp arq arq.bak

I also use the history shortcuts (I don't think that's the right term, but...) like

!!

to repeat the last command, or

^foo^bar

to replace foo by bar in the last command

share

ssh-copy-id for transferring ssh keys. The old way was to cat the key over ssh and even older was to scp the key and then cat the key. If you are using a non-standard ssh port then this will do the trick for you ..

ssh-copy-id -i /path/to/key '-p nonstandardport hostname'

Other wise..

ssh-copy-id -i /path/to/key hostname

share

Virtual consoles. Most people know how to use (Ctrl-)Alt-F1 to get to the first console, etc., but what if you have more than 12? You can use Alt-Left and Alt-Right to cycle through the rest of them. :-D

share
    
wow, dude, at that point, just use screen. hehe :) i have it in my .screenrc to show a taskbar at the bottom letting me know screens are open – Roy Rico Mar 11 '10 at 23:54
    
and what id the 12'th console is running a desktop environment(e.g. gnome)? – Behrooz Apr 23 '11 at 12:29

I like the debian-goodies package:

Description: Small toolbox-style utilities for Debian systems
 These programs are designed to integrate with standard shell tools,
 extending them to operate on the Debian packaging system.
 .
  dgrep  - Search all files in specified packages for a regex
  dglob  - Generate a list of package names which match a pattern
 .
 These are also included, because they are useful and don't justify
 their own packages:
 .
  debget       - Fetch a .deb for a package in APT's database
  dpigs        - Show which installed packages occupy the most space
  debman       - Easily view man pages from a binary .deb without extracting
  debmany      - Select manpages of installed or uninstalled packages
  checkrestart - Help to find and restart processes which are using old
                 versions of upgraded files (such as libraries)
  popbugs      - Display a customized release-critical bug list based on
                 packages you use (using popularity-contest data)
and also moreutils, which is basically awesome pipe tools on wheels:
Description: additional unix utilities
 This is a growing collection of the unix tools that nobody thought
 to write thirty years ago.
 .
 So far, it includes the following utilities:
  - sponge: soak up standard input and write to a file
  - ifdata: get network interface info without parsing ifconfig output
  - ifne: run a program if the standard input is not empty
  - vidir: edit a directory in your text editor
  - vipe: insert a text editor into a pipe
  - ts: timestamp standard input
  - combine: combine the lines in two files using boolean operations
  - pee: tee standard input to pipes
  - zrun: automatically uncompress arguments to command
  - mispipe: pipe two commands, returning the exit status of the first
  - isutf8: check if a file or standard input is utf-8
  - lckdo: execute a program with a lock held
Homepage: http://kitenet.net/~joey/code/moreutils/
share

A great "feature" I use every day at work: The ability to have SSH listen on port 443 so I can create a tunnel which bypasses my work firewall, allowing me to run a local SOCKS proxy tunneled through SSH to my internet facing Linux server.

I can completely ignore my corporate firewall.

share
3  
It always make me chuckle to think of the people who believe that you can "lock down" a network. If leave a single port for traffic to get out on, you're screwed. – baudtack May 22 '09 at 23:29
    
what if they see 3GB of data flowing through port 443 from your computer? – Wadih M. Aug 23 '10 at 18:05
    
Add corkscrew to the nix, and even https proxies aren't an issue!! – Niall Donegan Jan 10 '11 at 2:15
    
@Wadih.M:this is the way:ssh through 443, send back a udp packet from remote machine to another port with netcat.send another udp packet from where you are to the remote machine.congrats.the port is punched. – Behrooz Apr 23 '11 at 12:32

I've always liked "man hier" for helping people that are new to the system.

share

Perhaps one of the great features of Linux, as opposed to Windows, is that there are next to no hidden features by design. Hopefully if any undocumented features are important enough to know about the community does its job.

share

Here are a few I've used on a regular basis:

  • sar - shows system activity
  • vmstat - virtual memory stats
  • iostat - io stats
  • pkill - like pgrep but allows you to kill the returned process ID
  • xargs -I<string> - allows for replacing strings with piped data
  • at - schedule a task
  • tkdiff - graphical diff utility

These are probably not "hidden", but I find them extremely useful:

  • df -hk - show disk usage in human readable format
  • ls -ltr - list files sorted by date
  • while :; do...done - (Bash) replacement for watch if unavailable
  • perl -e - run a Perl snippet on the command-line
  • free -kt - show memory information (kilobytes w/total)
share
    
I think it's easier to write while true; do ... than while [ 1 ]; do ... – ptman May 23 '10 at 17:30

Start with this article with some great Linux tricks. My favorite is pstree, which displays your processes in a tree format so you can see which process spawned which.

curl beats the hell out of wget for pretty much anything you would use wget for.

zgrep and zless are great for searching gzipped log files so you don't have to pipe them through gunzip or leave uncompressed stuff around in /var/log.

share
    
Synopsis, for the lazy ;-) 1.pgrep (like ps meets grep) 2.pstree (lists the processes in a tree format) 3.bc (an arbitrary precision calculator language) 4.split (splits large files into chunks) 5.nl (numbers lines) 6.mkfifo (make two commands communicate through a named pipe) 7.ldd (prints shared library dependencies) 8.col (strips out line feeds) 9.xmlwf (checks if an XML document is well formed) 10.lsof (lists open files) – username May 12 '09 at 21:13
    
(GNU) ps has flags for printing a tree. Is there anything other than being easier to remember going on for pstree? – ptman May 23 '10 at 17:36
    
Why would curl be better than wget? And: zgrep and zless are only the beginning; there is also zcat and zmore. – Mei May 18 '11 at 16:55

The concept is that everything in Linux is a file.

All configurations are in text files, and everything in Linux is treated as a file. This is a much simpler approach which makes it very easy to change things in Linux. In Linux even your filesystem itself can be viewed as a file.

share
    
If you like this basic Unix concept, you'll like Plan9 OS. – wazoox Jun 26 '09 at 9:46

ESC + .

'Pastes' the last parameter from the previous line into the current prompt

e.g.

ls -l /home/someuser/somedir/somefile

followed by

rm ESC + .

translates to

rm /home/someuser/somedir/somefile
share
    
That's awesome. Didn't know that. – Andrew Taylor Jan 19 '11 at 15:18

I see the yes command very nice:

yes | do_you_agree

From man page:

NAME
       yes - output a string repeatedly until killed

SYNOPSIS
       yes [STRING]...
       yes OPTION
share

It's open source. Nothing is "hidden" if you bother to look.

share
    
Unfortunately open source tends to make "hidden" things rather difficult for you to look. – Spoike May 2 '09 at 6:58
5  
I'd have argued the opposite - if a feature is hidden in a closed source app, it's up to the dev shop to disclose it. In an open source app, the feature is there for everyone to see. Furthermore, the number of people actively developing is generally much larger (more people know and talk about 'hidden' features). Lastly, open source development is largely community oriented, so people generally tell each other about hidden features. – Tom Wright May 2 '09 at 15:35
    
I'm not following @Spoike's reasoning at all. Yes, there is more to look at, but what you see is what you get. – Paul Tomblin May 2 '09 at 16:34
    
I think you're arguing over semantics. How about we rename to "Obfuscated features of linux?" – spoulson May 5 '09 at 18:51
    
I would partially argue in Spoike's favor that maybe things are 'hidden' because of the general lack of good up to date documentation. Not to say this statement holds true for all projects, however there are a great deal of projects where full documentation (and I don't mean just how-to user guides) are somewhat lacking. KDE's administrative features, especially in regards to management of settings inside rc files and actual effects of setting a specific field immutable is a perfect example. In many cases I've found myself looking in the source code to find out how things are handled. – Jason Mock May 30 '09 at 20:40

The hidden feature is that there are no hidden features. The system provides a tremendous amount of power that you have complete access to. You have to understand every piece of it and every tool available to know what power is at your fingertips.

You should start by understanding every command in /bin, /sbin, /usr/bin, /usr/sbin, and every virtual file in /proc. Read the manpages, other documentation, and source as necessary.

If you aren't comfortable reading a man page or reading the source, that is your hidden feature.

share

grep, awk and sed

top

share
12  
Those aren't very hidden – jayrdub May 1 '09 at 21:58
  • od - dump files in octal and other formats. Useful to check for example if there's some BOM crap in the beginning of file
  • file - what might be the filetype of given file?
  • lshw, lsusb, lspci - list hardware
  • tracepath - are we dealing with MTU problem here?
  • netwox contains 200+ network related tools
  • ip - for network config/info
  • sysctl - filesystem/network/kernel info
  • ebtables - iptables for bridge
  • vconfig - VLAN configuration
  • brctl - Bridge configuration
  • socat - netcat on steroids
  • ipgrab - tcpdump-like utility that prints detailed header information
  • dig - What's up with DNS server now?
share

tee is awesome. Output to screen and logfile? Check.

share

Just found this one today:

dmidecode:
Reports information about your system's hardware as described in your system BIOS according to the SMBIOS/DMI standard. This information typically includes system manufacturer, model name, serial number, BIOS version, asset tag as well as a lot of other details of varying level of interest and reliability depending on the manufacturer. This will often include usage status for the CPU sockets, expansion slots (e.g. AGP, PCI, ISA) and memory module slots, and the list of I/O ports (e.g. serial, parallel, USB).

It definitely makes answering questions about somerandomserver27 at the colo facility a lot easier!

share
    
You'll like lspci, lsscsi, lsusb too :) – wazoox Aug 19 '10 at 10:30
    
Yup, I use the ls* commands on a regular basis - I just didn't realize there was anything that could pull BIOS level info :) – gharper Aug 20 '10 at 18:43

Not really a Linux thing, more of a Bash thing: process substitution,

diff some_local_file <(ssh somehost "cat some_remote_file")

This diffs a local and a remote file retrieved via SSH in one line.

share

Not the answer you're looking for? Browse other questions tagged or ask your own question.