# Linux command line best practices and tips?

I would like to open a discussion that would accumulate your Linux command line (CLI) best practices and tips.

I've searched for such a discussion to share the below comment but haven't found one, hence this post.

I hope we all could learn from this.

You are welcome to share your Bash tips, grep, sed, AWK, /proc and all other related Linux/Unix system administration, shell programming best practices for the benefit of us all.

-

## migrated from stackoverflow.comSep 26 '09 at 7:14

This question came from our site for professional and enthusiast programmers.

## locked by Chris S♦Jan 27 '12 at 4:17

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

Use screen, a free terminal multiplexer developed by the GNU Project that will allow you to have several terminals in one.

You can start a session and your terminals will be saved even when you connection is lost, so you can resume later or from home.

-
The most common switches I use are 'screen -D -R' for detaching and resuming my last screen session and 'screen -x' to view the same screen session from multiple logins. Also, get yourself a nice .screenrc from dotfiles.org/.screenrc –  Nick Devereaux Mar 5 '09 at 1:41
Another nice thing about screen is that you can share a terminal with other people when you want to collaborate on something, using screen -x –  gareth_bowles Mar 12 '09 at 18:40

SSH!
SSH is the god command--I think it's the most valuable over-all command to learn. The options can be pretty daunting, but it seems like I'm constantly learning to use new command-line options for SSH that I never thought would be necessary. I may have used them all at this point.

The more you use it, the more you learn about it. You can use it to do some AMAZING things.

Note: ALL these things are doable remotely with no setup on your server except to have ssh server running.

Mount a file system over the internet

search the net for SSHFS

Forward commands.

The SVN+SSH protocol is Subversion from a remote client to a server with NO DEAMON running on it! The SVN command starts the server through the ssh shell and passes the info back and forth through the existing pipe. The rsync program does the same thing, runs against a server with no rsync deamon by starting one itself via SSH. It's easy to write your own bash files to do similar tricks.

Chain to get through firewalls

I use this all the time to jump through my linux server at home to my mac.

Forward ports:
Seems only moderately useful until you realize that you can bounce through your home firewall and configure your router at home from work as though you were doing so from within your home network).

Forward X requests:

This is another amazing one. With or without an X server running on your remote system, you can run an x-windows program and the window will appear on your local screen. Just use the switch -X, that's all!

Since you don't have to have an X server running on your remote server, the CPU impact on your server is minimal, you can have a TINY Linux server that serves up huge apps to your powerful game PC running Windows and cygwin/X.

Of course VI and EMACS work over SSH, but when I'm running at home, sometimes I want more. I use ssh -X to start up a copy of Eclipse! If your server is more powerful than your laptop, you've got the GUI sitting right there on your laptop, but compiles are done on your server, so don't worry about system load.

Run in batch files

(meaning run a local batch file that "does stuff" on other systems):

Two things combine to make this one cool. One is you can eliminate password prompts by using (more secure) encryption keys. The second is you can specify a command on the SSH CLI. I've used this in some interesting ways--Like when a compile fails on the remote server, I would have it SSH into my computer and play a sound file).

Remember you can redirect the output from the remote command and use it within your local batch file, so you could also be locally tailing a compile running on your server.

Built in to Mac

Both server and client are built into both mac and linux. In the case of the Mac and Ubuntu, enabling a server is as simple as finding the right checkbox.

On a PC install cygwin or cygwin/X (cygwin/X will allow you to forward your x-window output from your Linux machine to your Windows PC--it installs an X server)

Important Tips/config file

Never use port 22 on your firewall. You'll get a lot of hack attempts, it's just not worth it. Just have your firewall forward a different port to your server.

There are extensive configuration options that allow you to simplify your ssh commands significantly. Here's an example of mine at work:

Host home
hostname billshome.hopto.org
Port=12345
user=bill
LocalForward=localhost:1025 mac:22


When I type "ssh home" (nothing else), it acts as though I had typed:

ssh -p 12345 bill@billshome.hopto.org


and then forwards my local port 1025 to my system "mac" at home. The reason for this is that I have another entry in my file:

Host mac
hostname localhost
port=1025


so that once I've done an "ssh home" and still have the window open, I can type "ssh mac" and the computer at work here will actually try to connect to its own port 1025 which has been forwarded to "mac:22" by the other command, so it will connect to my Mac at home through the firewall.

Edit--cool script!

I dug up an old script I just love--had to come back and post it for anyone here who might be interested. The script is called "authMe"

#!/bin/bash
if [ ! -f ~/.ssh/id_dsa.pub ]
then
echo 'id_dsa.pub does not exist, creating'
ssh-keygen -tdsa
fi
ssh $1 'cat >>.ssh/authorized_keys' <~/.ssh/id_dsa.pub  If you have this script in your home directory and there is a host you can connect to (via ssh), then you can type "./authMe hostName". If necessary it will create a public/private keypair for you, then it will ssh over to the other machine and copy your public key over (the ssh command will prompt you for a password...) After this, the SSH command should not ask for your password any more when attaching to that remote system, it will use the public/private keypair. If your remote computer is not always secure, you should consider setting a "passphrase" when prompted. You may also want to configure the ssh server on the far end to not allow text passwords (only keys) for additional security. - +1 very nice and compact post – Karsten Mar 3 '09 at 12:20 A small note on SSH over port 22: provided you've got a good root password and your users don't have terrible passwords, just running fail2ban (or something similar) is enough to keep out crack attempts. I found it really annoying to keep SSH on another port. – David Wolever Sep 13 '09 at 16:04 It can be irritating at times, but the volume of connects was just scary. I know that they aren't supposed to get through, but it's kind of like if you had someone walking up to your house and fiddling with your doorknob every minute--just unnerving. – Bill K Sep 14 '09 at 15:49 show 4 more comments I like to use cd -  to switch to the previous directory. Very useful! - I never knew this! Thanks! On a related note, pushd and popd do something similar, by building up a stack of dirs. These sort of tricks help when you are navigating around a deep dir structure. – KarstenF Mar 3 '09 at 19:40 show 1 more comment I've recently discovered the pv command (pipe viewer) which is like cat but with transfer details. So instead of $ gzip -c access.log > access.log.gz


You can use

$pv access.log | gzip > access.log.gz 611MB 0:00:11 [58.3MB/s] [=> ] 15% ETA 0:00:59  So instead of having no idea when your operation will finish, now you'll know! - show 4 more comments sudo !!  Rerun the previous command as root. [The current top command on the site http://www.commandlinefu.com, a site themed along the lines of this question.] - Disclosure - I run commandlinefu.com – codeinthehole Mar 2 '09 at 21:52 It would have saved this guy 16 characters: xkcd.com/149 – Bill K Mar 3 '09 at 19:06 add comment Press Ctrl-R and start typing a command (or any part of it) - it searches the command history. Hitting Ctrl-R again will jump to the next match, enter executes the currently displayed command, and right-arrow (at least) will let you edit it first. $ (reverse-i-search)svn': svn status


I had used Linux for something like 7 years as my main OS before learning of this, but now that I know it, it's quite handy.

-

The command line is a funny thing. I think that you can only learn so much on your own and the rest you learn by accident watching somebody else using a command line.

I was using the shell for years painstakingly typing in directory names by hand. One day I was watching a friend mess around on a system and he kept hitting the tab key. I asked "why are you hitting tab?". Answer: it tries to complete the directory or filename. Who would have guessed--tab completion! Type a bit of the file or directory, hit tab, and it will try to finish what you typed (behavior depends on which shell though).

One day, said friend was watching me on the command line and watched me type something like:

coryking@cory ~/trunk/mozi $pushd /etc /etc ~/trunk/mozi coryking@cory /etc$ popd
~/trunk/mozi
coryking@cory ~/trunk/mozi $ Who would have guessed!? He never knew about popd / pushd. Guess we are even... - To me tab completion seems like something "obvious", very basic, but I never knew about popd/pushd either. Funny indeed. :) – Jonik Mar 2 '09 at 20:46 Also see this answer to be able to tab-complete a lot more than commands, files and directories: stackoverflow.com/questions/603696//603919#603919 – Jonik Mar 2 '09 at 21:20 That's so true. I learned about bash's backward-history-search (ctrl-r) by watching someone. I later realized that it was a readline feature and worked the same in other programs that incorporate readline (mysql, gdb, python -i, clisp etc). – sigjuice Mar 23 '09 at 5:41 How could you use "the shell" for years and not know about tab completion? – prestomation Dec 18 '09 at 21:36 show 1 more comment Learn Vim. It is (arguably) the best editor, but most certainly the best editor available on a bare Linux server. - gvim is also best file browser on windows ;) – Piotr Findeisen Jun 16 '09 at 19:12 show 1 more comment It is sometimes useful to leave a program running even after you have logged out. I've seen some solutions that use nohup or even screen for that purpose. The simplest I know of is: $ your_command_here & disown


You can also detach a running program:

$your_command_here # Press <Control-Z> to stop the program and bring it to background:$ bg
$disown  - Using screen is great because you can "reconnect" to that app from anywhere later. – prestomation Mar 2 '09 at 21:54 show 2 more comments When I want to make sure that I use actual command and not an alias, I use a leading backslash: \rm -rf ~/tmp  - Well, yes unless the "command" is not alias in which case you would have to use \command rm -rf ~/tmp in that case \rm -rf ~/tmp is better – presario Mar 3 '09 at 8:55 show 4 more comments This tip will make your CLI more comfortable (at least it makes for me): create ~/.inputrc file with following contents: "\e[A": history-search-backward "\e[B": history-search-forward  Reload bash (eg by typing "exec bash"). When you type a prefix of a command and press the up arrow you will browse commands starting with your prefix, for example if you typed ssh it will show your former connections with remote shells. If your prompt is empty then the up arrow will browse the history the normal way. - Usually I bind those to \e[5~ and \e[6~ (aka pageup and pagedown) instead of rebinding the arrows, but it is more convenient than Ctrl-R/Ctrl-S :) – ephemient Mar 2 '09 at 21:06 +1 for exec bash – Dennis Williamson Sep 26 '09 at 15:01 show 3 more comments Use && instead of ; when executing multiple commands at once. It stops when an error occurs and does not execute the other commands. Classical example: ./configure && make && make install  - This practice is so underrated! It should be standard practice to end each line of a shell script with && and use "true" as the final command. – user18911 Mar 5 '09 at 2:57 what's funny is that I didn't know about the ';' command before--I had ALWAYS used '&&'. – Nick Klauer Mar 15 '09 at 2:45 You can start shell scripts with "#!/bin/sh -e". This stops if any line fails. – stribika Aug 8 '09 at 11:32 You can use a double pipe for the "else" clause: dosomething && echo "success" || echo "teh fail" – Dennis Williamson Sep 26 '09 at 15:04 There was a recent note on Planet Debian that it's bad practice to use the -e on the shebang line. Instead use set -e in the body of the script. This way the script doesn't work differently if someone happens to run in using sh$filename. –  ptman Nov 24 '09 at 11:00

When writing loops on the command line in bash I often prefix risky commands with the 'echo' command.

for item in items; do echo something-risky; done


This way I get to see the 'something-risky' in full before committing to run it. Helps when your risky command includes variable expansions and globs. And 'set -x' is very useful when writing bash scripts.

'set -x' enables debugging. See http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_02_03.html for information on bash debugging.

-
I do this too, very handy trick! –  user18911 Mar 5 '09 at 2:48
+1 This can be a life saver! or data saver... –  David Oneill Nov 30 '10 at 2:29

## Requirement

You need to test maximum bandwidth between 2 servers.

## Solution

On SERVER1 do :

nc -u -l 54321 > /dev/null


On SERVER2 do :

dd if=/dev/zero bs=1MB | nc -u SERVER1 54321 &
pid=$(pidof dd) while (( 1 )); do kill -USR1$pid; sleep 2; done


You will see output such as :

182682000000 bytes (183 GB) copied, 1555.74 seconds, 117 MB/s
182920+0 records in
182919+0 records out


117 MB/s is the interesting factor here, which shows the actual network transfer bandwidth.

Explanation:

As packet will start flowing over the network you will be able to see bandwidth statistics on SERVER2, which is a pretty good estimate of the actual maximum possible bandwidth between the 2 servers.

Copying by UDP (to prevent TCP overhead).

Copying to from memory (/dev/zero) on SERVER1 to memory (/dev/null) on SERVER2, thus preventing disk I/O from becoming the bottleneck.

-
Be sure that ssh is not configured to use compression in .ssh/config, 'though. Otherwise you'll get really nice numbers that are very far from the truth ;-) –  Joachim Sauer Mar 2 '09 at 20:46

Seeing color directory structures is easier.

alias ls="ls --color=tty"


Edit

alias ls="ls --color=auto"

-
I prefer --color=auto because it doesn't sent rubbish control chars if redirect to file you can also do alias grep="grep --color=auto" to have matching patterns highlighted. –  kyku Mar 2 '09 at 19:51

Some small pointers regarding log viewing:

1. You can use tail -F to keep watching the log after it is truncated (by example log4j).
2. You can use less: Open less, click SHIFT+F to emulate tail behaviour. Also useful combination is -S to disable line wrapping. less will enable you to search in the logs.
-
And with tail -f you can watch multiple logs at once! E.g.: tail -f log1.txt -f log2.txt –  Stu Thompson Feb 16 '12 at 12:12
show 1 more comment

I learned a trick from a friend a long while back to easily change a file's extension:

mv my_filename.{old,new}


Shell expansion will expand this to:

mv my_filename.old my_filename.new


This can also be used to rename a file in other ways, such as adding something in:

mv my_{,cool_}filename

-
Just be sure not to put any wildcards or other pattern expressions in the same command, or the glob could expand to more than just 2 arguments! mv *.{old,new} is likely to produce really surprising results. –  mtnygard Sep 13 '09 at 16:24
You can also type, for example, ls -l /bin/c then press Alt-Shift-{ and it will complete this to something like ls -l /bin/c{at,h{grp,mod,own,vt},p{,io},sh} for you. –  Dennis Williamson Sep 26 '09 at 16:05
show 1 more comment

Use "tar xf" to extract compressed archives. The j for bzip2 and z for gzip are not necessary, as tar will detect file type of your archive. Neither is '-' sign before arguments. You will save a lot of time (over a millenium ;-)).

Better yet use unfoo to extract any archive with a single command without any unnecessary arguments.

-

Install bash-completion package. It contains a number of predefined completion rules for shell. Enable it by typing "source /etc/bash_completion" if you distro doesn't do it for you. Then, for example whenever you complete after kpdf you will only see a list of directories and PDF files. It is as smart as to complete remote files after scp ssh://server/XXX (if you enabled passwordless logins).

-

I use constantly these ones

ALT-. (ESC+. in some terminals) copies last used argument (super-useful)

CTRL-W deletes word

CTRL-L clear terminal (like clear command but faster)

ALT-B (ESC+B in some terminals) move backward a word

ALT-F (ESC+F in some terminals) move forward a word

CTRL-R search in history

-
Thank you for sharing the shortcuts! I did not get ALT-B/F to work in Mac. –  Masi Mar 23 '09 at 18:07
show 1 more comment

In bash, I use !$a lot. It repeats the last argument of the last command: ls /really/long/command/argument/that/you/dont/want/to/repeat.txt vi !$


It will run the ls, then if you want to edit it, you do not have to retype it, just use !$. This is really useful for long path/file names. Also, !* repeats all the previous command's arguments. I don't use that as much, but it looks useful. I known they've been mentioned, but I use vim, screen, and cd - a lot. I forgot noclobber: set -o noclobber  From man bash: If set, bash does not overwrite an existing file with the >, >&, and <> redirection operators. This may be overridden when creating output files by using the redirection operator >| instead of >. - I don't use$!, as it's really easy to fat finger, but I like hitting ^[-. (escape-period) to get the same effect. –  David Wolever Sep 13 '09 at 16:19
Depending on your keyboard, etc., you can do Alt-. as well. –  Dennis Williamson Sep 26 '09 at 15:26
show 1 more comment

Switch from bash to zsh, and see your productivity improve:

• Really intelligent, scriptable tab completion. It completes not just command lines but all the options, names of man pages, package names (for apt-get / emerge etc), and what have you. And provides a helpful explanation of the options during the completion. All this without using any scrollback space after the command has been issued.
• Tab completion of wildcards; write cat *.txt, press tab, and choose between any files that match the expression.
• Switch directories just by typing their names.
• Several line editor modes. Can behave like vi or emacs if you want it to.
• Customizable key bindings to do whatever you wish.
• Themeable prompts, including the ability to put prompt information on the right side of the screen and have it auto-hide when typing a long command

Google will tell you many more benefits.

-
you should give more then a link. sounds interesting, but sell it to me man! –  Cory R. King Mar 2 '09 at 20:47

Old school, moving a directory tree from one place to another, preserving symlinks, permissions and all that good stuff:

tar cBf - . | (cd /destination; tar xvBpf -)


or across the network

tar cBf - . | rsh foo.com "cd /destination; tar xvBpf -)


New school:

rsync -urltv . /destination


or across the network

rsync -urltv -e ssh . foo.com:/destination

-
-a is the same as '-rlptgoD' - that is recurse, copy symlinks as symlinks, preserve perms, preserve timestamps, preserve groups, preserve owners, copy device files and special files as device and special files. Also, between machines, -z (compress files before sending, decompress after receiving) is useful. rsync -avz [-e ssh] is your friend :-) –  dr-jan Apr 20 '09 at 21:10

Requirement: You have a directory containing large list of files you would like to delete. rm -r will fail!

example

find /var/spool/mqueue/ | wc -l
191545
rm -f /var/spool/mqueue/*
-bash: /bin/rm: Argument list too long


Solution:

find /var/spool/mqueue/ -xdev -exec command rm -f '{}' +


Explanation:

Edit: Fixing explanation following @ephemient comment.

find will supply arguments to rm by the maximum allowed arguments. This will allows rm to delete files in batches which is the fastest technique that I know of without using the -delete operation of find itself. It's equivalent to

find /var/spool/mqueue -xdev -print0 | xargs -0 rm -f


which you may find useful if your find does not support -exec ... +.

-
Replace \; with +: then find will do what xargs does by default, which is batch up arguments as much as possible (without running over the argument list limit). Or just use -delete intead of -exec rm (GNU Findutils needed for both) –  ephemient Mar 2 '09 at 21:09
Interesting, I hadn't heard of '+'. –  user18911 Mar 5 '09 at 3:01

I use this two tips so often that i thought it would be a good idea to share :

!foo


Will launch the last command in the history file beginning with "foo" (I often use it when compiling, !gcc for example.)

The other one is a keyboard shortcut (Ctrl+O instead of Return) this will execute the command AND display the next command in the history file. For example when I compile and test a file i always do 3 or 4 commands, make, cd to the dir of the test, run the test, cd to the dir of the makefile. Using Ctrl+O this makes this task a lot easier :)

Hope this help!

-
+1 for Ctrl-O –  Dennis Williamson Sep 26 '09 at 16:09

How to use subversion in Linux without the help of fancy graphical interfaces that may not be available.

svn co <repo-url> . # Checks out project into current folder

svn update # Get latest changes from server

svn status # Shows what files have changed locally

svn commit # Commit your changes to the repo


This holds back a lot of developers from using Linux, strangely enough.

-

Most overlooked old-school command: find

Yesterday I repaired a gross permissions bug:

for u in nr dias simonpj; do
sudo -u $u find . -type d -exec chmod g+s '{}' ';' done  - Don't forget the -perm option to find to locate files that have permissions set a certain way. – Dennis Williamson Sep 26 '09 at 16:07 show 4 more comments For shells like bash and ksh, you can make the command line respond to navigation commands like your text editor: set -o emacs  or set -o vi  will allow you to search your history and move about the command line in the way that you are used to doing in text files (e.g. in vi mode, pressing ESC then typing /string will search your previous commands for "string" - you can use n and N to move between matches) - show 1 more comment grep is my friend. Seriously. List .rb files that contain the text class foo: grep -l "class foo" .rb  List .rb files that don't contain the text class foo: grep -L "class foo" *.rb  List .rb files that don't contain foo or bar (you can use any regexp with -e, but you need to escape the operators): grep -L -e "foo\|bar" *.rb  - show 1 more comment Shell substitution is performed with ^: /home/eugene$ ls foo.txt
foo.txt
/home/eugene $^ls^rm rm foo.txt /home/eugene$ ls foo.txt
ls: cannot access foo.txt: No such file or directory
`
-
Actually I prefer "!!:gs/foo/bar" since I often find that foo is mentioned more than once in the previous command and I want a global replacement (e.g. "mv foo.txt foo.bak"). –  user18911 Mar 5 '09 at 3:07