# Linux command line best practices and tips?

I would like to open a discussion that would accumulate your Linux command line (CLI) best practices and tips.

I've searched for such a discussion to share the below comment but haven't found one, hence this post.

I hope we all could learn from this.

You are welcome to share your Bash tips, grep, sed, AWK, /proc and all other related Linux/Unix system administration, shell programming best practices for the benefit of us all.

-

## migrated from stackoverflow.comSep 26 '09 at 7:14

This question came from our site for professional and enthusiast programmers.

## locked by Chris S♦Jan 27 '12 at 4:17

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

## Requirement

You need to test maximum bandwidth between 2 servers.

## Solution

On SERVER1 do :

nc -u -l 54321 > /dev/null


On SERVER2 do :

dd if=/dev/zero bs=1MB | nc -u SERVER1 54321 &
pid=$(pidof dd) while (( 1 )); do kill -USR1$pid; sleep 2; done


You will see output such as :

182682000000 bytes (183 GB) copied, 1555.74 seconds, 117 MB/s
182920+0 records in
182919+0 records out


117 MB/s is the interesting factor here, which shows the actual network transfer bandwidth.

Explanation:

As packet will start flowing over the network you will be able to see bandwidth statistics on SERVER2, which is a pretty good estimate of the actual maximum possible bandwidth between the 2 servers.

Copying by UDP (to prevent TCP overhead).

Copying to from memory (/dev/zero) on SERVER1 to memory (/dev/null) on SERVER2, thus preventing disk I/O from becoming the bottleneck.

-
Be sure that ssh is not configured to use compression in .ssh/config, 'though. Otherwise you'll get really nice numbers that are very far from the truth ;-) –  Joachim Sauer Mar 2 '09 at 20:46

I'm surprised no one mentioned bash's built-in fc command (fc stands for Fix Command).

Use it to edit your previous command in an editor (i.e. vim) instead of in the command line, and execute it upon quitting the editor. Pretty handy.

fc [-e ename] [-nlr] [first] [last]
fc -s [pat=rep] [command]


-

To copy part of a file system to a new hard disk one can use

mkfs.ext4 /dev/sdb
mkdir /mnt/newhd
mount /dev/sdb /mnt/newhd/
rsync -av --hard-links --acls --one-file-system --xattrs /home/maxim/ /mnt/newhd/
echo '/dev/sdb /home/maxim ext4 defaults,user_xattr,noatime 0 1' >> /etc/fstab

-

I dont these commands are in the above list...!!!

1. find . -name .svn -type d |xargs rm -rf

Remove all .svn folders

2. bash -x script.sh

print line and execute it in BASH

3. Ctrl + [

the same as [Esc] in vim

4. shopt -s autocd

Automaticly cd into directory

5. df -i

View the current number of free/used inodes in a file system

6. sudo !!

Run the last command as root

7. python -m SimpleHTTPServer

Serve current directory tree at http://$HOSTNAME:8000/ 8. netstat -tlnp Lists all listening ports together with the PID of the associated process 9. Below are some ways to number input.txt: cat -n$ cat -n input.txt 1 123 2 3 456 4 5 789 6 7 8 abc 9 10 def 11 12 ghi

-

I use constantly these ones

ALT-. (ESC+. in some terminals) copies last used argument (super-useful)

CTRL-W deletes word

CTRL-L clear terminal (like clear command but faster)

ALT-B (ESC+B in some terminals) move backward a word

ALT-F (ESC+F in some terminals) move forward a word

CTRL-R search in history

-
Thank you for sharing the shortcuts! I did not get ALT-B/F to work in Mac. –  Masi Mar 23 '09 at 18:07

Easy sums/averages/grouping with awk:

cat tests
ABC 50
DEF 70
XYZ 20
DEF 100
MNP 60
ABC 30

cat tests | awk '{sums[$1] +=$2; tot += $2; qty++}\ END { for (i in sums) printf("%s %s\n", i, sums[i]); printf("Total: %d\nAverage: %0.2f\n", tot, tot/qty)} ' MNP 60 ABC 80 XYZ 20 DEF 170 Total: 330 Average: 55.00  - To rename a multiple files in a similar fashion I found the following script very useful and robust over the years. It just puts the output of ls into your favorite text editor. You just modify the text, save, close. The files are renamed accordingly. It's especially great when you combine this with Vi column editing (Ctrl-v, select a block, I to insert before or A to insert after, type text, Esc). #!/usr/bin/ruby RM = '/bin/rm' MV = '/bin/mv' from = Dir.entries('.').sort; from.delete('.'); from.delete('..') from.sort! from.delete_if {|i| i =~ /^\./} # Hidden files tmp = "/tmp/renamer.#{Time.now.to_i}.#{(rand * 1000).to_i}" File.open(tmp, 'w') do |f| from.each {|i| f.puts i} end ENV['EDITOR'] = 'vi' if ENV['EDITOR'].nil? system("#{ENV['EDITOR']} #{tmp}") to = File.open(tmp) {|f| f.readlines.collect{|l| l.chomp}} #{RM} #{tmp} if to.size != from.size STDERR.puts "renamer: ERROR: number of lines changed" exit(1) end from.each_with_index do |f, i| puts #{MV} -v --interactive "#{f}" "#{to[i]}" unless f == to[i] end  I call this script renamer. - I found Git version control to be: • Snappy • A pleasure to use • Useful for a projects of almost any size (100K to 100GB; 1 to 100k files) Here is how I do it: # Create new repository # (for now, it will live in .git/ - a single directory) git init # Commit all I got so far git add . git commit # Add new or modified files manually git add *.c git status git commit # Add all modified files git status git commit -a # Redo last commit git commit -a --amend # View log git log # Reset everything (files and git history) back to # what it was at 96223554b3e3b787270b1f216c19ae38e6f83ca5 git branch this-was-a-mistake git reset --hard 9622 # Everything is back in time ls git log  - CTRL+] x to forward search for a character "x", and Meta, CTRL+] x for backward search. On most systems, Meta can be ESC or ALT. For ESC, you press ESC then release, then combine CTRL and ], and then press the character to search for. For ALT, press down CTRL + ALT + ] at the same time, then the target character. I find it's useful when editing history command. For very long and very complicated command. I use fc to open vi(probably actually vim on linux) to edit the command. - For long command editng, in bash at least, you can 'set -o vi' to enable vi-style keybindings directly in your shell. – pboin Oct 25 '10 at 14:49 Meta+. in bash for cycling through the last argument from previous commands. Great for running tail and grep in various combinations. - Sometimes while working in terminal you need to open a file using some GUI application associated with it like for pdf or mp3 files. You don't have to remember the exact name of that command, just use: gnome-open some-file.pdf  BTW, the shortest alias I use is: alias o=gnome-open  Very handy. - IMHO, *nix most important command ever is... man :) Almost everything one needs to know can be found with man and using man prevents us from interrupting our co-workers. And dealing with interruptions is one of our biggest concerns... - Bash parameter expansions are great: $ FOO=leading-trailing.txt

$echo${FOO#leading-} # remove leading match
trailing.txt

$echo${FOO%.txt} # remove trailing match

$echo${FOO/-*/-following} # match a glob and substitute


Need to rename a bunch of files? Combine with a for loop:

$for FILE in file*.txt; do mv -v$FILE ${FILE#file-}; done file-01.txt -> 01.txt file-02.txt -> 02.txt file-03.txt -> 03.txt  Once you have a basic grasp on them you'll be surprised how often they're useful. - • For smaller directory trees with documentation to browse find .  • To empty a file from shell > file.txt  • To return to my home directory cd  - ## Quick & Simple data integrity verification using nothing more then bash & md5sum This can prove to be priceless in terms of debugging trouble when moving binary files over the network... You should embrace this technique as common practice for each copy of valuable data to ensure 100% data integrity. # Setup some test data... mkdir -p /tmp/mdTest/dir{1,2,3} for i in seq 1 3; do echo$RANDOM > /tmp/mdTest/dir$i/file$i ; done


# md5 hash calculation on the test data

cd /tmp/mdTest/
TMPMD5LIST=$(mktemp); (find -type f -exec md5sum '{}' \;) >$TMPMD5LIST; mv $TMPMD5LIST list.md5sum  # data integraty verification from the hash cd /tmp/mdTest/ md5sum --check list.md5sum ./dir3/file3: OK ./dir1/file1: OK ./dir2/file2: OK  ### Unit test: Let's break one of the files. echo$RANDOM >> /tmp/mdTest/dir1/file1
md5sum --check list.md5sum
./dir3/file3: OK
./dir1/file1: FAILED
./dir2/file2: OK
md5sum: WARNING: 1 of 3 computed checksums did NOT match

-

Here are Collection of Linux, UNIX User Management Commands http://www.linuxconfig.net/2009/11/16/linux-unix-user-management-commands.html

-

I like to keep track of everything I do. One command that I learned in college was 'script'. This takes any output on your terminal and logs it to a file. What I didn't learn in college is how to make every terminal a script. Now I have this in my .login file:

exec script ~/.typescript/date +%Y%m%d%H%M%S.$$ Make sure that ~/.typescript/ exists before you add that to the end of your .login file. :) - The 'tee' command is really useful for when you are outputting to a file and want to see the progress at the same time. This is especially helpful for when you are logging output to a file and need to watch it as it progresses. Instead of doing something like: ./program > file & tail -f file  You can use the tee command on one line: ./program | tee file  - ALWAYS start any command or pipeline with # (comment) and remove it when you finish writing the command. Gives you a 2nd chance at spotting rm -rf / like things. - You can follow more than one file with tail -f: tail -f logfile1 logfile2  the updates will be intermingled depending on the order of occurrence: ==> logfile1 <== event-a event-b ==> logfile2 <== event-p event-q ==> logfile1 <== event-c  If you want a cleaner display, use watch and omit the -f from tail: watch tail logfile1 logfile2  - It is sometimes useful to leave a program running even after you have logged out. I've seen some solutions that use nohup or even screen for that purpose. The simplest I know of is:  your_command_here & disown  You can also detach a running program:  your_command_here # Press <Control-Z> to stop the program and bring it to background:  bg  disown  - Using screen is great because you can "reconnect" to that app from anywhere later. – prestomation Mar 2 '09 at 21:54 Seeing color directory structures is easier. alias ls="ls --color=tty"  Edit alias ls="ls --color=auto"  - I prefer --color=auto because it doesn't sent rubbish control chars if redirect to file you can also do alias grep="grep --color=auto" to have matching patterns highlighted. – kyku Mar 2 '09 at 19:51 When I want to make sure that I use actual command and not an alias, I use a leading backslash: \rm -rf ~/tmp  - Well, yes unless the "command" is not alias in which case you would have to use \command rm -rf ~/tmp in that case \rm -rf ~/tmp is better – presario Mar 3 '09 at 8:55 Use screen, a free terminal multiplexer developed by the GNU Project that will allow you to have several terminals in one. You can start a session and your terminals will be saved even when you connection is lost, so you can resume later or from home. - The most common switches I use are 'screen -D -R' for detaching and resuming my last screen session and 'screen -x' to view the same screen session from multiple logins. Also, get yourself a nice .screenrc from dotfiles.org/.screenrc – Nick Devereaux Mar 5 '09 at 1:41 Another nice thing about screen is that you can share a terminal with other people when you want to collaborate on something, using screen -x – gareth_bowles Mar 12 '09 at 18:40 The command line is a funny thing. I think that you can only learn so much on your own and the rest you learn by accident watching somebody else using a command line. I was using the shell for years painstakingly typing in directory names by hand. One day I was watching a friend mess around on a system and he kept hitting the tab key. I asked "why are you hitting tab?". Answer: it tries to complete the directory or filename. Who would have guessed--tab completion! Type a bit of the file or directory, hit tab, and it will try to finish what you typed (behavior depends on which shell though). One day, said friend was watching me on the command line and watched me type something like: coryking@cory ~/trunk/mozi  pushd /etc /etc ~/trunk/mozi coryking@cory /etc  popd ~/trunk/mozi coryking@cory ~/trunk/mozi   Who would have guessed!? He never knew about popd / pushd. Guess we are even... - To me tab completion seems like something "obvious", very basic, but I never knew about popd/pushd either. Funny indeed. :) – Jonik Mar 2 '09 at 20:46 Also see this answer to be able to tab-complete a lot more than commands, files and directories: stackoverflow.com/questions/603696//603919#603919 – Jonik Mar 2 '09 at 21:20 That's so true. I learned about bash's backward-history-search (ctrl-r) by watching someone. I later realized that it was a readline feature and worked the same in other programs that incorporate readline (mysql, gdb, python -i, clisp etc). – sigjuice Mar 23 '09 at 5:41 How could you use "the shell" for years and not know about tab completion? – prestomation Dec 18 '09 at 21:36 Old school, moving a directory tree from one place to another, preserving symlinks, permissions and all that good stuff: tar cBf - . | (cd /destination; tar xvBpf -)  or across the network tar cBf - . | rsh foo.com "cd /destination; tar xvBpf -)  New school: rsync -urltv . /destination  or across the network rsync -urltv -e ssh . foo.com:/destination  - -a is the same as '-rlptgoD' - that is recurse, copy symlinks as symlinks, preserve perms, preserve timestamps, preserve groups, preserve owners, copy device files and special files as device and special files. Also, between machines, -z (compress files before sending, decompress after receiving) is useful. rsync -avz [-e ssh] is your friend :-) – dr-jan Apr 20 '09 at 21:10 This tip will make your CLI more comfortable (at least it makes for me): create ~/.inputrc file with following contents: "\e[A": history-search-backward "\e[B": history-search-forward  Reload bash (eg by typing "exec bash"). When you type a prefix of a command and press the up arrow you will browse commands starting with your prefix, for example if you typed ssh it will show your former connections with remote shells. If your prompt is empty then the up arrow will browse the history the normal way. - Usually I bind those to \e[5~ and \e[6~ (aka pageup and pagedown) instead of rebinding the arrows, but it is more convenient than Ctrl-R/Ctrl-S :) – ephemient Mar 2 '09 at 21:06 +1 for exec bash – Dennis Williamson Sep 26 '09 at 15:01 My Favorite And Frequently Used: List contents of tar.gz file tar -tzf filename.tar.gz will match line containing S1 OR S2 OR S3 OR S4 grep 'S1.S2.S3.*S4' file Lists all subdirectories of current directory ls -d */ Total size of directory du -sh find a date: TIMESTAMP=date '+%y%m%d%H%M' move process from foreground to background Ctrl-z and then bg Entire word to uppercase echo "word" | awk '{print toupper(0)}' Checks equality between numbers x -eq y Check is x is equal to y x -ne y Check if x is not equal to y x -gt y Check if x is greater than y x -lt y Check if x is less than y Checks equality between strings x = y Check if x is the same as y x != y Check if x is not the same as y -n x Evaluates to true if x is not null -z x Evaluates to true if x is null Command Line Parameters for ' test ' -d check if the file is a directory -e check if the file exists -f check if the file is a regular file -g check if the file has SGID permissions -r check if the file is readable -s check if the file's size is not 0 -u check if the file has SUID permissions -w check if the file is writetable -x check if the file is executable print first field of the last line" awk '{ field = 1 }; END{ print field }' important build in variables # Number of command line arguments. Useful to test no. of command line args in shell script. * All arguments to shell @ Same as above - Option supplied to shell$$ PID of shell

\$! PID of last started background process (started with &)

-

Some small pointers regarding log viewing:

1. You can use tail -F to keep watching the log after it is truncated (by example log4j).
2. You can use less: Open less, click SHIFT+F to emulate tail behaviour. Also useful combination is -S to disable line wrapping. less will enable you to search in the logs.
-
And with tail -f you can watch multiple logs at once! E.g.: tail -f log1.txt -f log2.txt –  Stu Thompson Feb 16 '12 at 12:12

I'm moving towards never typing "rm" at a command prompt. Instead, I type "ls", and if I like the list of removed files I edit the command (easily possible with bash and ksh).

Edit to add something from the comments: "rm -i" will prompt for each deletion, which accomplishes the same purpose. Thanks!

-