Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I would like to open a discussion that would accumulate your Linux command line (CLI) best practices and tips.

I've searched for such a discussion to share the below comment but haven't found one, hence this post.

I hope we all could learn from this.

You are welcome to share your Bash tips, grep, sed, AWK, /proc and all other related Linux/Unix system administration, shell programming best practices for the benefit of us all.

share

migrated from stackoverflow.com Sep 26 '09 at 7:14

This question came from our site for professional and enthusiast programmers.

locked by Chris S Jan 27 '12 at 4:17

This question exists because it has historical significance, but it is not considered a good, on-topic question for this site, so please do not use it as evidence that you can ask similar questions here. This question and its answers are frozen and cannot be changed. More info: help center.

show 9 more comments

85 Answers

Requirement

You need to test maximum bandwidth between 2 servers.


Solution

On SERVER1 do :

nc -u -l 54321 > /dev/null

On SERVER2 do :

dd if=/dev/zero bs=1MB | nc -u SERVER1 54321 &
pid=$(pidof dd)
while (( 1 )); do kill -USR1 $pid; sleep 2; done

You will see output such as :

182682000000 bytes (183 GB) copied, 1555.74 seconds, 117 MB/s
182920+0 records in
182919+0 records out

117 MB/s is the interesting factor here, which shows the actual network transfer bandwidth.

Explanation:

As packet will start flowing over the network you will be able to see bandwidth statistics on SERVER2, which is a pretty good estimate of the actual maximum possible bandwidth between the 2 servers.

Copying by UDP (to prevent TCP overhead).

Copying to from memory (/dev/zero) on SERVER1 to memory (/dev/null) on SERVER2, thus preventing disk I/O from becoming the bottleneck.

share
4  
Be sure that ssh is not configured to use compression in .ssh/config, 'though. Otherwise you'll get really nice numbers that are very far from the truth ;-) –  Joachim Sauer Mar 2 '09 at 20:46
show 4 more comments

I'm surprised no one mentioned bash's built-in fc command (fc stands for Fix Command).

Use it to edit your previous command in an editor (i.e. vim) instead of in the command line, and execute it upon quitting the editor. Pretty handy.

fc [-e ename] [-nlr] [first] [last]
fc -s [pat=rep] [command]

Read more about it here

share
add comment

To copy part of a file system to a new hard disk one can use

mkfs.ext4 /dev/sdb
mkdir /mnt/newhd
mount /dev/sdb /mnt/newhd/
rsync -av --hard-links --acls --one-file-system --xattrs /home/maxim/ /mnt/newhd/
echo '/dev/sdb /home/maxim ext4 defaults,user_xattr,noatime 0 1' >> /etc/fstab
share
add comment

I dont these commands are in the above list...!!!

  1. find . -name .svn -type d |xargs rm -rf

    Remove all .svn folders

  2. bash -x script.sh

    print line and execute it in BASH

  3. Ctrl + [

    the same as [Esc] in vim

  4. shopt -s autocd

    Automaticly cd into directory

  5. df -i

    View the current number of free/used inodes in a file system

  6. sudo !!

    Run the last command as root

  7. python -m SimpleHTTPServer

    Serve current directory tree at http://$HOSTNAME:8000/

  8. netstat -tlnp

    Lists all listening ports together with the PID of the associated process

  9. Below are some ways to number input.txt:

    cat -n

    $ cat -n input.txt 1 123 2 3 456 4 5 789 6 7 8 abc 9 10 def 11 12 ghi

share
add comment

I use constantly these ones

ALT-. (ESC+. in some terminals) copies last used argument (super-useful)

CTRL-W deletes word

CTRL-L clear terminal (like clear command but faster)

ALT-B (ESC+B in some terminals) move backward a word

ALT-F (ESC+F in some terminals) move forward a word

CTRL-E jump to EOL

CTRL-A jump to BOL

CTRL-R search in history

share
1  
Thank you for sharing the shortcuts! I did not get ALT-B/F to work in Mac. –  Masi Mar 23 '09 at 18:07
show 1 more comment

Easy sums/averages/grouping with awk:

cat tests
ABC 50
DEF 70
XYZ 20
DEF 100
MNP 60
ABC 30

cat tests | awk '{sums[$1] += $2; tot += $2; qty++}\
   END { for (i in sums) 
     printf("%s %s\n", i, sums[i]); 
     printf("Total: %d\nAverage: %0.2f\n", tot, tot/qty)} ' 
MNP 60
ABC 80
XYZ 20
DEF 170
Total: 330
Average: 55.00
share
add comment

To rename a multiple files in a similar fashion I found the following script very useful and robust over the years.

It just puts the output of ls into your favorite text editor. You just modify the text, save, close. The files are renamed accordingly.

It's especially great when you combine this with Vi column editing (Ctrl-v, select a block, I to insert before or A to insert after, type text, Esc).

#!/usr/bin/ruby

RM = '/bin/rm'
MV = '/bin/mv'

from = Dir.entries('.').sort; from.delete('.'); from.delete('..')
from.sort!

from.delete_if {|i| i =~ /^\./} # Hidden files

tmp = "/tmp/renamer.#{Time.now.to_i}.#{(rand * 1000).to_i}"

File.open(tmp, 'w') do |f|
  from.each {|i| f.puts i}
end

ENV['EDITOR'] = 'vi' if ENV['EDITOR'].nil?
system("#{ENV['EDITOR']} #{tmp}")

to = File.open(tmp) {|f| f.readlines.collect{|l| l.chomp}}
`#{RM} #{tmp}`

if to.size != from.size
  STDERR.puts "renamer: ERROR: number of lines changed"
  exit(1)
end

from.each_with_index do |f, i|
  puts `#{MV} -v --interactive "#{f}" "#{to[i]}"` unless f == to[i]
end

I call this script renamer.

share
add comment

I found Git version control to be:

  • Snappy
  • A pleasure to use
  • Useful for a projects of almost any size (100K to 100GB; 1 to 100k files)

Here is how I do it:

# Create new repository
# (for now, it will live in .git/ - a single directory)
git init

# Commit all I got so far
git add .
git commit

# Add new or modified files manually
git add *.c
git status
git commit

# Add all modified files
git status
git commit -a

# Redo last commit
git commit -a --amend

# View log
git log

# Reset everything (files and git history) back to 
# what it was at 96223554b3e3b787270b1f216c19ae38e6f83ca5
git branch this-was-a-mistake
git reset --hard 9622

# Everything is back in time
ls
git log
share
add comment

CTRL+] x to forward search for a character "x", and Meta, CTRL+] x for backward search. On most systems, Meta can be ESC or ALT. For ESC, you press ESC then release, then combine CTRL and ], and then press the character to search for. For ALT, press down CTRL + ALT + ] at the same time, then the target character.

I find it's useful when editing history command.

For very long and very complicated command. I use fc to open vi(probably actually vim on linux) to edit the command.

share
1  
For long command editng, in bash at least, you can 'set -o vi' to enable vi-style keybindings directly in your shell. –  pboin Oct 25 '10 at 14:49
add comment

Meta+. in bash for cycling through the last argument from previous commands. Great for running tail and grep in various combinations.

share
show 1 more comment

Sometimes while working in terminal you need to open a file using some GUI application associated with it like for pdf or mp3 files. You don't have to remember the exact name of that command, just use:

gnome-open some-file.pdf

BTW, the shortest alias I use is:

alias o=gnome-open

Very handy.

share
add comment

IMHO, *nix most important command ever is... man :)

Almost everything one needs to know can be found with man and using man prevents us from interrupting our co-workers. And dealing with interruptions is one of our biggest concerns...

share
add comment

Bash parameter expansions are great:

$ FOO=leading-trailing.txt

$ echo ${FOO#leading-} # remove leading match
trailing.txt

$ echo ${FOO%.txt} # remove trailing match
leading-trailing

$ echo ${FOO/-*/-following} # match a glob and substitute
leading-following.txt

Need to rename a bunch of files? Combine with a for loop:

$ for FILE in file*.txt; do mv -v $FILE ${FILE#file-}; done
file-01.txt -> 01.txt
file-02.txt -> 02.txt
file-03.txt -> 03.txt

Once you have a basic grasp on them you'll be surprised how often they're useful.

share
add comment
  • For smaller directory trees with documentation to browse

    find .
    
  • To empty a file from shell

    > file.txt
    
  • To return to my home directory

    cd
    
share
add comment

Quick & Simple data integrity verification

using nothing more then bash & md5sum

This can prove to be priceless in terms of debugging trouble when moving binary files over the network... You should embrace this technique as common practice for each copy of valuable data to ensure 100% data integrity.

Setup some test data...

mkdir -p /tmp/mdTest/dir{1,2,3}
for i in `seq 1 3`; do echo $RANDOM > /tmp/mdTest/dir$i/file$i ; done

md5 hash calculation on the test data

cd /tmp/mdTest/
TMPMD5LIST=$(mktemp); (find  -type f -exec md5sum '{}' \;) > $TMPMD5LIST; mv $TMPMD5LIST list.md5sum

data integraty verification from the hash

cd /tmp/mdTest/
md5sum --check list.md5sum
./dir3/file3: OK
./dir1/file1: OK
./dir2/file2: OK

Unit test: Let's break one of the files.

echo $RANDOM >> /tmp/mdTest/dir1/file1
md5sum --check list.md5sum
./dir3/file3: OK
./dir1/file1: FAILED
./dir2/file2: OK
md5sum: WARNING: 1 of 3 computed checksums did NOT match
share
add comment

I like to keep track of everything I do. One command that I learned in college was 'script'. This takes any output on your terminal and logs it to a file. What I didn't learn in college is how to make every terminal a script. Now I have this in my .login file:

exec script ~/.typescript/`date +%Y%m%d%H%M%S`.$$

Make sure that ~/.typescript/ exists before you add that to the end of your .login file. :)

share
add comment

The 'tee' command is really useful for when you are outputting to a file and want to see the progress at the same time. This is especially helpful for when you are logging output to a file and need to watch it as it progresses.

Instead of doing something like:

./program > file &
tail -f file

You can use the tee command on one line:

./program | tee file
share
add comment

ALWAYS start any command or pipeline with # (comment) and remove it when you finish writing the command. Gives you a 2nd chance at spotting rm -rf / like things.

share
add comment

You can follow more than one file with tail -f:

tail -f logfile1 logfile2

the updates will be intermingled depending on the order of occurrence:

==> logfile1 <==
event-a
event-b
==> logfile2 <==
event-p
event-q
==> logfile1 <==
event-c

If you want a cleaner display, use watch and omit the -f from tail:

watch tail logfile1 logfile2
share
show 2 more comments

It is sometimes useful to leave a program running even after you have logged out. I've seen some solutions that use nohup or even screen for that purpose. The simplest I know of is:

$ your_command_here & disown

You can also detach a running program:

$ your_command_here
# Press <Control-Z> to stop the program and bring it to background:
$ bg
$ disown
share
6  
Using screen is great because you can "reconnect" to that app from anywhere later. –  prestomation Mar 2 '09 at 21:54
show 2 more comments

Seeing color directory structures is easier.

alias ls="ls --color=tty"

Edit

alias ls="ls --color=auto"
share
5  
I prefer --color=auto because it doesn't sent rubbish control chars if redirect to file you can also do alias grep="grep --color=auto" to have matching patterns highlighted. –  kyku Mar 2 '09 at 19:51
show 7 more comments

When I want to make sure that I use actual command and not an alias, I use a leading backslash:

\rm -rf ~/tmp
share
4  
Well, yes unless the "command" is not alias in which case you would have to use \command rm -rf ~/tmp in that case \rm -rf ~/tmp is better –  presario Mar 3 '09 at 8:55
show 4 more comments

Use screen, a free terminal multiplexer developed by the GNU Project that will allow you to have several terminals in one.

You can start a session and your terminals will be saved even when you connection is lost, so you can resume later or from home.

share
2  
The most common switches I use are 'screen -D -R' for detaching and resuming my last screen session and 'screen -x' to view the same screen session from multiple logins. Also, get yourself a nice .screenrc from dotfiles.org/.screenrc –  Nick Devereaux Mar 5 '09 at 1:41
4  
Another nice thing about screen is that you can share a terminal with other people when you want to collaborate on something, using screen -x –  gareth_bowles Mar 12 '09 at 18:40
add comment

The command line is a funny thing. I think that you can only learn so much on your own and the rest you learn by accident watching somebody else using a command line.

I was using the shell for years painstakingly typing in directory names by hand. One day I was watching a friend mess around on a system and he kept hitting the tab key. I asked "why are you hitting tab?". Answer: it tries to complete the directory or filename. Who would have guessed--tab completion! Type a bit of the file or directory, hit tab, and it will try to finish what you typed (behavior depends on which shell though).

One day, said friend was watching me on the command line and watched me type something like:

coryking@cory ~/trunk/mozi $ pushd /etc
/etc ~/trunk/mozi
coryking@cory /etc $ popd
~/trunk/mozi
coryking@cory ~/trunk/mozi $

Who would have guessed!? He never knew about popd / pushd. Guess we are even...

share
1  
To me tab completion seems like something "obvious", very basic, but I never knew about popd/pushd either. Funny indeed. :) –  Jonik Mar 2 '09 at 20:46
3  
Also see this answer to be able to tab-complete a lot more than commands, files and directories: stackoverflow.com/questions/603696//603919#603919 –  Jonik Mar 2 '09 at 21:20
1  
That's so true. I learned about bash's backward-history-search (ctrl-r) by watching someone. I later realized that it was a readline feature and worked the same in other programs that incorporate readline (mysql, gdb, python -i, clisp etc). –  sigjuice Mar 23 '09 at 5:41
2  
How could you use "the shell" for years and not know about tab completion? –  prestomation Dec 18 '09 at 21:36
show 1 more comment

Old school, moving a directory tree from one place to another, preserving symlinks, permissions and all that good stuff:

tar cBf - . | (cd /destination; tar xvBpf -)

or across the network

tar cBf - . | rsh foo.com "cd /destination; tar xvBpf -)

New school:

rsync -urltv . /destination

or across the network

rsync -urltv -e ssh . foo.com:/destination
share
1  
-a is the same as '-rlptgoD' - that is recurse, copy symlinks as symlinks, preserve perms, preserve timestamps, preserve groups, preserve owners, copy device files and special files as device and special files. Also, between machines, -z (compress files before sending, decompress after receiving) is useful. rsync -avz [-e ssh] is your friend :-) –  dr-jan Apr 20 '09 at 21:10
show 4 more comments

This tip will make your CLI more comfortable (at least it makes for me):

create ~/.inputrc file with following contents:

"\e[A": history-search-backward
"\e[B": history-search-forward

Reload bash (eg by typing "exec bash"). When you type a prefix of a command and press the up arrow you will browse commands starting with your prefix, for example if you typed ssh it will show your former connections with remote shells. If your prompt is empty then the up arrow will browse the history the normal way.

share
3  
Usually I bind those to \e[5~ and \e[6~ (aka pageup and pagedown) instead of rebinding the arrows, but it is more convenient than Ctrl-R/Ctrl-S :) –  ephemient Mar 2 '09 at 21:06
4  
+1 for exec bash –  Dennis Williamson Sep 26 '09 at 15:01
show 3 more comments

My Favorite And Frequently Used:

List contents of tar.gz file

tar -tzf filename.tar.gz

will match line containing S1 OR S2 OR S3 OR S4

grep 'S1.S2.S3.*S4' file

Lists all subdirectories of current directory

ls -d */

Total size of directory
du -sh

find a date:

TIMESTAMP=date '+%y%m%d%H%M'

move process from foreground to background

Ctrl-z and then bg

Entire word to uppercase

echo "word" | awk '{print toupper($0)}'

Checks equality between numbers

x -eq y Check is x is equal to y

x -ne y Check if x is not equal to y

x -gt y Check if x is greater than y

x -lt y Check if x is less than y

Checks equality between strings

x = y Check if x is the same as y

x != y Check if x is not the same as y

-n x Evaluates to true if x is not null

-z x Evaluates to true if x is null

Command Line Parameters for ' test '

-d check if the file is a directory

-e check if the file exists

-f check if the file is a regular file

-g check if the file has SGID permissions

-r check if the file is readable

-s check if the file's size is not 0

-u check if the file has SUID permissions

-w check if the file is writetable

-x check if the file is executable

print first field of the last line"

awk '{ field = $1 }; END{ print field }'

important build in variables

$# Number of command line arguments. Useful to test no. of command line args in shell script.

$* All arguments to shell

$@ Same as above

$- Option supplied to shell

$$ PID of shell

$! PID of last started background process (started with &)

share
show 1 more comment

Some small pointers regarding log viewing:

  1. You can use tail -F to keep watching the log after it is truncated (by example log4j).
  2. You can use less: Open less, click SHIFT+F to emulate tail behaviour. Also useful combination is -S to disable line wrapping. less will enable you to search in the logs.
share
1  
And with tail -f you can watch multiple logs at once! E.g.: tail -f log1.txt -f log2.txt –  Stu Thompson Feb 16 '12 at 12:12
show 1 more comment

I'm moving towards never typing "rm" at a command prompt. Instead, I type "ls", and if I like the list of removed files I edit the command (easily possible with bash and ksh).

Edit to add something from the comments: "rm -i" will prompt for each deletion, which accomplishes the same purpose. Thanks!

share
show 5 more comments

Not the answer you're looking for? Browse other questions tagged or ask your own question.