Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I've just run the following in bash:

uniq .bash_history > .bash_history

and my history file ended up completely empty.

I guess I need a way to read the whole file before writing to it. How is that done?

PS: I obviously thought of using a temporary file, but I'm looking for a more elegant solution.

share|improve this question
    
It's because the files get opened from right to left. See also stackoverflow.com/questions/146435/… – kaerast Apr 24 '10 at 18:38
    
You have to write the output to a new file in the same directory and rename that on top of the old file. Any other approach will risk losing your data if it is interrupted halfway through. Some tools may hide this step from you. – kasperd Apr 10 at 17:04
    
Or, bash won't put consecutive dupes in its history if you set HISTCONTROL to include ignoredups; see the manpage. – dave_thompson_085 Apr 13 at 4:54

12 Answers 12

up vote 27 down vote accepted

I recommend using sponge from moreutils. From the manpage:

DESCRIPTION
  sponge  reads  standard  input  and writes it out to the specified file. Unlike
  a shell redirect, sponge soaks up all its input before opening the output file.
  This allows for constructing pipelines that read from and write to the same 
  file.

To apply this to your problem, try:

uniq .bash_history | sponge .bash_history
share|improve this answer
1  
It's like cat, but with sucking capabilities :D – MilliaLover Apr 28 '10 at 0:11

I just wanted to contribute another answer that is simple, and doesn't use sponge (because it's often not included in lightweight environments).

echo "$(uniq .bash_history)" > .bash_history

should have the desired result. The string in backticks gets executed before .bash_history is opened for writing. As explained in Phil P's answer, by the time .bash_history is read in the original command, it has already been truncated by the '>' operator.

edited: the quotation marks preserve line breaks in the output.

share|improve this answer
5  
I'm not normally a fan of answers that dredge up an ancient question that already has a valid, accepted answer - but this is elegant, well-written, and makes a strong argument for its necessity (lightweight environments); for me, it really adds something to the existing set of answers. Welcome to SF, Hart (you've been here a month, but I think this is your first substantive posting). I hope to read more answers from you like this one! – MadHatter Oct 20 '13 at 9:02
3  
This is the best solution. I had to use a subshell $() instead of backticks due to some escaping issues. – CMCDragonkai Jul 5 '15 at 3:22

The problem is that your shell is setting up the command pipeline before running the commands. It's not a matter of "input and output", it's that the file's content is already gone before uniq even runs. It goes something like:

  1. The shell opens the > output file for writing, truncating it
  2. The shell sets up to have file-descriptor 1 (for stdout) be used for that output
  3. The shell executes uniq, perhaps something like execlp("uniq", "uniq", ".bash_history", NULL)
  4. uniq runs, opens .bash_history and finds nothing there

There are various solutions, including the in-place editing and the temporary file usage which others mention, but the key is to understand the problem, what's actually going wrong and why.

share|improve this answer

use sponge from moreutils

uniq .bash_history | sponge .bash_history
share|improve this answer

Another trick to do this, without using sponge, is the following command:

{ rm .bash_history && uniq > .bash_history; } < .bash_history

This is one of the cheats described in the excellent article “In-place” editing of files on backreference.org.

It basically opens the file for reading, then "removes" it. It's not really removed, though: There's an open file descriptor pointing to it, and as long as that remains open, the file is still around. Then it creates a new file with the same name and writes the unique lines to it.

Disadvantage of this solution: If uniq fails for some reason, your history will be gone.

share|improve this answer

This sed script removes adjacent duplicates. With the -i option, it does the modification in-place. It's from the sed info file:

sed -i 'h;:b;$b;N;/^\(.*\)\n\1$/ {g;bb};$b;P;D' .bash_history
share|improve this answer
    
sed still uses the temp file, added an answer with strace illustration (not that it really matters) :-) – Kyle Brandt Apr 25 '10 at 1:15
1  
@Kyle: True, but "out of sight, out of mind". Personally, I would use the explicit temporary file since something like process input > tmp && mv tmp input is much simpler and more readable than using sed trickery simply to avoid a temp file and it won't overwrite my original if it fails (I don't know if sed -i fails gracefully - I would think it would though). Besides, there are lots of things you can do with the output-to-temp-file method that can't be done in-place without something even more involved than this sed script. I know you know all this, but it may benefit some onlooker. – Dennis Williamson Apr 25 '10 at 2:54

As an interesting tidbit, sed uses a temp file as well (this just does it for you):

$ strace sed -i 's/foo/bar/g' foo    
open("foo", O_RDONLY|O_LARGEFILE)       = 3
...
open("./sedPmPv9z", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600) = 4
...
read(3, "foo\n"..., 4096)               = 4
write(4, "bar\n"..., 4)                 = 4
read(3, ""..., 4096)                    = 0
close(3)                                = 0
close(4)                                = 0
rename("./sedPmPv9z", "foo")            = 0
close(1)                                = 0
close(2)                                = 0

Description:
The tempfile ./sedPmPv9z becomes fd 4, and the foo files becomes fd 3. The read operations are on fd 3, and the writes on fd 4 (the temp file). The foo file is then overwritten with the temp file in the rename call.

share|improve this answer

A temporary file is pretty much it, unless the command in question happens to support in place editing (uniq doesn't - some seds do (sed -i)).

share|improve this answer

You can use Vim in Ex mode:

ex -sc '%!uniq' -cx .bash_history
  1. % select all lines

  2. ! run command

  3. x save and close

share|improve this answer

Another solution:

uniq file 1<> líneas.txt
share|improve this answer

You can use tee as well, using uniq output as input:

uniq .bash_history | tee .bash_history
share|improve this answer
    
No you cannot do this askubuntu.com/a/752451 – Steven Penny Apr 23 at 19:59

if you're looking for a one-liner, how about this:

uniq .bash_history | cat > .bash_history

By the way, you have to use sort before uniq.

Best regards,

Martin

share|improve this answer
    
I just want to remove adjacent duplicates – MilliaLover Apr 24 '10 at 18:57
1  
That command made my history file zero-sized too – MilliaLover Apr 24 '10 at 19:20
3  
Useless use of cat, this does exactly the same thing as the OP's command. – James Apr 24 '10 at 21:38

protected by Sven 2 days ago

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.