Linux: How to use a file as input and output at the same time?

Question

I've just run the following in bash:

uniq .bash_history > .bash_history

and my history file ended up completely empty.

I guess I need a way to read the whole file before writing to it. How is that done?

PS: I obviously thought of using a temporary file, but I'm looking for a more elegant solution.

Answer 1

I recommend using sponge from moreutils. From the manpage:

DESCRIPTION
  sponge  reads  standard  input  and writes it out to the specified file. Unlike
  a shell redirect, sponge soaks up all its input before opening the output file.
  This allows for constructing pipelines that read from and write to the same 
  file.

To apply this to your problem, try:

uniq .bash_history | sponge .bash_history

Answer 2

use sponge from moreutils

uniq .bash_history | sponge .bash_history

Answer 3

The problem is that your shell is setting up the command pipeline before running the commands. It's not a matter of "input and output", it's that the file's content is already gone before uniq even runs. It goes something like:

1. The shell opens the > output file for writing, truncating it
2. The shell sets up to have file-descriptor 1 (for stdout) be used for that output
3. The shell executes uniq, perhaps something like execlp("uniq", "uniq", ".bash_history", NULL)
4. uniq runs, opens .bash_history and finds nothing there

There are various solutions, including the in-place editing and the temporary file usage which others mention, but the key is to understand the problem, what's actually going wrong and why.

Answer 4

This sed script removes adjacent duplicates. With the -i option, it does the modification in-place. It's from the sed info file:

sed -i 'h;:b;$b;N;/^\(.*\)\n\1$/ {g;bb};$b;P;D' .bash_history

Answer 5

As an interesting tidbit, sed uses a temp file as well (this just does it for you):

$ strace sed -i 's/foo/bar/g' foo    
open("foo", O_RDONLY|O_LARGEFILE)       = 3
...
open("./sedPmPv9z", O_RDWR|O_CREAT|O_EXCL|O_LARGEFILE, 0600) = 4
...
read(3, "foo\n"..., 4096)               = 4
write(4, "bar\n"..., 4)                 = 4
read(3, ""..., 4096)                    = 0
close(3)                                = 0
close(4)                                = 0
rename("./sedPmPv9z", "foo")            = 0
close(1)                                = 0
close(2)                                = 0

Description:
The tempfile ./sedPmPv9z becomes fd 4, and the foo files becomes fd 3. The read operations are on fd 3, and the writes on fd 4 (the temp file). The foo file is then overwritten with the temp file in the rename call.

Answer 6

A temporary file is pretty much it, unless the command in question happens to support in place editing (uniq doesn;t - some seds do (sed -i)).

Answer 7

if you're looking for a one-liner, how about this:

uniq .bash_history | cat > .bash_history

By the way, you have to use sort before uniq.

Best regards,

Martin