How to change a file on the fly along a pipe?

I'm probably looking for a way to buffer a file at the beginning of a pipe, which in contrast to this:

    cat foo.txt | grep bar > foo.txt

... would preserve the input data from destruction by the pipe itself. Is there such a buffer in stock?


I would guess sed still might create the temp file, but the following might do what you want? (Using strace on this might show you if sed creates a temp file or not).

sed -i '/bar/!d' foo.txt

The exclamation inverts the match, d is for delete, so this removes all lines that don't have bar in them.


Try using sponge from moreutils like this:

sed "s/root/toor/" /etc/passwd | grep -v joey | sponge /etc/passwd

It collects the whole input before writing to it's output.

  • 1
    I never knew about sponge, that's a fantasic tool that solves an annoying problem – Rory Dec 18 '09 at 13:01

Use >> to preserve the contents.

cat foo.txt | grep bar >> foo.txt

Now that will append to the file.

AFAIK, there is no direct way to prepend data to a file in shell. If you want to prepend you may need to use a temporary file in between.


Depending on how complex your command line is, you may get mileage out of

cat foo.txt | grep bar | tee -a foo.txt

The best way to do this is to remember how files work in unix -- the file exists as long as there is a link to the file (directory or process opening the file).

So, open the file, delete that directory entry, then run your process that writes to a new directory entry with the same name but linked to a different inode.

Lastly, the nice thing about this is that it works for any sort of pipeline operation, not just things that can do the temporary file buffer hack.

{ rm file ; sed -e s/this/that/g > file ; } < file

A word of warning -- this may cause disaster if you must have your rewrite be an atomic operation. unix doesn't offer clean ways of locking files, especially not at shell level. This was an issue in the dark ages if you were doing something like editing a password file on a busy system that was running something like NIS. Any time more than one process is reading / writing to a file, be very careful if your system is busy or important.

The only operations that are 100% certain to be atomic are directory entry manipulations -- rm / ln / mv (on the same filesystem).

So now things get longer and uglier..

The math part of this requires a posix shell or similarly extended bourne shell

      test -f file.$c.tmp

  grep stuff < file > file.$c.tmp && mv file.$c.tmp file

Ooops, UUOC, useless use of cat :) you don't need cat there :

grep something somefile > someotherfile

As sekenre said, there is sponge. Or maybe you may want to try perl inline edition ( -i switch ) :

  • print only matching output:

    perl -ni -e 'm/foo/g and print ' bar.txt

  • replace "foo" with "bar" in the file :

    perl -pi -e 's/foo/bar/g' bar.txt

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.