Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

How to change a file on the fly along a pipe?

I'm probably looking for a way to buffer a file at the beginning of a pipe, which in contrast to this:

    cat foo.txt | grep bar > foo.txt

... would preserve the input data from destruction by the pipe itself. Is there such a buffer in stock?

share|improve this question

6 Answers 6

up vote 6 down vote accepted

I would guess sed still might create the temp file, but the following might do what you want? (Using strace on this might show you if sed creates a temp file or not).

sed -i '/bar/!d' foo.txt

The exclamation inverts the match, d is for delete, so this removes all lines that don't have bar in them.

share|improve this answer
    
Yeah, this is it, thanks. –  Alex Jun 24 '09 at 11:57

Try using sponge from moreutils like this:

sed "s/root/toor/" /etc/passwd | grep -v joey | sponge /etc/passwd

It collects the whole input before writing to it's output.

share|improve this answer
    
I never knew about sponge, that's a fantasic tool that solves an annoying problem –  Rory Dec 18 '09 at 13:01

Use >> to preserve the contents.

cat foo.txt | grep bar >> foo.txt

Now that will append to the file.

AFAIK, there is no direct way to prepend data to a file in shell. If you want to prepend you may need to use a temporary file in between.

share|improve this answer

Depending on how complex your command line is, you may get mileage out of

cat foo.txt | grep bar | tee -a foo.txt
share|improve this answer

The best way to do this is to remember how files work in unix -- the file exists as long as there is a link to the file (directory or process opening the file).

So, open the file, delete that directory entry, then run your process that writes to a new directory entry with the same name but linked to a different inode.

Lastly, the nice thing about this is that it works for any sort of pipeline operation, not just things that can do the temporary file buffer hack.

{ rm file ; sed -e s/this/that/g > file ; } < file

A word of warning -- this may cause disaster if you must have your rewrite be an atomic operation. unix doesn't offer clean ways of locking files, especially not at shell level. This was an issue in the dark ages if you were doing something like editing a password file on a busy system that was running something like NIS. Any time more than one process is reading / writing to a file, be very careful if your system is busy or important.

The only operations that are 100% certain to be atomic are directory entry manipulations -- rm / ln / mv (on the same filesystem).

So now things get longer and uglier..

The math part of this requires a posix shell or similarly extended bourne shell

c=0
    while
      f=$SECONDS
      test -f file.$c.tmp
    do
      c=$(($c+1))
    done

  grep stuff < file > file.$c.tmp && mv file.$c.tmp file
share|improve this answer

Ooops, UUOC, useless use of cat :) you don't need cat there :

grep something somefile > someotherfile

As sekenre said, there is sponge. Or maybe you may want to try perl inline edition ( -i switch ) :

  • print only matching output:

    perl -ni -e 'm/foo/g and print ' bar.txt

  • replace "foo" with "bar" in the file :

    perl -pi -e 's/foo/bar/g' bar.txt

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.