Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have two scripts that watch a directory (postgres warm-standby)

one script copies compressed files into the directory and then uncompresses them.

the other script watches for the uncompressed files and then ingests them into postgres.

Im wondering if i could his a case where the ingestor script could read teh output of gunzip while it was writing it?

in short: does gunzip

  1. decompress and then change name
  2. copy into new name as it uncompresses
  3. something else
share|improve this question

2 Answers 2

up vote 1 down vote accepted

It is non-atomic, as you can see if you do an ls during a large compression:

dfranke@alatar:~$ dd if=/dev/zero of=bigfile bs=1M count=3072
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB) copied, 39.1106 s, 82.4 MB/s
dfranke@alatar:~$ gzip bigfile &
[1] 19876
dfranke@alatar:~$ ls -l bigfile*
-rw-r--r-- 1 dfranke staff 3221225472 2010-11-08 17:16 bigfile
-rw------- 1 dfranke staff     114688 2010-11-08 17:18 bigfile.gz
dfranke@alatar:~$ ls -l bigfile*
-rw-r--r-- 1 dfranke staff 3221225472 2010-11-08 17:16 bigfile
-rw------- 1 dfranke staff     212992 2010-11-08 17:18 bigfile.gz
share|improve this answer
    
thanks for testing that out. good to know :) –  Arthur Ulfeldt Nov 8 '10 at 22:19
    
@Arthur: Even more informative is strace gunzip bigfile.gz. You'll see an open of bigfile.gz, then read, then open of bigfile, then write to bigfile, close of bigfile.gz, close of bigfile, then finally unlink of bigfile.gz. (I used a smaller file, so you'd probably see more read and write operations for a bigger file.) –  Dennis Williamson Nov 8 '10 at 22:43

It's probably not a good idea to base the correctness of your scripts on assumptions about how a particular utility (such as gunzip) works, since the internal behaviour of a utility can change in subsequent releases. I recommend, instead, that you use the proper locking methods (as provided by your particular OS) to ensure mutual exclusion. Most Linux distros provide flock for use in shell scripts.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.