2

I have two scripts that watch a directory (postgres warm-standby)

one script copies compressed files into the directory and then uncompresses them.

the other script watches for the uncompressed files and then ingests them into postgres.

Im wondering if i could his a case where the ingestor script could read teh output of gunzip while it was writing it?

in short: does gunzip

  1. decompress and then change name
  2. copy into new name as it uncompresses
  3. something else

2 Answers 2

1

It is non-atomic, as you can see if you do an ls during a large compression:

dfranke@alatar:~$ dd if=/dev/zero of=bigfile bs=1M count=3072
3072+0 records in
3072+0 records out
3221225472 bytes (3.2 GB) copied, 39.1106 s, 82.4 MB/s
dfranke@alatar:~$ gzip bigfile &
[1] 19876
dfranke@alatar:~$ ls -l bigfile*
-rw-r--r-- 1 dfranke staff 3221225472 2010-11-08 17:16 bigfile
-rw------- 1 dfranke staff     114688 2010-11-08 17:18 bigfile.gz
dfranke@alatar:~$ ls -l bigfile*
-rw-r--r-- 1 dfranke staff 3221225472 2010-11-08 17:16 bigfile
-rw------- 1 dfranke staff     212992 2010-11-08 17:18 bigfile.gz
1
  • @Arthur: Even more informative is strace gunzip bigfile.gz. You'll see an open of bigfile.gz, then read, then open of bigfile, then write to bigfile, close of bigfile.gz, close of bigfile, then finally unlink of bigfile.gz. (I used a smaller file, so you'd probably see more read and write operations for a bigger file.) Nov 8, 2010 at 22:43
1

It's probably not a good idea to base the correctness of your scripts on assumptions about how a particular utility (such as gunzip) works, since the internal behaviour of a utility can change in subsequent releases. I recommend, instead, that you use the proper locking methods (as provided by your particular OS) to ensure mutual exclusion. Most Linux distros provide flock for use in shell scripts.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .