Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I need to deploy an automated process (via 1 min cron script) that looks for tar files in a specific directory. If a tar file is found, it is untarred to the appropriate location and then the tar file is deleted.

The tar files are automatically copied to this server over SSH from another server. In some cases, the tar files are extremely large, with lots of files.

The problem that I am expecting to run into: If it takes > 1 minute for the tar file to be copied to the server, and the cron script runs once every minute, it's going to see the .tar.gz file and try to do untar it, even though the tar file is still in the process of being written to.

Is there any way (via bash commands) to test if a file is currently being written to, or if it's only a partial file, etc?

One alternative I was thinking of was to have the file be copied as a different file extension (like .tar.gz.part) and then renamed to .tar.gz after the transfer is complete. But I figured I'd try to figure out if there is simply a way to determine if the file is whole at the command line first... Any clues?

share|improve this question
    
How exactly is the file being transferred? For example, rsync uses a temporary filename during the transfer (by default), and only after the file is completely transferred, renames it to the actual filename. –  Piskvor Mar 27 at 15:07
add comment

3 Answers

up vote 4 down vote accepted

You are on the right track, renaming the file is an atomic operation, so performing the rename after upload is simple, elegant and not error prone. Another approach I can think of is to use lsof | grep filename.tar.gz to check if the file is being accessed by another process.

share|improve this answer
    
Do you mean 'not' error prone? –  Jakobud Aug 8 '12 at 16:09
    
Yeah, thanks for correction! –  Alex Aug 8 '12 at 16:10
    
Checking the process is a good way to do it but I think I'm going to go with the renaming route because it's so simple and fool proof. –  Jakobud Aug 8 '12 at 16:27
add comment

Your best bet is to use lsof to determine if a file has been opened by any process:

#  lsof -f -- /var/log/syslog
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
rsyslogd 1520 syslog    1w   REG  252,2    72692 16719 /var/log/syslog

You can't easily tell if it's in the process of being written to, but if it is being written to, it MUST be open.


Edit: let's solve the actual problem here rather than try to implement the proposed solution!

Use rsync to transfer the file:

○ → rsync -e ssh remote:big.tar.gz .

This way, the file won't be copied over top of the existing one but copied into a temporary file (.big.tar.gz.XXXXXX) until transfer is complete, then moved into place.

share|improve this answer
add comment

The best way to do this is to use incron. It allows you to set an inotify watch on a directory which will then notify you of file operations. In this case, you should watch the dir for a close_write. That'll allow you to then run your command once the file was closed after a write.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.