Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Hi I have a set of neat scripts that backup my dbs into db.sql.gz files under a folder on on ec2 host. My files are of the form (as you can see they are weekly rolling backups)

/var/backs/db/db1_Mon.sql.gz
/var/backs/db/db1_Tue.sql.gz
/var/backs/db/db1_Wed.sql.gz
...
...
/var/backs/db/db4_Fri.sql.gz
/var/backs/db/db4_Sat.sql.gz
/var/backs/db/db4_sun.sql.gz

Now I would like to somehow (well using s3sync) at say 3am every morning stream these to an s3 bucket called mys3bucket/myec2instance/var/backs/db/ and then all my .sql.gz files under that directory.

To do this I have the following script:

#!/bin/sh
cd /var/backs
ruby s3sync/s3sync.rb -r -s -v  /var/backs/db mys3bucket:myec2instance/var/backs/db/ > /var/backs/logs/s3sync.log

and to invoke the whole contraption I use the following cron entry:

1 3 * * * /var/backs/s3.sh # this is where the script resides.

The problem with all this is that it works rather intermittently .... Actually if I kick it off manually it tends to work find ... pretty efficient

Any thoughts would be mighty appreciated.

share|improve this question
    
What size instance are you using; I have had problems in development with long running tasks on micro instances failing. –  EasyEcho Nov 7 '12 at 21:45
    
As a matter of fact ... most of these are small ... now the instances themselves run like champs. –  user1172468 Nov 7 '12 at 21:54
    
Have you tried the python implementation of s3sync? –  cjc Nov 7 '12 at 22:30
    
@cjc ... actually no but I will ... is there an alternative tool to s3sync ... I must say I'm thoroughly disappointed with s3sync for now :-( –  user1172468 Nov 8 '12 at 1:15
    
I can't think of a good alternative on the command line. s3cmd, the Python version, works well for us, though we've had to use the most recent version because of slight bugginess with the older version that's packaged on, say, Ubuntu. –  cjc Nov 8 '12 at 2:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.