Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I'm running a nightly bash script to sync a remote folder (source) with a local folder (target). I've tested this script, based on rsync, and it works fine in a root shell. It takes time since there are hundred of gigs to copy but it works.

Once I use it in crontab my server runs out of memory.

My server has 8GB of RAM, 4GB of swap, and as I said, the script never goes OOM when manually ran from a shell. It's a default Centos 5.5 installation. I could split the load and sync all the 2nd level dirs in a find/for script, but I'd like to keep it simple and only sync the top level directories.

I cannot make too many tests since this server is used to host websites and other services and I cannot afford to hang it just for testing purpose. Do you know a setting that could allow cron to finish this job normally ?

#!/bin/bash

BACKUP_PATH="/root/scripts/backup"

rsync -av --delete /net/hostname/source/ /export/target/ >     $BACKUP_PATH/backup_results_ok 2>  $BACKUP_PATH/backup_results_error

edit: cron configuration is default, as /etc/security/limits.conf, which is all commented out.

share|improve this question
    
Is this a VPS or a dedicated server? Maybe the hosting provider has some limitations on cron jobs. –  HD. Oct 14 '10 at 14:49
2  
Have you checked /etc/security/limits.conf ..? –  danlefree Oct 14 '10 at 15:30
    
No, it's a dedicated server. I have physical access of it. –  Rosco Oct 14 '10 at 15:31

2 Answers 2

Even though limits.conf is commented out, I suggest that you test just to make sure. One way would be to create a cronjob that just contains something like "ulimit -a | Mail -s 'limits' me@example.com" to have the info emailed to you. Once you know what the limits are, then you can just reset them in the shell script that actually runs the rsync:

#!/bin/bash
ulimit -d unlimited
ulimit -m unlimited
ulimit -s unlimited
rsync [...]
share|improve this answer

The issue is you need to setup your ulimits explicitely (just like Deutsch recommend), because of system setting. When you invoke a script from the cron, limits are not always set (according to the distribution). Set it in your script and it will work.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.