0

I'm trying to run Rsync to move static files from our Dedicated server to our new cloud server.

The following command works but when it runs it is stopping any web pages being served

rsync -azP --delete -e 'ssh -p 1234' 
    /var/www/vhosts/example.com/httpdocs/images/20* 
    user@server:/var/www/vhosts/example.com/httpdocs/images/.

So I've tried using nice and ionice to lower the CPI and I/O priority, and using a different cipher over SSH - I'd found posts suggesting this will lower the server load:

nice -n19 ionice -c2 -n7 rsync -azP --delete --rsh="ssh -c arcfour" -e 'ssh -p 1234' 
    /var/www/vhosts/example.com/httpdocs/images/20* 
    user@server:/var/www/vhosts/example.com/httpdocs/images/.

But this does not make a difference, as soon as the script runs pages on http://example.com stop serving, and once I use CTRL+C to kill the Rsync the command the pages start serving again.

I could put this in a crontab job to run when site traffic is low, but I'd like to be able to sync the servers through the day.

Is there a way of getting Rsync to stop blocking apache or is there another way I can transfer the data?

I've also tried making a tar.gz archive and transferring via scp, but the file is 2.5GB and I just get the same problem.

We will be moving all our static content onto Amazon S3 - but that's a much larger job and I'm hoping I can just get the static content moved across in the meantime to kill off the old machine

  • If you are using virtualization, try to do some hostsystem-based solution (I know it is dirty - sorry). – peterh says reinstate Monica Sep 16 '14 at 8:11
  • It's transferring from a Dedicated machine to a Cloud server within Rackspace, so I don't think that's an option – Pete Sep 16 '14 at 9:46
0

I would not expect that Apache would stop serving during transfer. I can think of four things that could slow down Apache.

  • Bandwidth saturation. Use --bwlimit when starting rsync.
  • CPU (although rsync shouldn't be a problem). Use top to monitor. Use nice or renice to reduce priority.
  • Disk IO. Use ionice with idle priority.
  • Memory. Consider dropping the number of Apache child processes to release some memory. top may show some issues.

For the initial transfer, I would consider using tar to transfer the content. You can then use rsync to pick up any changes.

As has already been noted, consider remounting with noatime or relatime as updating the atime can put a lot of load on the disks.

I would consider using sar to monitor the system to see which resource is the bottleneck. It can monitor all your resources. Capture the data or output to file so you can examine it a leisure.

0

Get faster disks. Rsync do not block apache. Shortage of io operations do.

  • if you are using RAID, check its health – thanasisk Sep 16 '14 at 8:06
  • That would mean upgrading the hard discs in my dedicated machine, which we're about to get rid of. But thanks, I had thought it would be CPU, hadn't thought of I/O – Pete Sep 16 '14 at 8:56
  • Also I've tried using the ionice command to reduce the i/o operations to best-effort – Pete Sep 16 '14 at 9:04
  • try remounting the partitions without atime - that will give you some I/O leeway – thanasisk Sep 16 '14 at 9:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.