1

I've built an incremental backup solution which uses RSYNC to backup a few of our servers. I'm using PHP to run through config files to get the information for each server that needs to be backed up. PHP then calls RSYNC to handle the remote backup of the servers, incrementally.

This works perfectly on all of our servers, and takes just a few minutes to finish....all except for one server. This server has a lot of data, and it seems that RSYNC just hangs on it. It takes over 3 days to do a single incremental backup. My guess is that it's stuck on building the file list.

When I run the below command on the folder I want to backup, here are the 'iused' results.

df -i folder/
54176307

Is this simply too much data for RSYNC to handle? Should I be looking into another alternative? The backup server is currently running on version 3.0.8, however, the clients being backed up are all running RSYNC 2.6.9. Do you think upgrading everything to 3.0.8 would make a difference and reduce the 3 day backup time for this server?

Thanks, Jacob

6
  • 1
    what's the origin filesystem?
    – Chopper3
    Jul 20, 2012 at 18:36
  • It wouldn't hurt to try upgrading to the latest rsync. It's very easy to build it from source, if pre-built packages are not readily available. Jul 20, 2012 at 18:38
  • 1
    What flags are you passing to rsync? Jul 20, 2012 at 19:46
  • 1
    It will make a huge difference -- just look at the rsync change log.
    – jftuga
    Jul 20, 2012 at 19:58
  • To answer your questions regarding the filesystem, they are Mac OS X Journaled drives.
    – Jacob Haug
    Jul 20, 2012 at 23:54

2 Answers 2

3

I doubt that the upgrade alone will provide the sort of improvement you're looking for. At 72 hours, you'd probably want an order of magnitude performance increase (7.2 hours). If you're looking for 2-3 hours, good luck without an SSD and a good network.

With 55 million inodes (assuming approx as many files), you're going to have to seriously reconsider your approach. First, if you are using an ext variant I'd consider benchmarking a different FS.

Second, if you're using an ext FS (say ext3/4), the first thing I'd do is shut off atime! With atime on, every time a file is read/looked at the filesystem has to do a tiny write to the disk as atime means "access time". By shutting it off, you lose the ability to see when a file was accessed but that's the way the cookie crumbles. If you are using a standard SATA disk, assume you can do 100 IOs per second (IOPS). Each access write takes one of those (worst case). That means 100 files a second just to verify it's existence, and by the time you read it you are using even more IOPS. 55000000/100 = 550000s = 152 hours. Once you figure in the kernel's very good algorithms to merge the IOPS, you've probably found your bottleneck.

In your /etc/fstab, use the mount option:

noatime,nodiratime 

to completely disable atimes. Leave off nodiratime to leave access times off for directories. If you have a lot of directories, I'd recommend turning it off.

I bet this alone will help dramatically.

Here's an example fstab:

# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
# / was on /dev/sda1 during installation
UUID=66188c62-0d8c-46d8-a44f-8f673ca6ac98 /               ext4    errors=remount-ro,discard,noatime,nodiratime 0       1
# swap was on /dev/sda6 during installation
UUID=c3f40312-d6f9-4bb7-b426-602a4b7a6c47 none            swap    sw              0       0
4
  • 1
    I find it useful to know if a file was accessed since it was last created or updated. Consider using relatime to record that the file was read after it was modified or created.
    – BillThor
    Jul 20, 2012 at 23:26
  • This is great information. Very helpful. We're using the Mac OS X Journaled filesystem, but much of the information you provided is very helpful. Mac OS X has a process similar to this called fseventd, and I have disabled that already. Do you have any suggestions for a replacement? Is there even a possible solution for this large of a directory?
    – Jacob Haug
    Jul 20, 2012 at 23:58
  • noatime implies nodiratime (see here). So if you mount with the former, you get the latter automatically. Jul 21, 2012 at 0:40
  • @Jacob, at the end of the day you just need to find your bottleneck. If you free up IOPS, network becomes your bottleneck, then CPU, then back to IOPS, then network and so on. This is computer science;) I would first try to find your bottleneck and fix that one. I bet it's IOPS. With linux you can use systat to monitor your i/os, merges, % utilization and time it took the disk to service the last round of requests. This comment box is too small to go into much detail, but each setup is different and there are no rules except if sysstat is showing 100% util, then get a raid or ssd (or both).
    – fimbulvetr
    Jul 24, 2012 at 15:07
0

I have a couple of scripts that do something of the sort. I think the right solution is to have find scour the file system for things that have changed since the last backup and then have rsync to the "syncing" however I have not worked that out.

Instead what I have is 2 scripts 1 that finds top level directories to backup and the other that backs up each of these in parallel. I find with NFS file store I get pretty high CPU utilization at around 10 parallel rsyncs. Since the job is neary CPU bound at that point whereas a single rsync is closer to 7% of the cpu I use xargs to run each individual job but to run 7 jobs simutaneously with the -P option.

I can email script if anyone is interested. They should be pretty readable.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .