I need to migrate about 1TB of data comprised of smaller files (most under 100KB) to another server. I've not even completely enumerated the files but estimates are between 1-2 million.

The initial copy using SCP took over a week. Now we have to synchronize changes. Hundreds to thousands of files are added daily.

I've attemped using rsync (v3) but it is taking too long. By the time it finishes, we will be back to having data out of sync again.

I've seen similar questions here but they are a bit older and wonder if there are any new tools to help with this process.

Issues are futher complicated by the source data being on a shared iSCSI system with poor read performance.

The latest strategy may be to redo the data migration and have the developers write a tool to log all of the new files that are added during the migration process. The directory structure keys off a unique identifier is is very broad and deep, so new files are scattered within this structure and rewritting the app to put new files into a specific directory will not work.

Any strategies appreciated.

OS is RHEL 5 going to RHEL 6.

  • Instead of copying to a new server, how about using a NAS that both servers mount? – ott-- Dec 6 '11 at 16:50
  • Do you need real-time or near-real-time synchronization, or is the secondary machine just a backup? – voretaq7 Dec 6 '11 at 17:18
  • This sounds like a full migration from one system to another. – ewwhite Dec 6 '11 at 17:20
  • Hardware is located with a on demand infrastructure provider with limited budget. Ideally, this would be done on a SAN with LUN replication but that is not in the works. This is for a real estate site where we are moving millions of images to a dedicated system to manage growing storage needs and performance issues. – jeffatrackaid Dec 6 '11 at 18:16
  • Dev's are going to re-pull last 30 days of data from primary source. Apparently that is easier than digging in the database. Thanks for the suggestions. – jeffatrackaid Dec 6 '11 at 20:31

I'd be tempted to answer "stop abusing the file system by treating it like a database" but I'm sure it wouldn't help you much ;)

First, you need to understand that if your limitation is in the bandwidth available on read, there isn't anything you can do to improve performance using a simple synch command. In such a case, you'll have to split the data when it's written either by changing the way the files are created (which means, as you guessed correctly, asking the devs to change the source program) or by using a product that does does geo-mirroring (like, for instance double-take: check around as I'm sure you'll find alternatives, that's just an example).

In similar cases, the main cause of problem isn't typically the file data but rather the meta-data access. Your first strategy will therefore be to divide the load into multiple process that act on (completely) different directories: that should help the file system keep up with providing you with the meta-data you need.

Another strategy is to use your backup system for that: replay your last incremental backups on the target to keep the database in sync.

Finally, there are more exotics strategies that can be applied in specific cases. For instance, I solved a similar problem on a Windows site by writing a program that loaded the files into the file system every few minutes, thus keeping the FS clean.

  • This is a consulting engagement with limited input as to the application design, so we are operating a bit blind on that front. I am hoping that the database records the path to the new file. If so, then an initial sync followed by pulling a list from the DB and moving those files. Will repeat this process until the sync can be completed in under 24 hours. We can then flip DNS and move the hosting of these files to the new system. – jeffatrackaid Dec 6 '11 at 18:11
  • +1 for parallel execution recommendation. considering that but was avoiding having to script it up. – jeffatrackaid Dec 6 '11 at 18:21

I don't think anything has changed. If you can quiesce the data on the source system, I think some variant of tar will be the fastest. If not, rsync is still the next best way, making sure to use the whole-file switch and a less CPU-intensive compression algorithm (e.g. arcfour). Do you have any option to perform a block-level copy? You mention iSCSI storage. Will the new system have iSCSI-attached storage as well?

  • New system is RAID 10 SATA which is actually about 3x faster than the shared iSCSI. We have a block level backup but the restore process is pretty slow when doing a file level restore. Bare metal or full partition restores are faster. That is an option we are considering. – jeffatrackaid Dec 6 '11 at 18:11

This is being done in phases:

1) initial transer using scp 2) some data freshed with rsync 3) devs are writing a script to pull files added since step 1 to the system 4) will proxy data from original server to new server during dns change 5) change dns and get rid of under performing shared iSCSI services.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.