Tag Info

Hot answers tagged

7

There are many limits for transferring many small files. Some have already been mentioned: network latency, disk write speed, etc. However most of those can be optimized best by using "rsync". If the files don't exist on the destination, and you are pretty sure the process won't be interrupted, using tar piped to tar will be very efficient: cd /SOURCE/DIR ...


2

Based on the documentation you quoted I find it quite clear that one is covering the entire duration from one system call to the next, while the other covers only the time within a system call. The percentage of time spent inside system calls versus the percentage of time spent outside system calls will roughly tell you if a process is CPU bound. A CPU ...


2

The time outside the system call is the time spent running your program's code before it gets to the next system call.


1

Keep in mind that harddrives have a certain sector size. Writing normally goes per sector, so you should expect lower speeds below the sector size. In additionn most raid configuration either apply mirroring (same data on multiple disks, speeeds up reading as information only needs to be read from 1 disk instead of X disks, however writes have to go to all ...


1

There is an elegant solution developed by William Glick: parallelizing rsync. /bin/bash # SETUP OPTIONS export SRCDIR="/folder/path" export DESTDIR="/folder2/path" export THREADS="8" # RSYNC TOP LEVEL FILES AND DIRECTORY STRUCTURE rsync -lptgoDvzd $SRCDIR/ /$DESTDIR/ # FIND ALL FILES AND PASS THEM TO MULTIPLE RSYNC PROCESSES cd $SRCDIR; find . -type f | ...



Only top voted, non community-wiki answers of a minimum length are eligible