Tag Info

Hot answers tagged

8

If you really need a quick way to transfer files, and both systems are Linux-based, you can try UDR. This is really a form of rsync-over-UDP (using the open-source UDT framework) and is particularly handy for moving large numbers of files or transferring over high-bandwidth or high-latency links. In addition, encryption is disabled by default, so the ...


7

During an rsync transfer are you CPU bound, or is your link saturated. If your link is saturated, but your CPU is idle, then compress. If your CPU is maxed, and your link isn't, then do not compress.


3

I work this quite often, migrating data between servers and during impossibly-long data transfers... The short answer, of course, is to test with your specific data... This is really easy to do, right? Try a LAN transfer with compression off, then try it with compression on... In my experience with production data sets across a few environments, on a GigE ...


2

Revised answer: LXC containers share the same kernel as the host, so any filesystem they mount should be accessible from outside. If you do a cat /proc/mounts on the host, can you see the container filesystems? If you see a line like /dev/mapper/... /var/lib/lxc/o1/rootfs ext4 ... then you should be able to access /var/lib/lxc/o1/rootfs from the host, ...


2

If used in daemon mode without encryption, rsync can efficiently transfer large amount of small files. Give it another try using it in daemon mode.


2

You have virtually innumerable options for copying files between servers. FTP, SMB (drag and drop, or a command line copy like robocopy), BITS, HTTP, even SCP. Which one is "best" depends entirely on your preferences and specific situation (which you haven't told us enough about to make a determination on). As to speed, you're transferring 200GB over the ...


2

Sorry I dont have a new enough cred to just comment. But something like this should help you with the zipping of the folders. #! /bin/bash DIR=$1 i=0 arr=() zipnum=0 cd $DIR for item in * do if [[ -d $item ]]; then if [ $i -lt 2 ]; then arr+=($item) ((i++)) else ...


1

The below script was tested on CentOS 6.5. Walks through the files under a specified folder and adds them one by one to zip file(s) no bigger than a specified max-size. Adding files to a zip one at a time can cause CPU or Storage I/O spikes: #!/bin/bash files_folder="/home/username/public_html/" zip_folder="/home/username/" zip_maxsize=524288000 # ...


1

Like Zoredache said, if your link isnt saturated no -z. Also, something else you can consider to help manage long transfers is the bwlimit flag.


1

You can use this simple windows FTP tool, in command line; Check wiki page for manual. http://sourceforge.net/projects/ftpandfilesystemsynchronizer/


1

If you are using cmder (or msysgit/mingw that has scp & ssh), I just wrote a simple python script for this. It can be found here: https://gist.github.com/ceilfors/fb6908dc8ac96e8fc983 Sample usage: python ssh-copy-id.py user@remote-machine. Password will be prompted upon running the script.


1

In Windows 7 there is a ssh.exe Here is what worked for me: 1. create identity (on windows) c:\>ssh-keygen That created an identity file in the home directory. I changed the name of the public key to "id_rsa" 2. copy the file to the target linux system using the ssh Credits to http://serverfault.com/users/984/zoredache for his answer c:\>ssh ...


1

The OpenSSH server does not support this. WinSCP SFTP client can do this from a client side, if that helps. See http://winscp.net/eng/docs/resume#automatic By default is uses the .filepart suffix for files over 100 KB only, but you can configure it to use it for any file. See http://winscp.net/eng/docs/ui_pref_resume (I'm the author of WinSCP)


1

rsync -vP works for me. From the man page: "-P same as --partial --progress"


1

Depends on what you want to do.It's almost always faster to copy a single 10 gig file than 10 gig with 1000's of small files (because of connections etc) although you could speed up the small file size by using concurrent connections. SFTP/SCP are good bets for general use. Rsync and Robocopy (windows only) are more specific tools.Torrents are a very ...


1

Note: the following is all going off theory -- the real right way to make sure this is correct in your situation is to run tests on various combinations of options. The data connections in an rsync operation look something like this: Source disk <-> rsync instance <-> other rsync instance <-> destination disk In general, rsync is ...



Only top voted, non community-wiki answers of a minimum length are eligible