The tag has no usage guidance.

learn more… | top users | synonyms

138
votes
13answers
151k views

Copying a large directory tree locally? cp or rsync?

I have to copy a large directory tree, about 1.8 TB. It's all local. Out of habit I'd use rsync, however I wonder if there's much point, and if I should rather use cp. I'm worried about permissions ...
84
votes
12answers
11k views

Transfer 10TB of files from USA -> UK datacenter

I am migrating my server from the USA to the UK from one data center to another, my host said I should be able to achieve 11 megabytes per second. Operating system is Windows Server 2008 at both ends....
69
votes
11answers
10k views

Transfer 15TB of tiny files

I'm archiving data from one server to another. Initially I started a rsync job. It took 2 weeks for it to build the file list just for 5 TB of data and another week to transfer 1 TB of data. Then I ...
63
votes
23answers
102k views

How to copy a large number of files quickly between two servers

I need to transfer a huge amount of mp3s between two serves (Ubuntu). By huge I mean about a million files which are on average 300K. I tried with scp but it would have taken about a week. (about 500 ...
59
votes
4answers
114k views

robocopy transfer file and not folder

I'm trying to use robocopy to tranfer a single file from one location to another but robocopy seems to think I'm always specifying a folder. Here is an example: robocopy "c:\transfer_this.txt" "z:\...
47
votes
16answers
14k views

Favorite rsync tips and tricks

The more I use rsync the more I realise that it's a swiss army knife of file transfer. There are so many options. I recently found out that you can go --remove-source-files and it'll delete a file ...
43
votes
8answers
37k views

Quickest way to transfer 55GB of images to new server

I currently have two CentOS servers. I need to know how and what the quickest way would be to "tar" up the images directory and SCP it over? Is that the quickest way that I just suggested, because ...
29
votes
8answers
21k views

Is there an equivalent to ssh-copy-id for Windows?

Is there any equivalent or port of ssh-copy-id available for Windows? That is, is there an easy way to transfer SSH keys from a local machine to a remote server under Windows? In case it helps, I'm ...
27
votes
1answer
14k views

rsync - Exclude files that are over a certain size?

I am doing a backup of my desktop to a remote machine. I'm basically doing rsync -a ~ example.com:backup/ However there are loads of large files, e.g. wikipedia dumps etc. Most of the files I care a ...
21
votes
5answers
54k views

What is the best way to transfer a single large file over a high-speed, high-latency WAN link?

This looks related to this one, but it's somewhat different. There is this WAN link between two company sites, and we need to transfer a single very large file (Oracle dump, ~160 GB). We've got full ...
15
votes
10answers
27k views

Easiest way for fast transfer of files between linux servers?

I need to transfer files from one CentOS server to another. Will transfer 5MB files about every 10 minutes. Do not need encryption. What is an easy was for fast transfer of files? Is there ...
14
votes
3answers
19k views

Getting rsync to move file from source to destination?

Is rsync is a good choice for my project ? I have to : - copy files from source to destination folder via SSH, - be sure all files are copied, - delete source files after copy. - if I have conflict ...
13
votes
23answers
22k views

What is the simplest and fastest way to transfer large file through a Windows network?

I have a Window Server 2000 machine running MS SQL Server that stores over 20GB of data. The database is backed-up every day to the second harddrive. I want to transfer those backup files to another ...
13
votes
4answers
27k views

ASCII vs Binary vs Auto?

I'm transferring files between servers and just started noticing that some of them are getting modified to be one long continuous line as opposed to having the returns and line-breaks they originally ...
13
votes
9answers
11k views

Windows rsync that supports long file names, or an good alternative?

I currently rsync on a Linux host to copy things from Windows to my Linux box. But I can't copy files with long names. I have tried DeltaCopy, cwrsync, and cygwin. From what I have found there all ...
12
votes
6answers
24k views

How do you limit the bandwidth for a file copy?

I've got an old windows 2000 box in a remote location with a T1 connection and a vpn to my location. I normally use smb mounts to transfer files but now it's time to decommission the server and copy ...
12
votes
1answer
4k views

Getting rsync to delete files from the *sender* after transfer

I'm using rsync to move one directory to another. Is there some way to get rsync to incrementally delete a file from the sender after it has copied it to the reciever?
12
votes
4answers
7k views

Rsync show progress for individual file

I'm using rsync to upload/download a pretty large file (20Gb). I'm aware of --progress option, but what it does in my case is just showing me the result at the end. My problem is that during some ...
11
votes
2answers
16k views

What a pros and cons of FTP vs SSH for file transfers?

Why to consider setting up an FTP service on a server when files transfers work fine by means of SSH (with Midnight Commander on a Lunux and FileZilla on a Windows client)? What are pros and cons of ...
10
votes
6answers
8k views

what is the fastest and most reliable way of transferring a lot of files?

I'm trying to transfer about 100k files totaling 90gb. Right now I'm using rsync daemon but its slow 3.4mb/s and I need to do this a number of times. I'm wondering what options do I have that would ...
9
votes
4answers
7k views

Transfer files via SSH

I want to use file transfer via SSH on some scripts. I've read it's possible to tar over ssh. Where should I start reading?
9
votes
4answers
2k views

Linux to Linux, 10 TB transfer?

I've looked at all the previous similar questions, but the answers seemed to be all over the place and no one was moving a lot of data (100 GB is different from 10 TB). I've got about 10&...
6
votes
10answers
4k views

Using rsync to quickly upload a file that is similar to another file

I'm putting together a deployment script which tars up a directory of my code, names the tar file after the current date and time, pushes that up to the server, untars it in a directory of the same ...
6
votes
4answers
2k views

Puppet file transfer slow

I have a puppet master and slaves in different datacenters. The latency between them is ~40ms. When I run "puppet agent --test" on a slave to apply the latest manifest it takes ~360 seconds to finish. ...
6
votes
6answers
2k views

Remote file copy util (like rsync) but that will take account of data already copied (in this session)?

Let's say I have a directory with 2 files, both are identical and quite large (e.g. 2GB ea.) I want to rsync that directory to a remote host. As I understand it (and I could be wrong), rsync ...
6
votes
3answers
633 views

Hosted form of web-based file transfer tools [closed]

I'm looking for some software that will provide a service that resembles SendThisFile.com. My users generally turn to services like these when they are trying to send files that exceed the mail ...
5
votes
5answers
1k views

Bash: mv directory one at a time

I am trying to move all subdirectories of a folder to another share on the same server. If I do a mv *, I will run out of space since the folders are not removed until all folders get transferred. ...
5
votes
4answers
1k views

Transfer large amount of small files

I've to do a migration of two servers with large SAN attachments to our new VMWare environment. EDIT: I have to supply some additional intelligence as I have good answers regarding VMWare solution. ...
5
votes
2answers
253 views

Faster, secure, protocol/code required for long-distance transfer

I've ran into a problem and I'm looking for a new secure protocol/client/server that's faster over a 1Gb/s fibre link - let me tell you the story... I have a pair of redundant, diversely-routed, ...
5
votes
2answers
3k views

Best Practice: Apache File Upload

I am looking for a soultion for trusted users to upload pdf files via html forms (with maybe php involved). This is quite a standard ubuntu linux server with apache 2.x and php 5. I am wonderiung ...
5
votes
1answer
11k views

ESXi 5.5, slow network speed when writing to guests

This is an issue that has been bugging me for a lot time with ESXi. I had the issue with my old Dell server (ESXi 5.0 and 5.5) and even with my new Dell T20 (ESXi 5.5). For the current issue at hand I ...
5
votes
3answers
4k views

ESXi 4.1 slow file transfer

My problem is that when I transfer files to or from 2 of my ESXi 4.1 servers using Veeam's FastSCP utility, the transfer seems to be limited to about 5 MBs when I can transfer the same file from the ...
5
votes
3answers
121 views

What is the difference between the download pause-able and un-pause-able files?

Technically, what is the difference between the way of hosting (on the server) of a file that can be paused while downloading and resumed again, and a file that needs to be downloaded in one go?
5
votes
3answers
11k views

Windows Server 2008 x64, Large File Transfers, and Memory Usage

I'm following up on this: http://serverfault.com/questions/37605/copying-large-file-to-remote-server-causes-it-to-run-out-of-physical-memory But figured I'd post this separately. I have a 64-bit ...
5
votes
1answer
4k views

Enable Remote Desktop File Transfer

Most of the servers I RDP to support cut-and-paste file transfer (from my Win7 64 machine). One does not, and I can't figure out what configuration step is missing. I followed the steps outlined ...
4
votes
4answers
2k views

SSH file transfers hang after ~700KB

OVERVIEW I'm scripting SFTP file transfers on a Windows 7 box in the field. When I transfer zip files larger than a megabyte or so, the transfer (and SSH) stops unexpectedly after about 700KB. The ...
4
votes
2answers
8k views

Speed up rsync by running multiple instances at once?

I have to transfer a big directory to another server. My problem is that i cannot use the full bandwidth that would be available, because one concurrent tcp stream does not get as fast. Basically ...
4
votes
4answers
1k views

Openssh sftp-server: .filepart support?

I am trying to setup a SFTP server, running off Ubuntu Server 11.04. I installed openssh-server to provide SSH access. What I am trying to do is make file uploads run with a suffix (.filepart or ...
4
votes
4answers
6k views

Multicast File Transfers

I'm looking for some multicast file transfer software. Do you have any suggestions? Must be "quickly implementable", i.e. no Tivoli or other stuff that requires a massive infrastructure. Must run on ...
4
votes
3answers
3k views

Chunking large rsync transfers?

We use rsync to update a mirror of our primary file server to an off-site colocated backup server. One of the issues we currently have is that our file server has > 1TB of mostly smaller files (in ...
4
votes
6answers
2k views

Looking for a faster way to transfer file to a windows network share

Update: This is just for one way transfer. No need to sync. I am currently using Robocopy to transfer a huge file which is zipped and send to a remote network share. Is there a faster tool? Lately, ...
4
votes
3answers
15k views

Use scp to copy a file to different servers

I have to copy a file to different servers almost every day. What I usually do is: scp filename user@destinationhost:/destination/folder I run this same command changing the destination host over ...
4
votes
6answers
1k views

How can we share large files with people outside of our firewall?

We sometimes need to share/recieve large files (sometimes in the gigabytes) to outside contractors or customers. These files are mostly CAD drawings, but could include almost anything. We want to only ...
4
votes
1answer
437 views

Out of Disk Space, /mnt/hgfs to blame?

I'm running a Centos VM in VMWare Fusion 5.x. The VM is running a file transfer server and the files being sent are stored on a disk shared from the host, /mnt/hgfs/Aspera. It looks like the file ...
4
votes
3answers
483 views

Webapp for secure file delivery to customers?

Situation: For each customer, I have a script that generates some customized configuration files for their installation (host addresses, options, organization names, etc.) I need to get these to the ...
3
votes
8answers
3k views

How do I clone the contents of one hard drive onto another?

Can anyone recommend a different CD bootable utility I might try to clone a hard disk drive? I've so far failed to perform the seemingly simple task of cloning the contents from a small hard disk ...
3
votes
6answers
439 views

Replacement for FTP?

I have some fairly large file transfers going across the atlantic every morning by ftp. From time to time the transfers will be distrupted, likely due connectivity issues, and the files will be left ...
3
votes
1answer
219 views

How does a file get sent over an HTTP connection? [closed]

A teacher of mine says when you are browsing a web server such as apache2 that has file/directory listing enabled then it secretly opens an ftp server to transfer the file. This confuses me, though ...
3
votes
5answers
838 views

What is the most efficient way to push out/publish large sets files over a network?

Here's my scenario: We have a need to push out a set of files (a 'workspace') from a source server to multiple destination servers. The files are kept under version control (SVN) so the first step is ...
3
votes
7answers
2k views

Apache HTTP server - How to determine on the server if a download finished and of which file?

We have an Apache HTTP server up and running, which serves dynamically created zip archives to the users which are possibly several 100s of megabytes in size. As we create a new file with every click ...