The tag has no wiki summary.

learn more… | top users | synonyms

76
votes
12answers
8k views

Transfer 10TB of files from USA -> UK datacenter

I am migrating my server from the USA to the UK from one data center to another, my host said I should be able to achieve 11 megabytes per second. Operating system is Windows Server 2008 at both ...
76
votes
15answers
78k views

Copying a large directory tree locally? cp or rsync?

I have to copy a large directory tree, about 1.8 TB. It's all local. Out of habit I'd use rsync, however I wonder if there's much point, and if I should rather use cp. I'm worried about permissions ...
43
votes
23answers
64k views

How to copy a large number of files quickly between two servers

I need to transfer a huge amount of mp3s between two serves (Ubuntu). By huge I mean about a million files which are on average 300K. I tried with scp but it would have taken about a week. (about 500 ...
41
votes
15answers
9k views

Favorite rsync tips and tricks

The more I use rsync the more I realise that it's a swiss army knife of file transfer. There are so many options. I recently found out that you can go --remove-source-files and it'll delete a file ...
27
votes
2answers
53k views

robocopy transfer file and not folder

I'm trying to use robocopy to tranfer a single file from one location to another but robocopy seems to think I'm always specifying a folder. Here is an example: robocopy "c:\transfer_this.txt" ...
24
votes
8answers
12k views

Quickest way to transfer 55GB of images to new server

I currently have two CentOS servers. I need to know how and what the quickest way would be to "tar" up the images directory and SCP it over? Is that the quickest way that I just suggested, because ...
21
votes
5answers
28k views

What is the best way to transfer a single large file over a high-speed, high-latency WAN link?

This looks related to this one, but it's somewhat different. There is this WAN link between two company sites, and we need to transfer a single very large file (Oracle dump, ~160 GB). We've got full ...
14
votes
9answers
23k views

Easiest way for fast transfer of files between linux servers?

I need to transfer files from one CentOS server to another. Will transfer 5MB files about every 10 minutes. Do not need encryption. What is an easy was for fast transfer of files? Is there ...
14
votes
6answers
8k views

Is there an equivalent to ssh-copy-id for Windows?

Is there any equivalent or port of ssh-copy-id available for Windows? That is, is there an easy way to transfer SSH keys from a local machine to a remote server under Windows? In case it helps, I'm ...
14
votes
3answers
17k views

Getting rsync to move file from source to destination?

Is rsync is a good choice for my project ? I have to : - copy files from source to destination folder via SSH, - be sure all files are copied, - delete source files after copy. - if I have conflict ...
13
votes
1answer
7k views

rsync - Exclude files that are over a certain size?

I am doing a backup of my desktop to a remote machine. I'm basically doing rsync -a ~ example.com:backup/ However there are loads of large files, e.g. wikipedia dumps etc. Most of the files I care a ...
13
votes
9answers
9k views

Windows rsync that supports long file names, or an good alternative?

I currently rsync on a Linux host to copy things from Windows to my Linux box. But I can't copy files with long names. I have tried DeltaCopy, cwrsync, and cygwin. From what I have found there all ...
12
votes
23answers
15k views

What is the simplest and fastest way to transfer large file through a Windows network?

I have a Window Server 2000 machine running MS SQL Server that stores over 20GB of data. The database is backed-up every day to the second harddrive. I want to transfer those backup files to another ...
9
votes
1answer
3k views

Getting rsync to delete files from the *sender* after transfer

I'm using rsync to move one directory to another. Is there some way to get rsync to incrementally delete a file from the sender after it has copied it to the reciever?
8
votes
4answers
6k views

Transfer files via SSH

I want to use file transfer via SSH on some scripts. I've read it's possible to tar over ssh. Where should I start reading?
8
votes
6answers
15k views

How do you limit the bandwidth for a file copy?

I've got an old windows 2000 box in a remote location with a T1 connection and a vpn to my location. I normally use smb mounts to transfer files but now it's time to decommission the server and copy ...
7
votes
4answers
14k views

ASCII vs Binary vs Auto?

I'm transferring files between servers and just started noticing that some of them are getting modified to be one long continuous line as opposed to having the returns and line-breaks they originally ...
7
votes
6answers
5k views

what is the fastest and most reliable way of transferring a lot of files?

I'm trying to transfer about 100k files totaling 90gb. Right now I'm using rsync daemon but its slow 3.4mb/s and I need to do this a number of times. I'm wondering what options do I have that would ...
7
votes
3answers
936 views

linux to linux, 10TB transfer?

I've looked at all the previous similar questions, but the answers seemed to be all over the place and no one was moving a lot of data (100GB != 10TB). I've got about 10TB that I need to move from ...
7
votes
4answers
2k views

Rsync show progress for individual file

I'm using rsync to upload/download a pretty large file (20Gb). I'm aware of --progress option, but what it does in my case is just showing me the result at the end. My problem is that during some ...
7
votes
2answers
9k views

What a pros and cons of FTP vs SSH for file transfers?

Why to consider setting up an FTP service on a server when files transfers work fine by means of SSH (with Midnight Commander on a Lunux and FileZilla on a Windows client)? What are pros and cons of ...
6
votes
4answers
1k views

Puppet file transfer slow

I have a puppet master and slaves in different datacenters. The latency between them is ~40ms. When I run "puppet agent --test" on a slave to apply the latest manifest it takes ~360 seconds to finish. ...
6
votes
6answers
1k views

Remote file copy util (like rsync) but that will take account of data already copied (in this session)?

Let's say I have a directory with 2 files, both are identical and quite large (e.g. 2GB ea.) I want to rsync that directory to a remote host. As I understand it (and I could be wrong), rsync ...
6
votes
3answers
614 views

Hosted form of web-based file transfer tools [closed]

I'm looking for some software that will provide a service that resembles SendThisFile.com. My users generally turn to services like these when they are trying to send files that exceed the mail ...
5
votes
5answers
404 views

Bash: mv directory one at a time

I am trying to move all subdirectories of a folder to another share on the same server. If I do a mv *, I will run out of space since the folders are not removed until all folders get transferred. ...
5
votes
2answers
225 views

Faster, secure, protocol/code required for long-distance transfer

I've ran into a problem and I'm looking for a new secure protocol/client/server that's faster over a 1Gb/s fibre link - let me tell you the story... I have a pair of redundant, diversely-routed, ...
5
votes
2answers
2k views

Best Practice: Apache File Upload

I am looking for a soultion for trusted users to upload pdf files via html forms (with maybe php involved). This is quite a standard ubuntu linux server with apache 2.x and php 5. I am wonderiung ...
5
votes
3answers
9k views

Windows Server 2008 x64, Large File Transfers, and Memory Usage

I'm following up on this: http://serverfault.com/questions/37605/copying-large-file-to-remote-server-causes-it-to-run-out-of-physical-memory But figured I'd post this separately. I have a 64-bit ...
5
votes
1answer
2k views

Enable Remote Desktop File Transfer

Most of the servers I RDP to support cut-and-paste file transfer (from my Win7 64 machine). One does not, and I can't figure out what configuration step is missing. I followed the steps outlined ...
4
votes
6answers
653 views

Effect of distance from server on page load speed

I measured the load speed of a static image file severed by nginx from my server (using pingdom service). The server should normally process this request within few seconds. My server is located in ...
4
votes
10answers
4k views

Using rsync to quickly upload a file that is similar to another file

I'm putting together a deployment script which tars up a directory of my code, names the tar file after the current date and time, pushes that up to the server, untars it in a directory of the same ...
4
votes
2answers
5k views

Speed up rsync by running multiple instances at once?

I have to transfer a big directory to another server. My problem is that i cannot use the full bandwidth that would be available, because one concurrent tcp stream does not get as fast. Basically ...
4
votes
4answers
3k views

Multicast File Transfers

I'm looking for some multicast file transfer software. Do you have any suggestions? Must be "quickly implementable", i.e. no Tivoli or other stuff that requires a massive infrastructure. Must run on ...
4
votes
6answers
2k views

Looking for a faster way to transfer file to a windows network share

Update: This is just for one way transfer. No need to sync. I am currently using Robocopy to transfer a huge file which is zipped and send to a remote network share. Is there a faster tool? Lately, ...
4
votes
3answers
2k views

Chunking large rsync transfers?

We use rsync to update a mirror of our primary file server to an off-site colocated backup server. One of the issues we currently have is that our file server has > 1TB of mostly smaller files (in ...
4
votes
6answers
658 views

How can we share large files with people outside of our firewall?

We sometimes need to share/recieve large files (sometimes in the gigabytes) to outside contractors or customers. These files are mostly CAD drawings, but could include almost anything. We want to only ...
4
votes
1answer
2k views

ESXi 5.5, slow network speed when writing to guests

This is an issue that has been bugging me for a lot time with ESXi. I had the issue with my old Dell server (ESXi 5.0 and 5.5) and even with my new Dell T20 (ESXi 5.5). For the current issue at hand I ...
4
votes
1answer
294 views

Out of Disk Space, /mnt/hgfs to blame?

I'm running a Centos VM in VMWare Fusion 5.x. The VM is running a file transfer server and the files being sent are stored on a disk shared from the host, /mnt/hgfs/Aspera. It looks like the file ...
4
votes
3answers
4k views

ESXi 4.1 slow file transfer

My problem is that when I transfer files to or from 2 of my ESXi 4.1 servers using Veeam's FastSCP utility, the transfer seems to be limited to about 5 MBs when I can transfer the same file from the ...
4
votes
3answers
456 views

Webapp for secure file delivery to customers?

Situation: For each customer, I have a script that generates some customized configuration files for their installation (host addresses, options, organization names, etc.) I need to get these to the ...
3
votes
8answers
3k views

How do I clone the contents of one hard drive onto another?

Can anyone recommend a different CD bootable utility I might try to clone a hard disk drive? I've so far failed to perform the seemingly simple task of cloning the contents from a small hard disk ...
3
votes
6answers
419 views

Replacement for FTP?

I have some fairly large file transfers going across the atlantic every morning by ftp. From time to time the transfers will be distrupted, likely due connectivity issues, and the files will be left ...
3
votes
1answer
184 views

How does a file get sent over an HTTP connection? [closed]

A teacher of mine says when you are browsing a web server such as apache2 that has file/directory listing enabled then it secretly opens an ftp server to transfer the file. This confuses me, though ...
3
votes
5answers
642 views

What is the most efficient way to push out/publish large sets files over a network?

Here's my scenario: We have a need to push out a set of files (a 'workspace') from a source server to multiple destination servers. The files are kept under version control (SVN) so the first step is ...
3
votes
4answers
296 views

Moving file structure in Active Directory

I have recently installed a second Domain Controller in our Domain. Because our old file server is about to die from old age and dust. So my dire need is the knowledge on how to move the file ...
3
votes
4answers
364 views

SSH file transfers hang after ~700KB

OVERVIEW I'm scripting SFTP file transfers on a Windows 7 box in the field. When I transfer zip files larger than a megabyte or so, the transfer (and SSH) stops unexpectedly after about 700KB. The ...
3
votes
5answers
1k views

secure ftp to specific directory on Ubuntu?

can I set up a secure ftp for somebody (not a current user on the system) for 2 different directories on Ubuntu inside public_html? I don't want to give the user pure ssh account right now. Any ...
3
votes
2answers
360 views

Secure FTP/SCP “breaks” existing files while they are being transfered

Typically I upload updated files/scripts over Secure FTP to my server to replace existing copies of those files as I make changes. But this has the nasty behavior of writing directly to those files. ...
3
votes
4answers
1k views

10Gbps iSCSI network transferring at only 2Gbps

I have a performance issue with my SAN. I have a full 10Gbps private network setup. I have the following setup 1 - Mac with Connected with via Fiber Transciever 10GbE 1 - Windows 7 PC with 10GbE 1 - ...
3
votes
2answers
434 views

Sharing large (multi-Gb) files with clients

I wasn't sure if this was the best place for this question, but I think it is squarely in the realm of the IT admin so that's the reason I put it here. We need to share large files (several ...