Questions tagged [file-transfer]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
339 votes
18 answers
440k views

Copying a large directory tree locally? cp or rsync?

I have to copy a large directory tree, about 1.8 TB. It's all local. Out of habit I'd use rsync, however I wonder if there's much point, and if I should rather use cp. I'm worried about permissions ...
Amandasaurus's user avatar
  • 32.3k
143 votes
4 answers
447k views

robocopy transfer file and not folder

I'm trying to use robocopy to transfer a single file from one location to another but robocopy seems to think I'm always specifying a folder. Here is an example: robocopy "c:\transfer_this.txt" "z:\...
user avatar
123 votes
28 answers
239k views

How to copy a large number of files quickly between two servers

I need to transfer a huge amount of mp3s between two serves (Ubuntu). By huge I mean about a million files which are on average 300K. I tried with scp but it would have taken about a week. (about 500 ...
nicudotro's user avatar
  • 1,561
96 votes
11 answers
15k views

Transfer 10 TB of files from USA to UK datacenter

I am migrating my server from the USA to the UK from one data center to another. My host said I should be able to achieve 11 megabytes per second. The operating system is Windows Server 2008 at both ...
Paul Hinett's user avatar
  • 1,205
82 votes
12 answers
20k views

Transfer 15TB of tiny files

I'm archiving data from one server to another. Initially I started a rsync job. It took 2 weeks for it to build the file list just for 5 TB of data and another week to transfer 1 TB of data. Then I ...
lbanz's user avatar
  • 1,609
72 votes
8 answers
99k views

Quickest way to transfer 55GB of images to new server

I currently have two CentOS servers. I need to know how and what the quickest way would be to "tar" up the images directory and SCP it over? Is that the quickest way that I just suggested, because ...
Andrew Fashion's user avatar
60 votes
18 answers
30k views

Favorite rsync tips and tricks

The more I use rsync the more I realise that it's a swiss army knife of file transfer. There are so many options. I recently found out that you can go --remove-source-files and it'll delete a file ...
Amandasaurus's user avatar
  • 32.3k
59 votes
1 answer
43k views

rsync - Exclude files that are over a certain size?

I am doing a backup of my desktop to a remote machine. I'm basically doing rsync -a ~ example.com:backup/ However there are loads of large files, e.g. wikipedia dumps etc. Most of the files I care a ...
Amandasaurus's user avatar
  • 32.3k
56 votes
11 answers
288k views

Is there an equivalent to ssh-copy-id for Windows?

Is there any equivalent or port of ssh-copy-id available for Windows? That is, is there an easy way to transfer SSH keys from a local machine to a remote server under Windows? In case it helps, I'm ...
Matt V.'s user avatar
  • 837
22 votes
5 answers
78k views

What is the best way to transfer a single large file over a high-speed, high-latency WAN link?

This looks related to this one, but it's somewhat different. There is this WAN link between two company sites, and we need to transfer a single very large file (Oracle dump, ~160 GB). We've got full ...
Massimo's user avatar
  • 71.3k
22 votes
4 answers
21k views

Rsync show progress for individual file

I'm using rsync to upload/download a pretty large file (20Gb). I'm aware of --progress option, but what it does in my case is just showing me the result at the end. My problem is that during some ...
Salmon's user avatar
  • 321
19 votes
7 answers
33k views

Get list of transferred files from rsync?

I'm currently using rsync on a script that deploys a PHP application from a staging to a production server. Here is how: rsync -rzai --progress --stats --ignore-times --checksum /tmp/app_export/ root@...
Mauro's user avatar
  • 386
18 votes
2 answers
37k views

What a pros and cons of FTP vs SSH for file transfers?

Why to consider setting up an FTP service on a server when files transfers work fine by means of SSH (with Midnight Commander on a Lunux and FileZilla on a Windows client)? What are pros and cons of ...
Ivan's user avatar
  • 3,398
17 votes
4 answers
50k views

ASCII vs Binary vs Auto?

I'm transferring files between servers and just started noticing that some of them are getting modified to be one long continuous line as opposed to having the returns and line-breaks they originally ...
Sampson's user avatar
  • 530
16 votes
10 answers
35k views

Easiest way for fast transfer of files between linux servers?

I need to transfer files from one CentOS server to another. Will transfer 5MB files about every 10 minutes. Do not need encryption. What is an easy was for fast transfer of files? Is there ...
Alex L's user avatar
  • 591
15 votes
3 answers
54k views

Easy way to transfer files between host and LXC container on LVM

This is an easy task in the case of containers that share the same filesystem but I'm not sure what would be the proper approach for containers that use LVM disks. I know I could use rsync or scp but ...
s3v3n's user avatar
  • 366
15 votes
2 answers
15k views

Why is scp with compression slower than without?

I needed to transfer a 20 GB KVM vdisk file, storing the root filesystem of a CentOS 6.5 VM, from one lab server to another. The large file size and the fact that I had once compressed such a vdisk ...
pdp's user avatar
  • 778
15 votes
3 answers
21k views

Getting rsync to move file from source to destination?

Is rsync is a good choice for my project ? I have to : - copy files from source to destination folder via SSH, - be sure all files are copied, - delete source files after copy. - if I have conflict ...
user44782's user avatar
  • 265
14 votes
19 answers
28k views

What is the simplest and fastest way to transfer large file through a Windows network?

I have a Window Server 2000 machine running MS SQL Server that stores over 20GB of data. The database is backed-up every day to the second harddrive. I want to transfer those backup files to another ...
Sake's user avatar
  • 417
13 votes
6 answers
51k views

How do you limit the bandwidth for a file copy?

I've got an old windows 2000 box in a remote location with a T1 connection and a vpn to my location. I normally use smb mounts to transfer files but now it's time to decommission the server and copy ...
reconbot's user avatar
  • 2,455
13 votes
9 answers
16k views

Windows rsync that supports long file names, or an good alternative?

I currently rsync on a Linux host to copy things from Windows to my Linux box. But I can't copy files with long names. I have tried DeltaCopy, cwrsync, and cygwin. From what I have found there all ...
Zoredache's user avatar
  • 132k
12 votes
1 answer
6k views

Getting rsync to delete files from the *sender* after transfer

I'm using rsync to move one directory to another. Is there some way to get rsync to incrementally delete a file from the sender after it has copied it to the reciever?
Amandasaurus's user avatar
  • 32.3k
10 votes
7 answers
16k views

what is the fastest and most reliable way of transferring a lot of files?

I'm trying to transfer about 100k files totaling 90gb. Right now I'm using rsync daemon but its slow 3.4mb/s and I need to do this a number of times. I'm wondering what options do I have that would ...
incognito2's user avatar
9 votes
4 answers
8k views

Transfer files via SSH

I want to use file transfer via SSH on some scripts. I've read it's possible to tar over ssh. Where should I start reading?
Fernando Briano's user avatar
9 votes
4 answers
3k views

Linux to Linux, 10 TB transfer?

I've looked at all the previous similar questions, but the answers seemed to be all over the place and no one was moving a lot of data (100 GB is different from 10 TB). I've got about 10&...
lostincode's user avatar
9 votes
4 answers
63k views

Use scp to copy a file to different servers

I have to copy a file to different servers almost every day. What I usually do is: scp filename user@destinationhost:/destination/folder I run this same command changing the destination host over ...
Ruben's user avatar
  • 93
8 votes
5 answers
18k views

Should I use rsync compression over a gigabit LAN?

I normally use rsync's -z option to enable compression when transferring files over the internet. However, if I am on my own (idle) gigabit LAN, do I still want compression? Or will it be faster ...
Corey Goldberg's user avatar
8 votes
2 answers
3k views

Storage Space Write cache, and write speed

I have been working on my NAS that is using server 2012r2 with storage spaces. the drives are off the shelf 7200 rpm drives the drives are bitlocker encrypted the drives are in a 8 disk parity layout ...
Hunter Brelsford's user avatar
8 votes
2 answers
2k views

Debug slow lan (ssh, nfs) file transfer

I've got two linux boxes attached to a gigabit switch. They both have gigabit NICs, cables are cat7. Testing the network with iperf shows a fast connection but transferring files with rsync, scp, or ...
pinpox's user avatar
  • 83
7 votes
5 answers
1k views

Bash: mv directory one at a time

I am trying to move all subdirectories of a folder to another share on the same server. If I do a mv *, I will run out of space since the folders are not removed until all folders get transferred. ...
NinjaCat's user avatar
  • 576
7 votes
5 answers
12k views

Multicast File Transfers

I'm looking for some multicast file transfer software. Do you have any suggestions? Must be "quickly implementable", i.e. no Tivoli or other stuff that requires a massive infrastructure. Must run on ...
Jed Daniels's user avatar
  • 7,382
7 votes
3 answers
4k views

Copying terabytes of hundreds of thousands of files in folder slow

I am currently running FreeNAS and using SMB 3 on windows machines to copy folders with 80000+ files that are all about 35MB each. Here is the config FreeNAS 2x40Gbps connections bonded connection ...
cohortq's user avatar
  • 475
7 votes
4 answers
3k views

Puppet file transfer slow

I have a puppet master and slaves in different datacenters. The latency between them is ~40ms. When I run "puppet agent --test" on a slave to apply the latest manifest it takes ~360 seconds to finish. ...
Noodles's user avatar
  • 1,396
7 votes
1 answer
1k views

Is there a global config file for Unison synchronization tool?

I'd like all my unison profiles to have merge = Name * -> meld CURRENT1 CURRENT2 diff = /usr/bin/meld CURRENT1 CURRENT2 confirmmerge = true as options. I know I can add these options to each .prf ...
mp04's user avatar
  • 207
7 votes
3 answers
7k views

Chunking large rsync transfers?

We use rsync to update a mirror of our primary file server to an off-site colocated backup server. One of the issues we currently have is that our file server has > 1TB of mostly smaller files (in ...
Gabe Martin-Dempesy's user avatar
7 votes
3 answers
13k views

Checksumming to verify rsync transfers

So rsync runs some checksums in the course of deciding what to transfer (i.e. what blocks within a file). But is there any reason to trust the file you end up with on the receive side any more than ...
Chinasaur's user avatar
  • 263
6 votes
10 answers
5k views

Using rsync to quickly upload a file that is similar to another file

I'm putting together a deployment script which tars up a directory of my code, names the tar file after the current date and time, pushes that up to the server, untars it in a directory of the same ...
Simon Willison's user avatar
6 votes
4 answers
4k views

Transfer large amount of small files

I've to do a migration of two servers with large SAN attachments to our new VMWare environment. EDIT: I have to supply some additional intelligence as I have good answers regarding VMWare solution. ...
Dr I's user avatar
  • 975
6 votes
6 answers
2k views

Remote file copy util (like rsync) but that will take account of data already copied (in this session)?

Let's say I have a directory with 2 files, both are identical and quite large (e.g. 2GB ea.) I want to rsync that directory to a remote host. As I understand it (and I could be wrong), rsync ...
Amandasaurus's user avatar
  • 32.3k
6 votes
2 answers
7k views

Best Practice: Apache File Upload

I am looking for a soultion for trusted users to upload pdf files via html forms (with maybe php involved). This is quite a standard ubuntu linux server with apache 2.x and php 5. I am wonderiung ...
user12096's user avatar
  • 937
6 votes
3 answers
24k views

SCP hangs before File Transfer begins

I am trying to move a file from a virtual machine (Ubuntu 18.04) on my local system to a remote server using a very basic scp command. This issue is present only on one particular server, others work ...
ArunMKumar's user avatar
6 votes
3 answers
5k views

ESXi 4.1 slow file transfer on HP DL380 G5

My problem is that when I transfer files to or from 2 of my ESXi 4.1 servers using Veeam's FastSCP utility, the transfer seems to be limited to about 5 MBs when I can transfer the same file from the ...
Dennis Allen's user avatar
6 votes
3 answers
664 views

Hosted form of web-based file transfer tools [closed]

I'm looking for some software that will provide a service that resembles SendThisFile.com. My users generally turn to services like these when they are trying to send files that exceed the mail ...
Tim 's user avatar
  • 231
6 votes
1 answer
11k views

Enable Remote Desktop File Transfer

Most of the servers I RDP to support cut-and-paste file transfer (from my Win7 64 machine). One does not, and I can't figure out what configuration step is missing. I followed the steps outlined here:...
Eric J.'s user avatar
  • 762
5 votes
4 answers
7k views

SSH file transfers hang after ~700KB

OVERVIEW I'm scripting SFTP file transfers on a Windows 7 box in the field. When I transfer zip files larger than a megabyte or so, the transfer (and SSH) stops unexpectedly after about 700KB. The ...
Secure Shel's user avatar
5 votes
2 answers
305 views

Faster, secure, protocol/code required for long-distance transfer

I've ran into a problem and I'm looking for a new secure protocol/client/server that's faster over a 1Gb/s fibre link - let me tell you the story... I have a pair of redundant, diversely-routed, 1Gb/...
Chopper3's user avatar
  • 102k
5 votes
4 answers
4k views

Openssh sftp-server: .filepart support?

I am trying to setup a SFTP server, running off Ubuntu Server 11.04. I installed openssh-server to provide SSH access. What I am trying to do is make file uploads run with a suffix (.filepart or ...
The Mighty Rubber Duck's user avatar
5 votes
3 answers
50k views

How to speed up file copying onto a VMWare virtual server?

One of the sites I work at uses VMWare virtual servers. For the most part they work very well, but copying large files onto them from other places on the network is very slow. By large files I mean ...
codeulike's user avatar
  • 1,008
5 votes
2 answers
8k views

How do increase data transfer rate between my VMs

I saw lots of article,but didn't answer my question. My server use VMware vSphere Hypervisor. There are 20 virtual machines in my server. VM operating system : centos7 I already change my vSwitch ...
kevin su's user avatar
  • 163
5 votes
2 answers
13k views

When does the TCP engine decide to send an ACK?

In my LAN, I have a router that runs a Samba server and my PC connects to the router. I wiresharked during a uploading to the server and a downloading from the server. The wireshark results show ...
sliter's user avatar
  • 225

1
2 3 4 5
7