Questions tagged [file-transfer]
The file-transfer tag has no usage guidance.
314
questions
339
votes
18
answers
440k
views
Copying a large directory tree locally? cp or rsync?
I have to copy a large directory tree, about 1.8 TB. It's all local. Out of habit I'd use rsync, however I wonder if there's much point, and if I should rather use cp.
I'm worried about permissions ...
143
votes
4
answers
447k
views
robocopy transfer file and not folder
I'm trying to use robocopy to transfer a single file from one location to another but robocopy seems to think I'm always specifying a folder. Here is an example:
robocopy "c:\transfer_this.txt" "z:\...
123
votes
28
answers
239k
views
How to copy a large number of files quickly between two servers
I need to transfer a huge amount of mp3s between two serves (Ubuntu).
By huge I mean about a million files which are on average 300K.
I tried with scp but it would have taken about a week. (about 500 ...
96
votes
11
answers
15k
views
Transfer 10 TB of files from USA to UK datacenter
I am migrating my server from the USA to the UK from one data center to another. My host said I should be able to achieve 11 megabytes per second.
The operating system is Windows Server 2008 at both ...
82
votes
12
answers
20k
views
Transfer 15TB of tiny files
I'm archiving data from one server to another. Initially I started a rsync job. It took 2 weeks for it to build the file list just for 5 TB of data and another week to transfer 1 TB of data.
Then I ...
72
votes
8
answers
99k
views
Quickest way to transfer 55GB of images to new server
I currently have two CentOS servers. I need to know how and what the quickest way would be to "tar" up the images directory and SCP it over?
Is that the quickest way that I just suggested, because ...
60
votes
18
answers
30k
views
Favorite rsync tips and tricks
The more I use rsync the more I realise that it's a swiss army knife of file transfer. There are so many options. I recently found out that you can go --remove-source-files and it'll delete a file ...
59
votes
1
answer
43k
views
rsync - Exclude files that are over a certain size?
I am doing a backup of my desktop to a remote machine. I'm basically doing rsync -a ~ example.com:backup/ However there are loads of large files, e.g. wikipedia dumps etc. Most of the files I care a ...
56
votes
11
answers
288k
views
Is there an equivalent to ssh-copy-id for Windows?
Is there any equivalent or port of ssh-copy-id available for Windows? That is, is there an easy way to transfer SSH keys from a local machine to a remote server under Windows?
In case it helps, I'm ...
22
votes
5
answers
78k
views
What is the best way to transfer a single large file over a high-speed, high-latency WAN link?
This looks related to this one, but it's somewhat different.
There is this WAN link between two company sites, and we need to transfer a single very large file (Oracle dump, ~160 GB).
We've got full ...
22
votes
4
answers
21k
views
Rsync show progress for individual file
I'm using rsync to upload/download a pretty large file (20Gb). I'm aware of --progress option, but what it does in my case is just showing me the result at the end.
My problem is that during some ...
19
votes
7
answers
33k
views
Get list of transferred files from rsync?
I'm currently using rsync on a script that deploys a PHP application from a staging to a production server. Here is how:
rsync -rzai --progress --stats --ignore-times --checksum /tmp/app_export/ root@...
18
votes
2
answers
37k
views
What a pros and cons of FTP vs SSH for file transfers?
Why to consider setting up an FTP service on a server when files transfers work fine by means of SSH (with Midnight Commander on a Lunux and FileZilla on a Windows client)? What are pros and cons of ...
17
votes
4
answers
50k
views
ASCII vs Binary vs Auto?
I'm transferring files between servers and just started noticing that some of them are getting modified to be one long continuous line as opposed to having the returns and line-breaks they originally ...
16
votes
10
answers
35k
views
Easiest way for fast transfer of files between linux servers?
I need to transfer files from one CentOS server to another.
Will transfer 5MB files about every 10 minutes.
Do not need encryption.
What is an easy was for fast transfer of files?
Is there ...
15
votes
3
answers
54k
views
Easy way to transfer files between host and LXC container on LVM
This is an easy task in the case of containers that share the same filesystem but I'm not sure what would be the proper approach for containers that use LVM disks.
I know I could use rsync or scp but ...
15
votes
2
answers
15k
views
Why is scp with compression slower than without?
I needed to transfer a 20 GB KVM vdisk file, storing the root filesystem of a CentOS 6.5 VM, from one lab server to another. The large file size and the fact that I had once compressed such a vdisk ...
15
votes
3
answers
21k
views
Getting rsync to move file from source to destination?
Is rsync is a good choice for my project ?
I have to :
- copy files from source to destination folder via SSH,
- be sure all files are copied,
- delete source files after copy.
- if I have conflict ...
14
votes
19
answers
28k
views
What is the simplest and fastest way to transfer large file through a Windows network?
I have a Window Server 2000 machine running MS SQL Server that stores over 20GB of data. The database is backed-up every day to the second harddrive. I want to transfer those backup files to another ...
13
votes
6
answers
51k
views
How do you limit the bandwidth for a file copy?
I've got an old windows 2000 box in a remote location with a T1 connection and a vpn to my location. I normally use smb mounts to transfer files but now it's time to decommission the server and copy ...
13
votes
9
answers
16k
views
Windows rsync that supports long file names, or an good alternative?
I currently rsync on a Linux host to copy things from Windows to my Linux box. But I can't copy files with long names.
I have tried DeltaCopy, cwrsync, and cygwin. From what I have found there all ...
12
votes
1
answer
6k
views
Getting rsync to delete files from the *sender* after transfer
I'm using rsync to move one directory to another. Is there some way to get rsync to incrementally delete a file from the sender after it has copied it to the reciever?
10
votes
7
answers
16k
views
what is the fastest and most reliable way of transferring a lot of files?
I'm trying to transfer about 100k files totaling 90gb. Right now I'm using rsync daemon but its slow 3.4mb/s and I need to do this a number of times. I'm wondering what options do I have that would ...
9
votes
4
answers
8k
views
Transfer files via SSH
I want to use file transfer via SSH on some scripts. I've read it's possible to tar over ssh. Where should I start reading?
9
votes
4
answers
3k
views
Linux to Linux, 10 TB transfer?
I've looked at all the previous similar questions, but the answers seemed to be all over the place and no one was moving a lot of data (100 GB is different from 10 TB).
I've got about 10&...
9
votes
4
answers
63k
views
Use scp to copy a file to different servers
I have to copy a file to different servers almost every day. What I usually do is:
scp filename user@destinationhost:/destination/folder
I run this same command changing the destination host over ...
8
votes
5
answers
18k
views
Should I use rsync compression over a gigabit LAN?
I normally use rsync's -z option to enable compression when transferring files over the internet. However, if I am on my own (idle) gigabit LAN, do I still want compression? Or will it be faster ...
8
votes
2
answers
3k
views
Storage Space Write cache, and write speed
I have been working on my NAS that is using server 2012r2 with storage spaces.
the drives are off the shelf 7200 rpm drives
the drives are bitlocker encrypted
the drives are in a 8 disk parity layout
...
8
votes
2
answers
2k
views
Debug slow lan (ssh, nfs) file transfer
I've got two linux boxes attached to a gigabit switch. They both have gigabit NICs, cables are cat7.
Testing the network with iperf shows a fast connection but transferring files with rsync, scp, or ...
7
votes
5
answers
1k
views
Bash: mv directory one at a time
I am trying to move all subdirectories of a folder to another share on the same server. If I do a mv *, I will run out of space since the folders are not removed until all folders get transferred. ...
7
votes
5
answers
12k
views
Multicast File Transfers
I'm looking for some multicast file transfer software. Do you have any suggestions? Must be "quickly implementable", i.e. no Tivoli or other stuff that requires a massive infrastructure. Must run on ...
7
votes
3
answers
4k
views
Copying terabytes of hundreds of thousands of files in folder slow
I am currently running FreeNAS and using SMB 3 on windows machines to copy folders with 80000+ files that are all about 35MB each. Here is the config
FreeNAS
2x40Gbps connections bonded
connection ...
7
votes
4
answers
3k
views
Puppet file transfer slow
I have a puppet master and slaves in different datacenters. The latency between them is ~40ms. When I run "puppet agent --test" on a slave to apply the latest manifest it takes ~360 seconds to finish.
...
7
votes
1
answer
1k
views
Is there a global config file for Unison synchronization tool?
I'd like all my unison profiles to have
merge = Name * -> meld CURRENT1 CURRENT2
diff = /usr/bin/meld CURRENT1 CURRENT2
confirmmerge = true
as options. I know I can add these options to each .prf ...
7
votes
3
answers
7k
views
Chunking large rsync transfers?
We use rsync to update a mirror of our primary file server to an off-site colocated backup server. One of the issues we currently have is that our file server has > 1TB of mostly smaller files (in ...
7
votes
3
answers
13k
views
Checksumming to verify rsync transfers
So rsync runs some checksums in the course of deciding what to transfer (i.e. what blocks within a file). But is there any reason to trust the file you end up with on the receive side any more than ...
6
votes
10
answers
5k
views
Using rsync to quickly upload a file that is similar to another file
I'm putting together a deployment script which tars up a directory of my code, names the tar file after the current date and time, pushes that up to the server, untars it in a directory of the same ...
6
votes
4
answers
4k
views
Transfer large amount of small files
I've to do a migration of two servers with large SAN attachments to our new VMWare environment.
EDIT: I have to supply some additional intelligence as I have good answers regarding VMWare solution.
...
6
votes
6
answers
2k
views
Remote file copy util (like rsync) but that will take account of data already copied (in this session)?
Let's say I have a directory with 2 files, both are identical and quite large (e.g. 2GB ea.) I want to rsync that directory to a remote host. As I understand it (and I could be wrong), rsync ...
6
votes
2
answers
7k
views
Best Practice: Apache File Upload
I am looking for a soultion for trusted users to upload pdf files via html forms (with maybe php involved). This is quite a standard ubuntu linux server with apache 2.x and php 5.
I am wonderiung ...
6
votes
3
answers
24k
views
SCP hangs before File Transfer begins
I am trying to move a file from a virtual machine (Ubuntu 18.04) on my local system to a remote server using a very basic scp command. This issue is present only on one particular server, others work ...
6
votes
3
answers
5k
views
ESXi 4.1 slow file transfer on HP DL380 G5
My problem is that when I transfer files to or from 2 of my ESXi 4.1 servers using Veeam's FastSCP utility, the transfer seems to be limited to about 5 MBs when I can transfer the same file from the ...
6
votes
3
answers
664
views
Hosted form of web-based file transfer tools [closed]
I'm looking for some software that will provide a service that resembles SendThisFile.com. My users generally turn to services like these when they are trying to send files that exceed the mail ...
6
votes
1
answer
11k
views
Enable Remote Desktop File Transfer
Most of the servers I RDP to support cut-and-paste file transfer (from my Win7 64 machine). One does not, and I can't figure out what configuration step is missing.
I followed the steps outlined here:...
5
votes
4
answers
7k
views
SSH file transfers hang after ~700KB
OVERVIEW
I'm scripting SFTP file transfers on a Windows 7 box in the field. When I transfer zip files larger than a megabyte or so, the transfer (and SSH) stops unexpectedly after about 700KB. The ...
5
votes
2
answers
305
views
Faster, secure, protocol/code required for long-distance transfer
I've ran into a problem and I'm looking for a new secure protocol/client/server that's faster over a 1Gb/s fibre link - let me tell you the story...
I have a pair of redundant, diversely-routed, 1Gb/...
5
votes
4
answers
4k
views
Openssh sftp-server: .filepart support?
I am trying to setup a SFTP server, running off Ubuntu Server 11.04. I installed openssh-server to provide SSH access. What I am trying to do is make file uploads run with a suffix (.filepart or ...
5
votes
3
answers
50k
views
How to speed up file copying onto a VMWare virtual server?
One of the sites I work at uses VMWare virtual servers. For the most part they work very well, but copying large files onto them from other places on the network is very slow.
By large files I mean ...
5
votes
2
answers
8k
views
How do increase data transfer rate between my VMs
I saw lots of article,but didn't answer my question.
My server use VMware vSphere Hypervisor.
There are 20 virtual machines in my server.
VM operating system : centos7
I already change my vSwitch ...
5
votes
2
answers
13k
views
When does the TCP engine decide to send an ACK?
In my LAN, I have a router that runs a Samba server and my PC connects to the router.
I wiresharked during a uploading to the server and a downloading from the server.
The wireshark results show ...