Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Many of the links that connect NY and AMS are often saturated. That means that running a transfer over them (e.g., move 300GB at 1MB/s) would take an age if compared with what our connections can offer.

I came across to the problem in the past, like 3 years ago, when i was really newbie to coding and linux, i came to my conclusion that i will post on the bottom of this post. However, it's dirty and i don't like it. The script dosen't work as it is, since that it was written for a very specific environment, however it give you the idea.

My question is, do you know of anything better alternatives to transfer files across the ocean in a fast way?

#!/bin/sh

upto="$1"
filepath="$2"
remotepath="$3"

if [ ! -f ${filepath} ]
then
exit 0
fi

password=$(/all/script/password 10)
filesize=$(du -b ${2} | sed 's/\([0-9]*\)\(.*\)/\1/')

if [ $filesize -gt 5368709120 ]; then
parts="80"
elif [ $filesize -gt 2147483648 ]; then
parts="50"
elif [ $filesize -gt 1310720 ]; then
parts="20"
else
parts="2"
fi

splitsize=$(($filesize / $parts))

split -b "$splitsize" -a 2 "$2" /all/tmp/cup/${password}_

#UPLOAD
declare -a pwait
for tmpfile in /all/tmp/cup/${password}_*
do
    scp ${tmpfile} root@${upto}.domain.com:/all/tmp/cup/ &
        array_lenght=${#pwait[@]}
        pwait[${array_lenght}]=$!
done

#ATTENDERE
for prid in ${pwait[@]}
do
wait $prid
done

#UNISCI FILE REMOTO
ssh root@${upto}.domain.com "cat /all/tmp/cup/${password}_* > ${remotepath} && wait && rm -f /all/tmp/cup/${password}_*"

#RIMUOVI ROBA DI TROPPO LA
#eval ssh root@${upto}.domain.com rm -f /all/tmp/cup/${password}*

#REMOVE HERE
rm -f /all/tmp/cup/${password}_*

exit 0
share|improve this question
1  
Don't make people go somewhere else for more details; this is not hunt-'n-sniff. –  adaptr Sep 29 '12 at 17:51
    
Does it need to be secure or not? –  sonassi Sep 29 '12 at 18:13
1  
I really don't know what you're talking about, not only can I buy entirely uncontended multi-gigabit links from any major European city to NYC but its estimated that there's between 8 and 12 times more available bandwidth between Europe and the US than there is demand. If you have financial constraints then please let us know what they are. –  Chopper3 Sep 29 '12 at 19:21
    
i'm not in that business, i simply rent servers here and there, they all came with this restrictions, no matter how much i pay them. Sigle threaded connections are slow. –  cedivad Sep 29 '12 at 20:10
    
Based on that last comment I am going to assume that you are looking for a way to bypass the bandwidth restrictions imposed on your hosting accounts. How do you imagine opening multiple streams is going to help you? Such restrictions apply to the total bandwidth, not bandwidth per connection. –  John Gardeniers Sep 30 '12 at 4:01
show 4 more comments

2 Answers

Assuming that your network are not saturated (contrary to what you're stating in the question), you should be tuning your link to deal with the (comparatively) high bandwidth delay product like Andrew mentioned. (The articles referenced at that link include some info on what to tweak, when, and why.)


If in fact your network links ARE saturated (moving the maximum amount of data they can) the only solution is to add more bandwidth (either more fiber trunks between the two sites, paying another carrier for transit to offload some of the peak period traffic, or if you're using "dedicated" links paying for a higher CIR/adding more circuits to the loop).


How can you tell the difference?
Well, if starting more streams gets you more speed you haven't saturated your link. You're probably getting hit by the relatively long round-trip time from the US to Europe (as compared to the round-trip time on a local network).
(There's a point of diminishing returns here as the overhead for more TCP connections will eventually cause other bottlenecks to show up.)

If adding more streams provides no net increase in speed (two streams run at half the net speed of one) your link is saturated, and you need to add bandwidth to improve performance.


Other stuff to consider

You should seek to minimize the data being pushed over the pipe, using rsync or similar protocols if appropriate (rsync works best with small-ish change sets to large-ish collections of data).

Never underestimate the bandwidth of a FedEx overnight package with a couple of hard disks in it. Especially for initial syncs.

share|improve this answer
add comment

I would check the TCP/IP tuning options, for example window scaling, retransmission, the routing table as well icmp. If this is all working OK, and the networking stack on the OS is not Windows XP or Centos 5 or anything older than Vista, you should be OK the way that multi-threaded network connection is not required. Or, it would not improve better than 20%, so in fact, it would just defragment the filesystem and slow it down even more.

share|improve this answer
    
I'm using freebsd over SSDs. Tons of SSDs. I don't have any disk bottleneck. If i setup the server in the EU i don't have any problems at all. Should i try to tune the TCP/IP protocol even in this case? I've never done it. Thank you. –  cedivad Sep 30 '12 at 8:15
    
Yes, one connection should be OK. –  Andrew Smith Oct 1 '12 at 11:11
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.