Many of the links that connect NY and AMS are often saturated. That means that running a transfer over them (e.g., move 300GB at 1MB/s) would take an age if compared with what our connections can offer.
I came across to the problem in the past, like 3 years ago, when i was really newbie to coding and linux, i came to my conclusion that i will post on the bottom of this post. However, it's dirty and i don't like it. The script dosen't work as it is, since that it was written for a very specific environment, however it give you the idea.
My question is, do you know of anything better alternatives to transfer files across the ocean in a fast way?
#!/bin/sh
upto="$1"
filepath="$2"
remotepath="$3"
if [ ! -f ${filepath} ]
then
exit 0
fi
password=$(/all/script/password 10)
filesize=$(du -b ${2} | sed 's/\([0-9]*\)\(.*\)/\1/')
if [ $filesize -gt 5368709120 ]; then
parts="80"
elif [ $filesize -gt 2147483648 ]; then
parts="50"
elif [ $filesize -gt 1310720 ]; then
parts="20"
else
parts="2"
fi
splitsize=$(($filesize / $parts))
split -b "$splitsize" -a 2 "$2" /all/tmp/cup/${password}_
#UPLOAD
declare -a pwait
for tmpfile in /all/tmp/cup/${password}_*
do
scp ${tmpfile} root@${upto}.domain.com:/all/tmp/cup/ &
array_lenght=${#pwait[@]}
pwait[${array_lenght}]=$!
done
#ATTENDERE
for prid in ${pwait[@]}
do
wait $prid
done
#UNISCI FILE REMOTO
ssh root@${upto}.domain.com "cat /all/tmp/cup/${password}_* > ${remotepath} && wait && rm -f /all/tmp/cup/${password}_*"
#RIMUOVI ROBA DI TROPPO LA
#eval ssh root@${upto}.domain.com rm -f /all/tmp/cup/${password}*
#REMOVE HERE
rm -f /all/tmp/cup/${password}_*
exit 0