I want to change web hosting-vendors, from "Vendor O" to "Vendor N" for a particular site I'm managing. I have problems with Vendor O. I manage several sites currently hosted with Vendor N and everything is satisfactory.
By the way: All sites I manage are non-commercial. So keeping costs minimal is vital, and there's no alternative to using low-cost shared hosting.
Both Vendor O and Vendor N advertise "unlimited storage" and the bandwidth allowances are more than enough for typical site operations.
I'm held back by the amount of data to migrate to the new hosting: 55GB and growing. (The data is mostly JPGs of historical data -- not porn.) Seems like a huge amount of data to move. (Note: all this data has been accumulating on the server for several years, and there's nothing close to an off-site copy of the entire data set.) Obviously, it is impractical to download all the data from the old server and upload it to the new one using my office DSL.
Shell access is available at both ends. I've been studying direct server-to-server transfers using rsyn, scp, and tar+wget, etc. (I've found a number of informative posts about these here on serverfault.) Using an existing Vender N account, I've run a few server-to-server trials transferring data from Vendor O. If the numbers I got are representative, 55GB is going to take many weeks to transfer, running open-throttle. Does this result seem reasonable?
I haven't found much advice about the practical aspects of server-to-server transfers. For example, is it advisable to throttle back transfers to avoid hogging the pipes -- and getting noticed? Preliminary numbers indicate attempting to be nice could extend the transfer time to months. Or do hosting services already limit bandwidth, especially for low-end accounts?
Also I'm concerned about the definition of "unlimited storage" in the vendor features lists. Vendor O hasn't complained about the 55GB data load. What's the assurance that Vendor N will be OK with that much? Reading and re-reading Vendor N's service agreement hasn't helped. It seems to be their judgment call whether a customer's use of storage space is permissible or not. If the site data weighed only, say, 1GB, I doubt they'd even look at it, but I'm guessing 50GB+ might get some scrutiny. Or are the terms-of-service typically unenforced with respect to storage use except in really outrageous mis-uses?
The nightmare I imagine is spending weeks --or months-- transferring 95% of the data to Vendor N, then getting a notice from Vendor N saying, "Your use of storage is impermissible".
Am I framing the issues correctly? Am I missing something incredibly obvious? Suggestions, please.
Thanks for all your suggestions. First, I'll respond in summary:
--Many people suggest calling the new vendor and asking them about most of these issues. My experiences with Tech support has been generally disappointing in the past, but I have learned best results occur when I have studied the problem in detail. So… yeah, now I understand exactly what to ask, I'll call.
--Some people suggest one form or another of sneaker-net. Great idea, never thought of that. Send them a disk drive. Will the new vendor be willing to offer that level of service for a new shared-hosting account? If the best electronic transfer method is hopeless slow, it's worth asking.
--Many people STRONGLY suggested that a full site backup is necessary. You're preaching to the choir. I agree wholeheartedly! Circumstances partly beyond my control...
--People suggested various approaches based on nc (netcat, synch, sshfs + sync, and so on. Most of these are not possible because the old vendor fails to support them. (Another reason to leave that vendor!) The only candidate remaining was scp, which operates over SSH. The old vendor appears to block any attempt to SSH to the new vendor, but I was able to use scp from another of my sites on the new vendor.
Results of initial test:
Using SCP from the new vendor I was able to achieve peak transfer rates of 3.7MB/s or so -- but due to frequent transfer stalls the overall rate was much less. A group of 280 files weighing about 1.4GB in total required about an hour. Roughly, that indicates the entire file set will require under 40 hours to transfer -- that's looking very good.
Avoiding transfer stalls would be very helpful in reducing the transfer time. I didn't see any pattern. Often the transfers would stall before the first block of a given file was transferred (0%) but stalls up to 30 seconds or more occurred at random sometime in mid-transfer. Any ideas? Could the old vendor be doing some throttling? Could I be competing with other customers for bandwidth?