55gb to new hosting service; what does "unlimited storage" mean?

Question

I want to change web hosting-vendors, from "Vendor O" to "Vendor N" for a particular site I'm managing. I have problems with Vendor O. I manage several sites currently hosted with Vendor N and everything is satisfactory.

By the way: All sites I manage are non-commercial. So keeping costs minimal is vital, and there's no alternative to using low-cost shared hosting.

Both Vendor O and Vendor N advertise "unlimited storage" and the bandwidth allowances are more than enough for typical site operations.

I'm held back by the amount of data to migrate to the new hosting: 55GB and growing. (The data is mostly JPGs of historical data -- not porn.) Seems like a huge amount of data to move. (Note: all this data has been accumulating on the server for several years, and there's nothing close to an off-site copy of the entire data set.) Obviously, it is impractical to download all the data from the old server and upload it to the new one using my office DSL.

Shell access is available at both ends. I've been studying direct server-to-server transfers using rsyn, scp, and tar+wget, etc. (I've found a number of informative posts about these here on serverfault.) Using an existing Vender N account, I've run a few server-to-server trials transferring data from Vendor O. If the numbers I got are representative, 55GB is going to take many weeks to transfer, running open-throttle. Does this result seem reasonable?

I haven't found much advice about the practical aspects of server-to-server transfers. For example, is it advisable to throttle back transfers to avoid hogging the pipes -- and getting noticed? Preliminary numbers indicate attempting to be nice could extend the transfer time to months. Or do hosting services already limit bandwidth, especially for low-end accounts?

Also I'm concerned about the definition of "unlimited storage" in the vendor features lists. Vendor O hasn't complained about the 55GB data load. What's the assurance that Vendor N will be OK with that much? Reading and re-reading Vendor N's service agreement hasn't helped. It seems to be their judgment call whether a customer's use of storage space is permissible or not. If the site data weighed only, say, 1GB, I doubt they'd even look at it, but I'm guessing 50GB+ might get some scrutiny. Or are the terms-of-service typically unenforced with respect to storage use except in really outrageous mis-uses?

The nightmare I imagine is spending weeks --or months-- transferring 95% of the data to Vendor N, then getting a notice from Vendor N saying, "Your use of storage is impermissible".

Am I framing the issues correctly? Am I missing something incredibly obvious? Suggestions, please.

TIA,

hen3ry

---

Results

Thanks for all your suggestions. First, I'll respond in summary:

--Many people suggest calling the new vendor and asking them about most of these issues. My experiences with Tech support has been generally disappointing in the past, but I have learned best results occur when I have studied the problem in detail. So… yeah, now I understand exactly what to ask, I'll call.

--Some people suggest one form or another of sneaker-net. Great idea, never thought of that. Send them a disk drive. Will the new vendor be willing to offer that level of service for a new shared-hosting account? If the best electronic transfer method is hopeless slow, it's worth asking.

--Many people STRONGLY suggested that a full site backup is necessary. You're preaching to the choir. I agree wholeheartedly! Circumstances partly beyond my control...

--People suggested various approaches based on nc (netcat, synch, sshfs + sync, and so on. Most of these are not possible because the old vendor fails to support them. (Another reason to leave that vendor!) The only candidate remaining was scp, which operates over SSH. The old vendor appears to block any attempt to SSH to the new vendor, but I was able to use scp from another of my sites on the new vendor.

Results of initial test:

Using SCP from the new vendor I was able to achieve peak transfer rates of 3.7MB/s or so -- but due to frequent transfer stalls the overall rate was much less. A group of 280 files weighing about 1.4GB in total required about an hour. Roughly, that indicates the entire file set will require under 40 hours to transfer -- that's looking very good.

Avoiding transfer stalls would be very helpful in reducing the transfer time. I didn't see any pattern. Often the transfers would stall before the first block of a given file was transferred (0%) but stalls up to 30 seconds or more occurred at random sometime in mid-transfer. Any ideas? Could the old vendor be doing some throttling? Could I be competing with other customers for bandwidth?

Answer 1

You could just call them and explain the situation. More than likely they well allow you to transfer everything you need at full speed to get yourself started. You're paying money to host your data there, they should be flexible about actually getting your data there.

Answer 2

The only way to be sure of their policy is to ask them.

You also may want to consider sneakernet. Depending on where they're located, whether they would be able to perform part of the service, whether you'd be given physical access if they won't and other factors, you may be able to use an external USB drive to do the transfer. It could certainly be much faster than weeks or months. They may like it as well since it would keep that traffic off their lines. Fedex would be happy to participate, if needed.

Answer 3

Agree with the suggestions to call them.

If you end up doing the transfer over the network, consider the virtual certainty that a process which takes days or weeks will be interrupted and restarted. For that reason, I am inclined towards an approach such as rsync and away from an approach such as tar piped into netcat. If you are planning to make one big tarfile to copy, don't forget that this will approximately double your current storage usage during the transfer, on both ends, and that there's a good chance your dataset will change while the transfer is occurring, so you'll need some mechanism to update the remote copy with the changes.

Also, it sounds like you don't have a backup, which is a disaster waiting to occur. If your transfer process will create an extra physical copy of your data (like an external HDD), please consider updating that copy when your data changes. If your transfer process will occur over the net, please consider something like Amazon S3 (or Rackspace, or Azure, or whatever) both as an intermediate destination and as an ongoing backup. Low-cost hosting providers probably offer no guarantees about backing up your data - assuming they even try to do that, which may be too optimistic - and even if they do make the occasional backup, getting you up & running again may not be a big priority if they encounter some sort of tragedy.

Answer 4

If shell access is available at both ends, and if you have access to netcat (nc) the fastest way to transfer 55G of data is via netcat and tar. Here's how:

On the new server:

# nc -l -p 40022 | tar xf -

And on the current one:

# tar cf - /path/to/data | nc newserver 40022

This of course is unencrypted over the public internet, so don't do this if you are worried about the data you're tranferring.

Also, 55G using rsync over ssh (if that's available) won't be unbearably slow, and you'll get the benefit that interrupted transmissions are resumed.

Failing that, you should be able to scp it. This should be the same speed as the rsync as it uses the same transport but you'll have to restart if the connection drops.

I wouldn't worry about 55G - that's really a very small amount of data by today's standards - it won't cause any problems to your new supplier.

Answer 5

If I had 55 GB of data on a server somewhere that was at all important to me, I would use my DSL line to download it all overnight, as many nights as it takes, to make sure I have a backup that I control. Particularly if the server is with a low-cost provider. Though the provider may make backups, what happens to those backups if the provider goes out of business?

As for the provider-to-provider transfer, I'd expect speeds of at least one megabyte per second. Transferring 55 GB would take well under a day. But then you haven't told us exactly how cheap your providers are. If your new provider offers shell access, you can use wget or ftp to download the files from your old server.