We have 18 GB files on server for our website. I tried to take backups using cpanel backup wizards. Since, the files are growing, cpanel backup wizard is now creating corrupted zip files. So I found from Google that Filezilla would be best for transferring large files. But, when download was finished, I could not find some directories. I remember that I queued all files from root (CTRL+A) and started processing. One big zip failed, and I requeued that again.

What could be the problem for us? What are the other solutions we could follow? Cpanel backups are creating corrupted backups. Our download speed (500Kb to 1.5Mb). It would be good to know why we have corrupted backups and why we are failed and what are the other good solutions. I'm not even sure that I should contact hosting.

We are using a shared linux hosting.


So there really are two questions here that need to be considered:

1) Given how large my site is how can I generate valid backups.

2) Given those large backups how do I get it off my server so it works as a backup.

The first question can be solved by doing a backup without the homedir and then making a homedir backup. The homedir is probably most of the disk space and changes a lot less frequently. The cpanel-backup will have databases and email that are important to keep current.

The second question, is how do you get it off the server. I would stay away from FTP and use Rsync. Rsync can resume a half finished download and should be available if you have SSH login access.

  • i'll contact hosting to find out those information – Robi Dec 29 '12 at 17:01

You should

  • Ensure you are copying from a server/machine that can host big files (ie, without the common 4gb (or sometimes lower) limitations of some filesystems)

  • Ensure on both sides you use commands that can handle those big files (some tar, or gzip, or zip, etc are limited in filesize)

  • If you want to have some kind of "remote copy" of huge files, and those huge files don't change too much (or they are just growing when somethings "appends" new information at the end, for example: log files) : you should really use rsync to do the copying. Here again, depending on the OS on both sides, make sure you use the latest version possible, and be sure to understand its (sometimes tricky and "dangerous" parameters... a simple misused --delete option could wipe out entire directories on either end, if not used properly!). rsync has the bid advantage that it first see what changed in the file(s) you try to copy, and only send the changed parts over! If you go the rsyncway, it works best on the original files instead of on a compressed archive of them : rsync will more be able to see differences and similatities on the originals than on a compressed archive. And rsync can also use compression, which can be good if it's text or easily compressed, and 'bad' if it's already an heavily compressed file such as a .gz file for example.

  • and apparently your cpanel backups use "zip", which is limited in the filesize it can handle, iirc... so better switch to another compression scheme (gzip?) and, once again, make sure the underlying filesystems also can handle those sizes. – Olivier Dulac Dec 28 '12 at 15:12
  • the limit is just underneath 4gb for "regular" zip : en.wikipedia.org/wiki/Zip_%28file_format%29#Limits – Olivier Dulac Dec 28 '12 at 15:13
  • cpanel uses tar and gzip (its a linux product). – n8whnp Dec 28 '12 at 15:15
  • well, he said "cpanel backup wizard is now creating corrupted zip files" ^^. Maybe an older version? or a bad parameter somewhere?. Anyawy, the other points still need to be checked too (filesystem, and archiver/unarchiver used on both sides, and also be sure that transfer is done in binary mode if they are using some "clever" programs such as a regular FTP with default on changing CRLF on-the-fly and thus creating tar files with content of different size than what the header says) – Olivier Dulac Dec 28 '12 at 15:19
  • Someone trying to download an 18gb file with FTP probably does not know the difference between a zip and a gz ;) I have worked with cPanel as a product for 10+ years, it makes tar.gz files. – n8whnp Dec 28 '12 at 15:26

Thanks to everyone who answered the question. This is a late response and after a long time I came to my sense that I should share how I resolved our problem.

Later we decided to use rsync and it is working great for us. For corrupted backups, I was able to come to an another condition where unzipping the backups work fine on linux operating systems. They are just corrupted on windows computer. Then based on errors that popped up on unzip software and experience based on operating system, I finally figured out the script generates case sensitive files and folders on linux which are considered duplicate on windows system and the unzip software keeps warning about the duplicates.

Olivier Dulac asked me about OS where the files are being copied from/to. He is pretty right about OS difference which was the issue for me. (I am unable to reply to his comment for some reasons)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.