Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have 18 GB files on server for our website. I tried to take backups using cpanel backup wizards. Since, the files are growing, cpanel backup wizard is now creating corrupted zip files. So I found from Google that Filezilla would be best for transferring large files. But, when download was finished, I could not find some directories. I remember that I queued all files from root (CTRL+A) and started processing. One big zip failed, and I requeued that again.

What could be the problem for us? What are the other solutions we could follow? Cpanel backups are creating corrupted backups. Our download speed (500Kb to 1.5Mb). It would be good to know why we have corrupted backups and why we are failed and what are the other good solutions. I'm not even sure that I should contact hosting.

We are using a shared linux hosting.

share|improve this question
add comment

2 Answers 2

So there really are two questions here that need to be considered:

1) Given how large my site is how can I generate valid backups.

2) Given those large backups how do I get it off my server so it works as a backup.

The first question can be solved by doing a backup without the homedir and then making a homedir backup. The homedir is probably most of the disk space and changes a lot less frequently. The cpanel-backup will have databases and email that are important to keep current.

The second question, is how do you get it off the server. I would stay away from FTP and use Rsync. Rsync can resume a half finished download and should be available if you have SSH login access.

share|improve this answer
    
i'll contact hosting to find out those information –  Robi Dec 29 '12 at 17:01
add comment

You should

  • Ensure you are copying from a server/machine that can host big files (ie, without the common 4gb (or sometimes lower) limitations of some filesystems)

  • Ensure on both sides you use commands that can handle those big files (some tar, or gzip, or zip, etc are limited in filesize)

  • If you want to have some kind of "remote copy" of huge files, and those huge files don't change too much (or they are just growing when somethings "appends" new information at the end, for example: log files) : you should really use rsync to do the copying. Here again, depending on the OS on both sides, make sure you use the latest version possible, and be sure to understand its (sometimes tricky and "dangerous" parameters... a simple misused --delete option could wipe out entire directories on either end, if not used properly!). rsync has the bid advantage that it first see what changed in the file(s) you try to copy, and only send the changed parts over! If you go the rsyncway, it works best on the original files instead of on a compressed archive of them : rsync will more be able to see differences and similatities on the originals than on a compressed archive. And rsync can also use compression, which can be good if it's text or easily compressed, and 'bad' if it's already an heavily compressed file such as a .gz file for example.

share|improve this answer
    
and apparently your cpanel backups use "zip", which is limited in the filesize it can handle, iirc... so better switch to another compression scheme (gzip?) and, once again, make sure the underlying filesystems also can handle those sizes. –  Olivier Dulac Dec 28 '12 at 15:12
    
the limit is just underneath 4gb for "regular" zip : en.wikipedia.org/wiki/Zip_%28file_format%29#Limits –  Olivier Dulac Dec 28 '12 at 15:13
    
cpanel uses tar and gzip (its a linux product). –  n8whnp Dec 28 '12 at 15:15
    
well, he said "cpanel backup wizard is now creating corrupted zip files" ^^. Maybe an older version? or a bad parameter somewhere?. Anyawy, the other points still need to be checked too (filesystem, and archiver/unarchiver used on both sides, and also be sure that transfer is done in binary mode if they are using some "clever" programs such as a regular FTP with default on changing CRLF on-the-fly and thus creating tar files with content of different size than what the header says) –  Olivier Dulac Dec 28 '12 at 15:19
    
Someone trying to download an 18gb file with FTP probably does not know the difference between a zip and a gz ;) I have worked with cPanel as a product for 10+ years, it makes tar.gz files. –  n8whnp Dec 28 '12 at 15:26
show 6 more comments

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.