I'm wondering if there is an efficient way to backup files from an (S)FTP Server (Exavault), to Google Cloud Storage (GCS). We are using Exavault a managed FTP service and want to regularly back up files to GCS, then purge files older than 30 days from Exavault.

Current Implementation (Bloody Slow):

  1. Mount FTP to filesystem on Google Compute Instance using curlftpfs
  2. Run gsutil rsync to sync files from mounted storage to GCS Bucket
  3. Delete files based on mtime +30

This method is running really slow and I don't think it will be a reasonable solution at this point.

Are there any solutions that can do something like this, with ~ 500GB of data, more efficiently?

  • Are you using gsutil with the "-m" flag to parallelize it? – Brandon Yarbrough Nov 4 '16 at 0:31
  • Yes, of course. – Eric Uldall Nov 4 '16 at 0:36
  • You can try CrossFTP client to see if that improves the speed of the transfer. – Faizan Nov 8 '16 at 0:05

Multiple ftp clients, by mounting each top level directory in the system as a separate curlftpfs. Send to multiple cloud servers if maxing out the bandwidth.

Offline Media Import. Assuming you can get everyone involved to deal with physical media.

  • Not a bad idea. I found the issue was doing the rsync directly against curlftpfs, so I just mounted up a fat SSD and did a cp to disk, the gsutil rsync'd from there. Much faster. – Eric Uldall Nov 9 '16 at 23:48

Have you considered using a commercial software to accelerate your GCS backups? FileCatalyst Server has an option to mount GCS as a file system. Then you can use any FileCatalyst client application (HotFolder, CLI, Web based) to transfer files to FileCatalyst Server with GCS mounted as a virtual folder.

Here is the product link: https://filecatalyst.com/solutions/filecatalyst-direct/

Yes I work for FileCatalyst.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.