0

I've inherited an Ubuntu Server that is running an instance of Red5 which is storing the streams in a single directory. The streams directory has grown to over 3 million files, and the drive is reaching it's capacity. I've been asked to delete files that are over a certain date, and according to a database record lookup which I have written the script for. I realized before I ran the script that there is absolutely no backup for these streams and I am of course hesitant to run my script before I make a back up.

I do not have physical access to this server unfortunately, so I need to FTP the files. From another system, I connected via FTP and attempted to download the files via FTP to another machine using Filezilla client. The problem is Filezilla's last response is "150 Here comes the directory listing" but it hangs there for hours before it finally crashes Filezilla. It appears to be waiting for the LIST so it can build up the Queue I guess.

I tried to ssh into the server and a simple mput * in simple ftp, which works but its only uploading 1 file at a time which is going to take forever... I was hoping to use something like Filezilla so I could set multiple connections and download several at a time.

Is there an alternative that I can use here to send these files to another server? The other server isnt on the same network by the way, but the connection between is very fast.

Any help would be greatly appreciated. Thanks!

3
  • 1
    rsync maybe? However later you might want to look at some of the other questions about storing millions of files, a single mega folder is never going to perform well
    – TheLQ
    Mar 23, 2012 at 14:51
  • Yes we are redeveloping the application to not follow this storage scheme.
    – Sharbel
    Mar 23, 2012 at 17:48
  • You don't do backups???
    – Grant
    Oct 10, 2012 at 17:52

3 Answers 3

2

Some suggestions:

  • Try rsync.
  • You could try lftp instead of ftp. It can handle multiple connections.
  • In Your deletion-script replace "rm" with "echo" and check the output.
  • Maybe, You could use FXP, i.e. transferring the files directly between ftp servers. However, all servers I know have this feature disabled for security reasons.
1
  • Thanks for the reply. rsync really isn't an option as the ftp I am uploading to does not support it and I cannot install anything. lftp excited me when I read about it but it hangs too unfortunately..
    – Sharbel
    Mar 23, 2012 at 17:45
0

How compressible are the files? Could you Tar and Gzip the files first then transfer them?

1
  • Well the files are FLV files so they aren't going to compress much but remember I am running low on HD space so I would run out of space while the compressed file grows..
    – Sharbel
    Mar 23, 2012 at 17:47
0

How aout using an external HDD and FedEx?

The hosting company should be able to plug in an external drive if you don't have physical access?

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .