Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I've inherited an Ubuntu Server that is running an instance of Red5 which is storing the streams in a single directory. The streams directory has grown to over 3 million files, and the drive is reaching it's capacity. I've been asked to delete files that are over a certain date, and according to a database record lookup which I have written the script for. I realized before I ran the script that there is absolutely no backup for these streams and I am of course hesitant to run my script before I make a back up.

I do not have physical access to this server unfortunately, so I need to FTP the files. From another system, I connected via FTP and attempted to download the files via FTP to another machine using Filezilla client. The problem is Filezilla's last response is "150 Here comes the directory listing" but it hangs there for hours before it finally crashes Filezilla. It appears to be waiting for the LIST so it can build up the Queue I guess.

I tried to ssh into the server and a simple mput * in simple ftp, which works but its only uploading 1 file at a time which is going to take forever... I was hoping to use something like Filezilla so I could set multiple connections and download several at a time.

Is there an alternative that I can use here to send these files to another server? The other server isnt on the same network by the way, but the connection between is very fast.

Any help would be greatly appreciated. Thanks!

share|improve this question
1  
rsync maybe? However later you might want to look at some of the other questions about storing millions of files, a single mega folder is never going to perform well –  TheLQ Mar 23 '12 at 14:51
    
Yes we are redeveloping the application to not follow this storage scheme. –  Sharbel Mar 23 '12 at 17:48
    
You don't do backups??? –  Grant Oct 10 '12 at 17:52
add comment

3 Answers 3

Some suggestions:

  • Try rsync.
  • You could try lftp instead of ftp. It can handle multiple connections.
  • In Your deletion-script replace "rm" with "echo" and check the output.
  • Maybe, You could use FXP, i.e. transferring the files directly between ftp servers. However, all servers I know have this feature disabled for security reasons.
share|improve this answer
    
Thanks for the reply. rsync really isn't an option as the ftp I am uploading to does not support it and I cannot install anything. lftp excited me when I read about it but it hangs too unfortunately.. –  Sharbel Mar 23 '12 at 17:45
add comment

How compressible are the files? Could you Tar and Gzip the files first then transfer them?

share|improve this answer
    
Well the files are FLV files so they aren't going to compress much but remember I am running low on HD space so I would run out of space while the compressed file grows.. –  Sharbel Mar 23 '12 at 17:47
add comment

How aout using an external HDD and FedEx?

The hosting company should be able to plug in an external drive if you don't have physical access?

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.