Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

I have a lot of image files, and I want to upload all of them (about a million in a single directory) to RackSpace Cloud files in the fastest and most efficient way.

I'm using the python-cloudfiles script to upload them but it is very slow and I want to know different ways or python script code.

Probably it is very slow because it using one connection of each upload. I think to send all files in a tar and uncompress the directory is better way. but Cloud files do not support this way.

Who know any other way?

share|improve this question

1 Answer 1

Partition your upload set, e.g. into 26 sets by first letter of filename if the naming is statistically balanced enough, and use one uploader per set in parallel.

Btw, some of the cloud uploader tools have a problem with memory usage when uploading huge filesets, so keep an eye on that too...

share|improve this answer
    
Yes. I split files and copy upload using other server. but I guess connection limit for Rackspace cloudfiles upload server. –  andy kim Jul 9 '12 at 8:16
    
Might be dependent on cloud provider - with a lot of them doing tens of connections is MUCH faster esp with smaller files because of protocol latencies (confirmation of store requests etc...). Are you assuming there is a connection limit (I will not discuss how to get around it in that case), or do you know there is one? –  rackandboneman Jul 9 '12 at 9:51
    
Is ziping them and uploading them a good option? –  pahnin Nov 21 at 6:09

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.