Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a lot of image files, and I want to upload all of them (about a million in a single directory) to RackSpace Cloud files in the fastest and most efficient way.

I'm using the python-cloudfiles script to upload them but it is very slow and I want to know different ways or python script code.

Probably it is very slow because it using one connection of each upload. I think to send all files in a tar and uncompress the directory is better way. but Cloud files do not support this way.

Who know any other way?

share|improve this question

Partition your upload set, e.g. into 26 sets by first letter of filename if the naming is statistically balanced enough, and use one uploader per set in parallel.

Btw, some of the cloud uploader tools have a problem with memory usage when uploading huge filesets, so keep an eye on that too...

share|improve this answer
Yes. I split files and copy upload using other server. but I guess connection limit for Rackspace cloudfiles upload server. – andy kim Jul 9 '12 at 8:16
Might be dependent on cloud provider - with a lot of them doing tens of connections is MUCH faster esp with smaller files because of protocol latencies (confirmation of store requests etc...). Are you assuming there is a connection limit (I will not discuss how to get around it in that case), or do you know there is one? – rackandboneman Jul 9 '12 at 9:51
Is ziping them and uploading them a good option? – pahnin Nov 21 '14 at 6:09

If this is a one-time upload, I like turbolift. Just make sure to reduce the concurrency to prevent a high server load (e.g. --cc 4), and use --internal to upload over Service Net.

turbolift will use lots of CPU and RAM unless you reduce concurrency (seriously: you might crash your server if you don't!). This is great if you have a powerful server, not great if you have a small server.

share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.