0

I have a lot of image files, and I want to upload all of them (about a million in a single directory) to RackSpace Cloud files in the fastest and most efficient way.

I'm using the python-cloudfiles script to upload them but it is very slow and I want to know different ways or python script code.

Probably it is very slow because it using one connection of each upload. I think to send all files in a tar and uncompress the directory is better way. but Cloud files do not support this way.

Who know any other way?

Improve this question

2 Answers 2

Reset to default
1

Partition your upload set, e.g. into 26 sets by first letter of filename if the naming is statistically balanced enough, and use one uploader per set in parallel.

Btw, some of the cloud uploader tools have a problem with memory usage when uploading huge filesets, so keep an eye on that too...

Improve this answer
3
  • Yes. I split files and copy upload using other server. but I guess connection limit for Rackspace cloudfiles upload server.
    – andy kim
    Jul 9, 2012 at 8:16
  • Might be dependent on cloud provider - with a lot of them doing tens of connections is MUCH faster esp with smaller files because of protocol latencies (confirmation of store requests etc...). Are you assuming there is a connection limit (I will not discuss how to get around it in that case), or do you know there is one?
    – rackandboneman
    Jul 9, 2012 at 9:51
  • Is ziping them and uploading them a good option?
    – pahnin
    Nov 21, 2014 at 6:09
0

If this is a one-time upload, I like turbolift. Just make sure to reduce the concurrency to prevent a high server load (e.g. --cc 4), and use --internal to upload over Service Net.

turbolift will use lots of CPU and RAM unless you reduce concurrency (seriously: you might crash your server if you don't!). This is great if you have a powerful server, not great if you have a small server.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .