My friend runs a popular Youtube-to-GIF conversion site. Right now, he has converted 250,000 Youtube videos to GIFs (each video gets 6 thumbnails for 1.5m total GIF files) and serves about 80TB of bandwidth per month.

His server is IO blocking -- I'm not a guru admin, but it seems to be the harddrive seek time for non-sequential GIFs that's clogging everything up. He has a server with 100tb.com for $300/mo, and it comes with 100TB free bandwidth. At first, I advised him to get a CDN to solve his problems, because then the GIFs get served without consuming his server resources, and his main box could just handle the encoding -- We found one CDN for $600/mo that was too slow/unreliable, and the rest wanted at least $2000/mo for 80TB of bandwidth. We're trying to keep the whole project under $900/mo, right now.

So the cheapest bandwidth we can find is with 100TB, but we're outgrowing one server. We could add another server, but I don't really know how to partition the GIF storage so that the load is distributed evenly between two boxes. Our host recommended using software like Aflexi.net, but I'm sure there must be a cheaper solution.

Can anyone help? I'm a programmer by trade, not a sysadmin, but trying to learn the ropes. Thanks!


S3 is no alternative, the bill for 80 TByte will be over 8k$ alone per month.

It looks like you serve the GIFs right out of the filesystem. Why don't you put all the GIFs on 2 machines, use a hash-algorithm mapping the name to one of the 2 machines and deliver them this way? This would easily scale to more machines as long as your loadbalancer holds up…

| improve this answer | |
  • Can you talk more about this? I like the idea, and could easily setup A.gifs.com and B.gifs.com (not our real domain) but right now, all the hotlinks are to www.gifs.com/ABC.gif -- How would your proposed system handle requests? I'm imagining something like servers A and B round robin load balancing for www. hostname, then PHP does a md5sum(REQUEST_URI) and then pulls it from /nfs/{A|B}/ABC.gif. Does this make sense? Is there any clustering software to make this easier, e.g. for when we scale up to 3 servers? – linkedlinked Jun 21 '10 at 19:40
  • actually, if there is enough space on the harddrive, than your idea with the round-robin is really good. Just add both machines to your DNS-entry and let them take turns delivering the content. no need for load-balancing or hashes and you can easily add more machines. Just make sure that the content is in sync on both machines, maybe trigger a rsync after an upload or taking care of it in php. – Malte Diedrich Jun 21 '10 at 19:49
  • Do you think I could safely partition the data between the two servers and use NFS to serve it, using the hash idea? E.g. hashmap(ABC.gif) ==> A, and hashmap(XYZ.gif) ==> B, so if a client comes to server A looking for XYZ.gif, we serve it from /NFS/serverB/XYZ.gif (but the HTTP response still comes from server A) -- Would that work? I'm mostly afraid of the NFS latency or potential bandwidth fees for transferring between servers (asking my host about this now), but otherwise I can see how it would work. – linkedlinked Jun 21 '10 at 21:46
  • Would putting a dedicated Varnish box (or another caching reverse proxy) in front of the two machines be an idea? I don't know enough about this level of scaling to know if it'd help or hinder, but it sounds like it might help to me. – WheresAlice Jun 21 '10 at 22:38
  • varnish could help if the content could be delivered out of memory, if not, than you'd have the same IO-problem again. So if some GIFs are really popular, than yeah, varnish could help. I think the NFS way would work, but don't see the advantage over round-robin via DNS. You'd just add a lot more traffic to your machines. – Malte Diedrich Jun 22 '10 at 7:58

Dump the files to S3 and serve them from there. The poor man's CDN :)

If you need more processing power, you can do the conversions out of EC2 instances and dump directly to your "CDN" as well.

| improve this answer | |
  • This is my ideal situation, but as Malte pointed out, FAR too expensive. The $300/mo we pay for an 8 core server with 12GB RAM and 100TB free is absolutely a steal. Trying to pay for the bandwidth any other way, from any other host, looks to be way above our budget at this point. – linkedlinked Jun 21 '10 at 19:36

I can't comment on the other comments, but they sound good. I would look to lift some of the load from the file servers by keeping your most commonly accessed (i.e. most popular) files in a memory cache, i.e. have a http handler that does something like this:

  1. Receive the GIF request
  2. Check if its in memory, if so, serve to client
  3. If not, get from one of the file-servers (do some round-robin here) and add to memory cache
  4. Return GIF to client

If you can get a machine with a crap-load of RAM, you're laughing as it's quite likely you'll be able to fit a large percentage of your popular files in memory.

And when you saturate that, add another image-handler server and round-robin them. Keep doing this until something breaks, i.e. throughput, scalability, economy.

I've done something like this before to good effect.

| improve this answer | |

If it's just 2 machines, you can consider using DRBD to sync between both machines. Then just use PHP to decide randomly or algorithmically which server to pull from during a request. Simple but workable solution.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.