0

We are currently hosting a 3TB library of image galleries on our web server, all searchable by users. With about 400 active users at a certain time on the website, our disk reads are pushing 100% I/O utilization, and this looks like it could become a bottleneck. This is all caused by the actual image reads rather than database(which we are caching in RAM using Redis). We don't think using something like Varnish would help us, as the reads are pretty much random - we have about 7 million images, and no groups of them get significantly more reads then others as far as we can tell. We use Cloudflare's free plan, but it doesn't seem to be helping. We simply can't afford a full fledged CDN, as we push about 10TB of bandwidth a month and this would cost thousands of dollars.

3
  • 1
    SSD or a spinning disk JBOD with more spindles and higher performance disks.
    – dmourati
    Aug 7, 2014 at 16:00
  • Or more servers behind a load balancer, maybe only for the images.
    – Sven
    Aug 7, 2014 at 16:03
  • 1
    Can you edit your question to provide more specific details regarding the hardware config if your web server? Aug 7, 2014 at 16:14

1 Answer 1

0

Scale horizontally, not vertically. Instead of having one server do all of the work, setup many to do the same. They may not all host the same data, which allows a smaller set to access. Hadoop may be one way to do this. Another could be just in the way it's coded. putting a lot of little servers behind varnish would definitely be something that could help you out.

Even without fancy software, domain sharding is a way to reduce load on a single node: http://mvark.blogspot.com/2009/06/what-is-domain-sharding_17.html

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .