Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We are currently hosting a 3TB library of image galleries on our web server, all searchable by users. With about 400 active users at a certain time on the website, our disk reads are pushing 100% I/O utilization, and this looks like it could become a bottleneck. This is all caused by the actual image reads rather than database(which we are caching in RAM using Redis). We don't think using something like Varnish would help us, as the reads are pretty much random - we have about 7 million images, and no groups of them get significantly more reads then others as far as we can tell. We use Cloudflare's free plan, but it doesn't seem to be helping. We simply can't afford a full fledged CDN, as we push about 10TB of bandwidth a month and this would cost thousands of dollars.

share|improve this question
1  
SSD or a spinning disk JBOD with more spindles and higher performance disks. –  dmourati Aug 7 at 16:00
    
Or more servers behind a load balancer, maybe only for the images. –  Sven Aug 7 at 16:03
1  
Can you edit your question to provide more specific details regarding the hardware config if your web server? –  Twisty Aug 7 at 16:14

1 Answer 1

Scale horizontally, not vertically. Instead of having one server do all of the work, setup many to do the same. They may not all host the same data, which allows a smaller set to access. Hadoop may be one way to do this. Another could be just in the way it's coded. putting a lot of little servers behind varnish would definitely be something that could help you out.

Even without fancy software, domain sharding is a way to reduce load on a single node: http://mvark.blogspot.com/2009/06/what-is-domain-sharding_17.html

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.