Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 1 down vote favorite
2

I got an Apache webserver running in a EC2 platform. It's receiving a lot of uploads from users (big pictures files).

The problem is that immediately after upload I have to convert images to other sizes, because they have to be used in the "next page".

I chose ImageMagick, since I noticed even Flickr is using it (http://highscalability.com/flickr-architecture), and it takes about 4 seconds to generate all photo sizes.

The problem is when I have 50 concurrent users or more. I've done stress tests, and the server starts to get slower.

So, any idea how to tune Apache for massive uploads?

Any idea how to do everything faster?

share|improve this question

4 Answers

up vote 2 down vote

From the sounds of it your problem isn't apache, but rather the load on the server from trying to do concurrent image processing jobs. In light of that, no matter how you tune apache, you're still going to get worse performance as more and more users use your service.

First suggestion would be to offload the images to a cluster of machines for dedicated image processing. It's far more scalable than bulking up your web server to try and handle large scale image processing.

share|improve this answer
The 95% of the used CPU in that "stressed" moments, was because ImageMagick ... so if I move the process to other server, I will not save much time, but I will lose in network data transfer ... – user75317 Apr 8 '11 at 0:33
The issue is regarding scalability. If you have no intention of processing large datasets because you don't have the user base yet, then it's not a big deal to see spikes in your usage... after all they aren't simulating your normal load on the machine. If, however, you are looking to make your server respond quickly under even heavy usage, you need to think about moving the load-intensive processes off-site. The load hit isn't a big deal now, but it's your issue in the "stress test." I guarantee you Flickr has a cluster of machines distinct from their web servers doing their processing. – photoionized Apr 8 '11 at 0:44
Not to mention that the time involved in transferring files between 2 machines on EC2 should be incredibly negligible. – photoionized Apr 8 '11 at 0:46
I tend to get at least 5Mb/s transfer rates between ec2 instances for multi Gb scp. Farming image processing off to other machines seems to be the solution here. – Decado Apr 8 '11 at 6:50
up vote 1 down vote

We had the similar problem and figured out to improve our performance by re-compiling Imagemagick with --disable-openmp option which cause high-tension to CPU due to internal threads (unused/unwanted).

After searching for answers, I landed in this following thread which directed me to disable the above function, I am sure it will improve your performance in creating the images.

http://www.imagemagick.org/discourse-server/viewtopic.php?f=1&t=14307

share|improve this answer
up vote 0 down vote

Get more CPU power, more memory (RAM) - image resizing is a computationally and often memory intensive operation, if you are doing too much of it, you need more powerful hardware specifications.

If you are performing multiple resizes on a single image, try and ensure you do them in decreasing order of size, so you can perform later resizes using smaller source images (providing this gives acceptable quality).

share|improve this answer
Hi. I'm with medium instances at Amazon ... good enough for them I guess. The resizing is done in that way and I even spent a lot of time with ImageMagick parameters to do it in the faster way ... – user75317 Apr 8 '11 at 0:20
@user75317: Good, well then - if it is still slowing down, then you may well need more compute units, have you looked at usage statistics? You may well need a high-cpu instance type. – Orbling Apr 8 '11 at 0:28
up vote 0 down vote

Maybe you should recompile ImageMagick to use only Q8, 8-bit channel-depth. If JPEG files are used, this could lead to serious performance gain, because less memory will be used.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.