Tell me more ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.
up vote 5 down vote favorite

I am running a high traffic website on a Dedicated-Virtual 16GB RAM CentOS server hosted by Media Temple.
Very often the server dies because of high CPU usage (%1500)
When i check the running processes using "top" command i see a too many httpd processes running which i think are causing the high usage of CPU.
I have tried to tune mysql, tune apache, my.cnf, httpd.cnf, removed Apache modules that i do not use but that does not solve the problem.
Please let me know what values do you need to know in my server config in order to help me diagnosis the problem.
Thanks in advance.

share|improve this question
you can't have 1500% CPU without at least 15 processors - how are you determining this? – xaxxon Oct 23 '11 at 7:35
1500% as per of Media Template control panel status – sunn Oct 23 '11 at 7:52
Here is the top 15 lines of 'top' command pastebin.com/5c16JN4G – sunn Oct 23 '11 at 7:56
1  
How many is "too many"? On a 16GB box you might be able to run a thousand httpd instances without causing trouble if it doesn't do much else. – ceejayoz Oct 23 '11 at 18:46
1  
Can you access the Apache server-status page to see what all those httpds are doing? Is anything showing up in your error log? – MarkR Oct 24 '11 at 13:45
show 2 more comments

migrated from stackoverflow.com Oct 23 '11 at 18:22

4 Answers

up vote 5 down vote

what does tail -f /var/log/messages say when you experience high system loads?

Are your processes very I/O or very CPU intensive?

One observation:

Looks like you have 18G physical memory, but zero swap space..., e.g. your top command shows "Swap: 0k total" ... that means that you do not have any Swap Space configured.

You should always have a sufficient swap space on a UNIX system! Swap-size = 1...2-times RAM size is a good idea. Using a fast partition is a good idea. Really bad things happen if your UNIX system runs out of RAM and doesn't have Swap .. processes just die inexplicably.. that is a bad very thing! especially in production. Disk is cheap! add a generous swap partition! :-)

On any UNIX system, you do need swap space -- because that is where processes live while they are not scheduled on one of the CPUs. If you don't have swap space, really bad things will happen, because your system will run out of places (RAM) where to create new processes -- it can not put them in swap, because there is none, so you will see extremely high loads, the system freezing up, and processes die seemingly inexplicably.

To check your memory and swap space in use, you can run free -k. You can also check the swap configuration by running swapon -s .. this will not show any output if no swap is configured.

As a rule of thumb, configure Swap Space approximately 1..2 times the size of your physical memory. I'd say if you have 18G RAM, configure ~20..30G Swap -- don't be stingy with the Swap, because "disk is dirt-cheap"!

Two choices for creating swap space:

  • either create a swap partition, preferably on your fastest disk partition (best choice; or add a flash drive to your physical server)

  • or create a swap file (if you don't have disk space left to partition; this is also a good option if you later find out that you need more swap than you anticipated)

...then format that swap partition or swap file to make it usable for swapping with the mkswap command.

Once you enable the swap space with sudo swapon -a, you should see it listed in top and with swapon -s , and your system should behave much more nicely..

Articles about Swap Space:

https://www.linux.com/news/software/applications/8208-all-about-linux-swap-space

https://www.linux.com/learn/tutorials/442430-increase-your-available-swap-space-with-a-swap-file

http://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/

http://www.thegeekstuff.com/2010/08/how-to-add-swap-space/

http://lissot.net/partition/partition-08.html

http://tldp.org/LDP/sag/html/swap-space.html

Some pointers:

man -k swap

man mkswap

man fstab , vim /etc/fstab

EDIT

as this question was migrated to ServerFault .. I'll add a comment here, because ppl are complaining that I quote the old rule: " 1..2-times RAM size == Swap size "

A 2TB SATA drive costs around $70 these days, that means 20GB costs you about 70 cents -- it's dirt cheap! I know 20GB sounds a lot, but think of it as a very inexpensive "worst case insurance"!

Somebody said: "if your system filled up 18GB and is swapping onto 20GB, then you have bigger problems"

Certainly true, but having generous Swap Space if unexpected things go wrong, will save your server and the processes on it from crashing completely! You might still be able to log in, examine the system while it's still running (slowly), and repair things (albeit slowly), and not have to reboot.. On a production server, there is really no reason not to use generous swap.

If I had a server with 18GB RAM and it's running MongoDB for example (which is very RAM intensive) .. I would certainly configure very generous Swap Space, perhaps even 40..60GB ... just in case something unexpected goes wrong..

share|improve this answer
Here is my top pastebin.com/GSzDhZ22 – sunn Oct 23 '11 at 8:03
1  
"As a rule of thumb, configure Swap Space approximately 1..2 times the size of your physical memory." .. this was true about ten years ago. Today, if you have 20 GB of swap in active use, your server probably consumes more time in transferring stuff from and to swap, and has less time for actual work. – Janne Pikkarainen Oct 24 '11 at 9:01
Great answer, and generally true, but it should be clear that this remains a generalization. There are some environments where you can safely, through various controls, deploy a unix-based server and not have any swap. I've done this many times, though admittedly never in an environment where a daemon like apache httpd was running, as it spawns itself to take on new requests which is more challenging to manage. – sandroid Oct 24 '11 at 13:44
@sandroid : yes, certainly true! I just wanted to point out to user sunn that disk space is ridiculously cheap, and there's really no good reason to not configure swap in a regular server or desktop system -- even if it's 20GB (which sounds a lot, but it's dirt cheap). – Tilo Oct 25 '11 at 0:57
@JannePikkarainen : yes, you're absolutely right! The 1..2 times rule is old .. but think about how dirt cheap disk space is -- a 2TB disk costs ~$70 these days .. .that's 70 cents for the 20GB! There's no good reason to not add swap -- even if it's 20GB -- it's sort of a "worst-case insurance" when unexpected things happen.. – Tilo Oct 25 '11 at 1:01
show 1 more comment
up vote 0 down vote

20GB - 30GB of swap space very high. If you are swapping 20GB in Linux you have bigger problems.

share|improve this answer
Except the top information shows zero swap-space even configured. I'm not sure where you're getting 20GB of it. – sysadmin1138♦ Oct 24 '11 at 11:36
1  
"As a rule of thumb, configure Swap Space approximately 1..2 times the size of your physical memory. I'd say if you have 18G RAM, configure ~20..30G Swap " – Virulence Oct 24 '11 at 14:29
20GB of swap costs you about 70 cents these days.. there is really no reason to be stingy with it – Tilo Oct 25 '11 at 1:02
up vote 0 down vote

Almost certainly your machine is running out of memory because you have too many "fat" Apache processes doing little or no work.

Apache normally uses a "prefork" model. PHP normally runs in-process. This can be problematic, because of:

  • PHP causes the apache child processes to use lots of memory each.
  • Serving static files over slow connections, keeps child processes in use for a long time for no reason
  • HTTP keepalives, if turned on, also keep child processes in use.

A "bad" solution would be to turn off keepalives (or maybe reduce the keepalive time). Note that keepalives are VERY good for performance, particularly on HTTPS.

A "better" solution would be to switch to a better architecture. A better architecture would be one where you don't waste "fat" processes doing nothing. A typical implementation of a better architecture, would be Nginx + PHP/FastCGI.

I'm not necessarily advocating the use of Nginx - you could probably do it with Apache too, if you run a different MPM (Multiprocessing module e.g. worker, event, not prefork) and move PHP out-of-process with FastCGI.

share|improve this answer
up vote 0 down vote

In your case you do not need swap. If you want you could create 1-2GB swap partition or file to have more physical memory for file system cache.

I do not understand why cached memory and buffers memory is 0. Can you paste the output of cat /proc/meminfo ?

You have very high I/O load. This can be disk or network. Try to reduce the load. For disk mount the data partition with noatime. Try to find out which process is most I/O intensive. Useiotopanddstat` for this.

Try to upgrade the firmware and the divers (kernel modules) for the disk controller and for the network card. If you are running a VM make sure you are installing the paravirtualization drivers (e.g. VMware Tools). If you are running a 32bit kernel with PAE, try to switch to a 64bit distribution (you will reduce from the kernel time).

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.