1

In the last week suddenly I get errors for my server. I try to access my website I get 503 errors.

When I check top and other things I see these:

  • 60-65 httpd (apache) processes. (Normally I have 30-35 httpd processes)
    You can see graph here: httpd process
  • MySQL makes ~350 MB swap. (Normally it makes 40-50 MB swap)
    Tuning primer says that:
    • Current max_connections = 250
    • Current threads_connected = 96
    • Historic max_used_connections = 97
  • Server all graphs: general data
  • Inside httpd.conf
    • ServerLimit 60
    • MaxClients 60
  • In var\www\vhosts\example.com\logs\access_log there is nothing that would seem like (D)DOS attack. I see normal server request.
  • Nothing suspicious in var\log\httpd\access_log
  • Inside my var\www\vhosts\example.com\logs\error_log I see too many errors like this:
    • [Fri Jan 17 10:56:26 2014] [warn] [client 78.180.71.157] mod_fcgid: can't apply process slot for /var/www/cgi-bin/cgi_wrapper/cgi_wrapper
  • Inside \var\log\mysqld_log no error
  • Inside \var\log\error_log no error in that time
  • Inside \var\log\suexec_log no error

How can I diagnose what causes my website to make DISK IO 100%, load average 10, very high swapping, and server doesn't respond requests. And how can I stop this to happen in future ?

When I restart mysql and httpd problem solves. But how can I diagnose the cause

  • Sounds like resource exhaustion to me...1 GB of RAM is definitely not enough for a decent-sized website. The I/O is likely caused by your server swapping to keep up. – Nathan C Jan 17 '14 at 12:42
  • I'm planning to increase RAM in future. But now I need to solve this issue. Whether I would have 3 GB RAM, this would happen due to high number of requests. Server would swap and swap and can exhaust 3 GB RAM at the end. – trante Jan 17 '14 at 12:45
  • 3GB is not all that much for a system that's both a web and a database server with that kind of web service load. Try and increase to 8GB immediately, monitor, and warn management they might need more. – MadHatter Jan 17 '14 at 13:05
  • This website has 100-150 visitors in 30 minutes time. And in normal operations server needs 30-35 httpd processes. We can't afford 8 GB for this website we don't earn that much now. Besides, increasing RAM is easy I need to find the cause that occurs suddenly. – trante Jan 17 '14 at 13:12
2

The warning mod_fcgid: can't apply process slot for /var/www/cgi-bin/cgi_wrapper/cgi_wrapper means that Apache has exhausted all resources available to it and can't serve the client. The client will get a 503 error.

From the graphs, it's either a traffic spike or a software misconfiguration that's causing your problems. When RAM requests spike, your server runs out of RAM so it peaks I/O to swap everything it can to disk (hence the 100% I/O) which causes your iowait to also increase. Once all resources are exhausted or your server is sluggish enough to time-out to Apache, it issues this warning.

Restarting likely just frees up this memory. Your answer would be one of two things: 1) increase the RAM available to your server by an actual increase or by offloading MySQL (which is a memory hog) to another server, or 2) reviewing your code to see if there's a leak.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.