0

I have recently upgraded to a new webserver. All data/config was transferred from the old server, which did its job but recently ran low on disk space.

At first I noticed an odd thing: Randomly there were sudden load spikes with htop/iotop showing absolutely nothing (1-2 running processes with CPU/RAM/IO usage < 10 %, all others having state "S"). Excerpt from my uptime log:

02:25:01 up 221 days,  4:45,  3 users,  load average: 0,20, 2,53, 3,18
 02:26:01 up 221 days,  4:46,  3 users,  load average: 0,68, 2,27, 3,05
 02:27:01 up 221 days,  4:47,  3 users,  load average: 0,74, 2,01, 2,91
 02:28:01 up 221 days,  4:48,  3 users,  load average: 0,53, 1,71, 2,75
 02:29:01 up 221 days,  4:49,  3 users,  load average: 0,30, 1,44, 2,59
 02:30:01 up 221 days,  4:50,  3 users,  load average: 0,31, 1,24, 2,44
 02:31:01 up 221 days,  4:51,  3 users,  load average: 0,80, 1,23, 2,37
 02:32:01 up 221 days,  4:52,  3 users,  load average: 0,50, 1,07, 2,24
 02:33:01 up 221 days,  4:53,  3 users,  load average: 0,52, 0,98, 2,13
 02:34:01 up 221 days,  4:54,  3 users,  load average: 0,92, 1,05, 2,08
 02:35:01 up 221 days,  4:55,  3 users,  load average: 0,51, 0,91, 1,97
 02:36:01 up 221 days,  4:56,  3 users,  load average: 48,24, 13,44, 6,13
 02:37:01 up 221 days,  4:57,  3 users,  load average: 45,14, 18,40, 8,25
 02:38:01 up 221 days,  4:58,  3 users,  load average: 16,65, 15,08, 7,74
 02:39:01 up 221 days,  4:59,  3 users,  load average: 6,15, 12,34, 7,26
 02:40:01 up 221 days,  5:00,  3 users,  load average: 2,38, 10,14, 6,82
 02:41:01 up 221 days,  5:01,  3 users,  load average: 1,78, 8,57, 6,49
 02:42:01 up 221 days,  5:02,  3 users,  load average: 0,70, 7,03, 6,08
 02:43:01 up 221 days,  5:03,  3 users,  load average: 0,40, 5,79, 5,71
 02:44:01 up 221 days,  5:04,  3 users,  load average: 0,23, 4,76, 5,36
 02:45:01 up 221 days,  5:05,  3 users,  load average: 0,17, 3,92, 5,04
 02:46:01 up 221 days,  5:06,  3 users,  load average: 0,73, 3,43, 4,80
 02:47:01 up 221 days,  5:07,  3 users,  load average: 0,58, 2,89, 4,52
 02:48:01 up 221 days,  5:08,  3 users,  load average: 0,36, 2,41, 4,25
 02:49:01 up 221 days,  5:09,  3 users,  load average: 39,40, 14,55, 8,37
 02:50:01 up 221 days,  5:10,  3 users,  load average: 14,81, 11,99, 7,87
 02:51:01 up 221 days,  5:11,  3 users,  load average: 6,20, 10,05, 7,46
 02:52:01 up 221 days,  5:12,  3 users,  load average: 2,41, 8,26, 7,01
 02:53:01 up 221 days,  5:13,  3 users,  load average: 1,20, 6,83, 6,59
 02:54:01 up 221 days,  5:14,  3 users,  load average: 0,62, 5,64, 6,20
 02:55:01 up 221 days,  5:15,  3 users,  load average: 0,42, 4,69, 5,83
 02:56:01 up 221 days,  5:16,  3 users,  load average: 0,71, 4,01, 5,53
 02:57:01 up 221 days,  5:17,  3 users,  load average: 0,37, 3,31, 5,19
 02:58:01 up 221 days,  5:18,  3 users,  load average: 0,25, 2,75, 4,88
 02:59:01 up 221 days,  5:19,  3 users,  load average: 0,52, 2,40, 4,63
 03:00:01 up 221 days,  5:20,  3 users,  load average: 0,34, 2,01, 4,35
 03:01:01 up 221 days,  5:21,  3 users,  load average: 1,66, 2,06, 4,22
 03:02:01 up 221 days,  5:22,  3 users,  load average: 1,39, 1,91, 4,03
 03:03:01 up 221 days,  5:23,  3 users,  load average: 1,26, 1,76, 3,84
 03:04:01 up 221 days,  5:24,  3 users,  load average: 0,74, 1,53, 3,63
 03:05:01 up 221 days,  5:25,  3 users,  load average: 0,60, 1,35, 3,43
 03:06:01 up 221 days,  5:26,  3 users,  load average: 1,27, 1,42, 3,33
 03:07:01 up 221 days,  5:27,  4 users,  load average: 1,13, 1,37, 3,19
 03:08:01 up 221 days,  5:28,  4 users,  load average: 0,81, 1,21, 3,02
 03:09:01 up 221 days,  5:29,  4 users,  load average: 16,35, 6,29, 4,68
 03:10:01 up 221 days,  5:30,  4 users,  load average: 12,01, 7,55, 5,26
 03:11:01 up 221 days,  5:31,  4 users,  load average: 20,01, 10,72, 6,48
 03:12:01 up 221 days,  5:32,  4 users,  load average: 8,81, 9,19, 6,22
 04:25:01 up 221 days,  6:45,  5 users,  load average: 0,20, 0,36, 0,66
 04:26:01 up 221 days,  6:46,  5 users,  load average: 0,64, 0,47, 0,68
 04:27:01 up 221 days,  6:47,  5 users,  load average: 0,47, 0,45, 0,66

This seemed to happen completely independent of actual traffic or any running cronjob. However, I could pinpoint this to Apache2, so I finally lowered the numbers for the prefork module somewhat:

<IfModule mpm_prefork_module>
  StartServers          2
  MinSpareServers       25
  MaxSpareServers       75
  MaxClients            150
  MaxRequestsPerChild   500 
</IfModule>

Finally those load spikes were gone, but load still remained somewhat higher than on the old server (going up to 3/4 even during low traffic phases, e.g. at night).

Today morning while traffic was regularly increasing I noticed that Apache has become unresponsive (getting a lot of timeouts, even localhost-to-localhost HTTP connections failed). When I checked the server everything looked normal (load below 2, Mysql and Apache processes running with low CPU/IO %).

So for no better idea, I switched everything back to the old server which currently is running OK as it used to do (load peaks around 1).

This whole thing appears very strange to me because all the scripts/database have been just mirrored to the new server. It's also a brand new Xeon server with more RAM, while the former server is an older Opteron (both having SSDs).

I just upgraded from Debian 7 to Debian 8 but largely kept the default settings for all services (except above).

Any hints/help highly welcomed!

0

Finally found the issue. When commenting out this line in apache2.conf

Mutex file:${APACHE_LOCK_DIR} default

those load spikes were finally gone.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.