0

After ordering and setting up a new web server (64bit, 4 vCores, 6GB RAM, Ubuntu 14.04) with Apache 2.4.7 (and PHP7 fpm) as event MPM but during busier traffic hours Apache crashes. The error log is filled with hundreds of entries (even during less busier hours) like that:

[Thu Jul 21 17:51:16.633617 2016] [mpm_event:crit] [pid 9290:tid 139705492948732] (22)Invalid argument: ap_queue_pop failed
[Thu Jul 21 17:51:16.633622 2016] [mpm_event:crit] [pid 9290:tid 139705492948732] (22)Invalid argument: ap_queue_pop failed
[Thu Jul 21 17:51:16.633627 2016] [mpm_event:crit] [pid 9290:tid 139705492948732] (22)Invalid argument: ap_queue_pop failed
[Thu Jul 21 17:51:18.632095 2016] [mpm_event:alert] [pid 9313:tid 139705857640190] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread
[Thu Jul 21 17:51:19.632842 2016] [mpm_event:alert] [pid 9337:tid 139705857640190] (11)Resource temporarily unavailable: apr_thread_create: unable to create worker thread

I already have by default high system limits via ulimit -a:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1546681
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1546681
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

The blog entry from Mozilla didn't help either as my limit is already way higher than they suggest. I also read to increase the privvmpages value but it's high too:

$ cat /proc/user_beancounters
Version: 2.5
       uid  resource                     held              maxheld              barrier                limit              failcnt
   130922:  kmemsize                173862701            175349760  9223372036854775807  9223372036854775807                    0
            lockedpages                     0                    0              1572864              1572864                    0
            privvmpages               3327626              3330707              3670016              3670016                31705
            shmpages                    92638                92638  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numproc                       279                  363                  850                  850                    0
            physpages                  356751               363453                    0              1572864                    0
            vmguarpages                     0                    0              1835008              1835008                    0
            oomguarpages               121370               121370              1572864              1572864                    0
            numtcpsock                     69                   72  9223372036854775807  9223372036854775807                    0
            numflock                       11                   12  9223372036854775807  9223372036854775807                    0
            numpty                          2                    2  9223372036854775807  9223372036854775807                    0
            numsiginfo                      0                   63  9223372036854775807  9223372036854775807                    0
            tcpsndbuf                 1368640              1445072  9223372036854775807  9223372036854775807                    0
            tcprcvbuf                 1938112              1987264  9223372036854775807  9223372036854775807                    0
            othersockbuf               233512               241968  9223372036854775807  9223372036854775807                    0
            dgramrcvbuf                     0                    0  9223372036854775807  9223372036854775807                    0
            numothersock                  148                  150  9223372036854775807  9223372036854775807                    0
            dcachesize               61545508             61876256  9223372036854775807  9223372036854775807                    0
            numfile                      2133                 2280  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numiptent                     338                  338  9223372036854775807  9223372036854775807                    0

$ free -m 
             total       used       free     shared    buffers     cached
Mem:          6144       1823       4320        361          0       1553
-/+ buffers/cache:        270       5873
Swap:         1024          0       1024

The site gets totally up to 500.000 requests over the day but it was running on an way older (Software/Hardware) and smaller server (half the resources) before without any (resource) issues. My mpm-event settings:

<IfModule mpm_event_module>
    StartServers             5
    ServerLimit              32
    MinSpareThreads          60
    MaxSpareThreads          200
    ThreadLimit              64
    ThreadsPerChild          25
    MaxRequestWorkers        800
    MaxConnectionsPerChild   20000
</IfModule>

The load is usually around 0.05 and the RAM isn't even used more than 10% often. Any idea what I am doing wrong?

(Small note: It's a service people use for free and I built in my free time.)

5
  • Is there anyone who could help?
    – DevDavid
    Commented Jul 27, 2016 at 19:49
  • Did you ever solve this? I am seeing the same issue, details here: serverfault.com/questions/817593/… Commented Dec 1, 2016 at 12:54
  • 1
    My server hoster did some changes to the server (which they didn't explain in detail) in the system limits which fixed that error.
    – DevDavid
    Commented Dec 1, 2016 at 16:16
  • Thanks! If you are able, could you compare the current limits vs. those posted above? Might give me something to work on. Much appreciated. :) Commented Dec 1, 2016 at 16:26
  • 1
    Yes, here's the diff: diffchecker.com/R1RDmEqs Also I suggested that as solution in your thread.
    – DevDavid
    Commented Dec 1, 2016 at 18:30

0

You must log in to answer this question.

Browse other questions tagged .