4

On my way to the office this morning, every website on our shared VPS started giving the same error (several times, not the typical memory_limit error which is fatal):

Warning: Unknown: Unable to allocate memory for pool. in Unknown on line 0

The shared server is a 64-bit OpenVZ container running cPanel. There are only ~6 VPSes on the host-- this is the largest one at only 4GB. The host itself has 24GB RAM. As the below graphs show, the memory usage on the host and VPS are both rather low. CPU Usage/Disk/Host all seem to be normal. RlimitMem was set to 583653034, yet the memory usage is about the same as it usually is.

Apache 2.2, PHP 5.2 (mod_php)

Restarting Apache has corrected the problem for now. However, I'd like to prevent it from happening again and I'm not sure what was limiting the memory. RlimitMem was set to 583653034, yet the memory usage is about the same as it usually is. There's seems to be plenty of memory: what caused this error?

VPS Memory Usage

shared vm memory usage, hovering around 50%

Host Memory Usage

host vm memory usage, used hovering at 20%, cached at 65% until 6AM, where it dropped to ~60%, buffered at ~10%

APC Information

 apc.ttl=0
 apc.shm_size=0
 apc.mmap_file_mask=(blank)

1 Segment(s) with 32.0 MBytes (mmap memory, pthread mutex locking)

4 Answers 4

6

That is definitely the error you get when APC runs out of memory. When I (re)build servers, I often forget to increase this value to 128 M (suitable for my application) and that is the exact error you see.

7
  • Why doesn't the error come up sooner? When I upped the value it filled the used memory reported by apc.php quickly exceeded the default max value.
    – Reece45
    Feb 18, 2011 at 20:09
  • @AlR: You'd have to ask the developer or look at the code yourself. It wouldn't surprise me that APC leaks memory. There is an open bug report regarding this. The work around is what I already told you, increase your APC cache size.
    – hobodave
    Feb 18, 2011 at 21:03
  • @AlR: Regarding the delay you are describing. The error doesn't occur simply because APC "fills" it's allocated memory. APC actually handles that by deleting older items from the cache and inserting the new one. Over time it gets wonky and can no longer do this. I'm assuming there is a memory leak.
    – hobodave
    Feb 18, 2011 at 22:05
  • Does the workaround prevent the issue from coming up at all, or does it just make it more rare? If it doesn't come up at all, is it because the 128M doesn't fill up?
    – Reece45
    Feb 19, 2011 at 11:48
  • @AlR: It doesn't come up at all. At least not in the 7-21 days that go between us bouncing Apache. I see that you aren't using the user cache, just the opcode cache. There's really no good reason not to size your cache such that all of your PHP files can fit into it. With a 32M cache and your files taking up at least 200M you're really beating the hell out of APC with all the deletion/insertions that must be going on.
    – hobodave
    Feb 19, 2011 at 20:23
1

Any failcounts in /proc/bc/resources?

All failcounts should be 0 or stay same since the last incident.

You need to:

  1. Increase the resource limits with vzct set <CTID> ... --save on the resources that have fail counts (see man vzctl the set section). You can also modify the resource limits directly in /etc/vz/conf/. Probably in all cases you need to reboot the containers after increasing the limits.

    To be safe, increase the settings (both barrier and limit) for problematic resources to x2 (twice) the maxheld.

  2. Write down the current fail counts and keep an eye on them so that they don't increase any more.

For more info on controlling various resources, you can use http://wiki.openvz.org/Resource_shortage as a starting point.

3
  • For another container (the 4GB one described above is 105, the problematic one is 189)
    – Reece45
    Feb 19, 2011 at 11:44
  • @Reece, 105 and 189 - what resource is that for? privvmpages? Feb 21, 2011 at 22:34
  • Sorry I wasn't clear before. VEID 105 is the container I described above. According to /proc/bc/resources, VEID 105 had no failcounts. VEID is another container, it did have failcounts under it. While I don't think this was the problem, it is something I was suspecting. Thanks for the info (+1)!
    – Reece45
    Feb 22, 2011 at 11:10
0

Can you tell us a little more about how subscribed the host is? (e.g. how much total RAM have you committed to the VMs, how much RAM to each of the VMs on the host) I understand your largest is 4GB, but if the others are all 3GB and the system is subscribed to 19GB, that should show 5GB memory free for the hosting OS. Either my math, assumptions, or something else doesn't gel with the host memory graph above (showing 94% occupied RAM).

4
  • 5.5GB alloted through the VMs. Because OpenVZ is more containers than VMs, the file-cache is shared between them. The graph shows that 63.3% of the RAM is allocated to cache (which frees up immediately if needed), and 9% to buffers (There's actually a investigation in process to find out why it is so high (2GB) underway). In reality, its about only 3GB used (see serverfault.com/questions/9442/…)
    – Reece45
    Feb 18, 2011 at 17:28
  • Are you using APC with PHP? This may be related to stackoverflow.com/questions/3723316/… Feb 18, 2011 at 17:32
  • Sorry, it's 16GB, I failed at my math :). None of the utilise more than 50% (only 2 exceed 600MB usage)
    – Reece45
    Feb 18, 2011 at 17:34
  • APC is installed, but it doesn't quite explain the problem. ttl was already set to 0, shm_size was only 32M (matched the value of kernel..shmmax). Bringing it up to 512M, it doesn't take long to bring that up to >32M (200M right now, after visiting a few of the websites). There are no user entries. Because of the speed that this memory is used up, wouldn't a "Unable to allocate memory for pool" show up more quickly?
    – Reece45
    Feb 18, 2011 at 18:56
0

Shared memory allocation sysctl limits? Have a look at shmmax and friends in /etc/sysctl.conf.

Alternatively, are you running 32-bit PHP and/or Apache on a 64-bit OS? We've seen bizarre 'oh my god I'm out of memory and I'm gonna fill up swap even through there's 16GB free physical memory' behaviour from high-memory-usage 32-bit apps on 64-bit Linux.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .