Some environment details first:

Intel Server Board S2600GZ
2 x Intel Xeon CPU E5-2620
Intel RAID Controller RS2BL (LSI SAS2108) with 4TB LVM Volume made of SAS disks

Ubuntu 12.04.4 LTS / Linux 3.11.0-24-generic x86_64 (with latest updates)
qemu/KVM (libvirt) with 6 VM's (running with no problems in spite of the situation)
glusterfs server 3.4.5 (seems to work normally too)
some other lightweght soft (e.g. bind9, keepalived, openvpn etc.)
NO custom / experimental / homegrown software!

For already a long time we have a very strange problem with one of our Ubuntu servers: periodically it floods the syslog with "allocation failure" messages like these:

Aug 28 07:00:18 srvname kernel: [4210234.157335] irqbalance: page allocation failure: order:4, mode:0x1040d0
Aug 28 07:00:19 srvname kernel: [4210234.711173] zabbix_agentd: page allocation failure: order:4, mode:0x1040d0
Aug 28 07:00:20 srvname kernel: [4210235.938599] zabbix_agentd: page allocation failure: order:4, mode:0x1040d0
Aug 28 07:00:34 srvname kernel: [4210250.307283] zabbix_agentd: page allocation failure: order:4, mode:0x1040d0
Aug 28 07:00:51 srvname kernel: [4210267.170359] irqbalance: page allocation failure: order:4, mode:0x1040d0
Aug 28 07:01:02 srvname kernel: [4210278.625530] zabbix_agentd: page allocation failure: order:4, mode:0x1040d0
Aug 28 07:01:19 srvname kernel: [4210295.671569] zabbix_agentd: page allocation failure: order:4, mode:0x1040d0

The messages are logged approximately every 30 secs and they do reflect the real picture: the processes shown in this log snippet are really failing (e.g. zabbix agent fails to transmit data to zabbix server). But it's just the tip of the iceberg. While this memory exhaustion is going on any process that needs to read the /proc directory (e.g. ps, top, mpstat etc.) crashes right after start because it fails to read it (/proc also can't be listed manually with ls) and this event is immediately logged to syslog with the same order 4 allocation failure error.

With all this, there's enough of free RAM (1/4 of the total size), but if we check it out by blocks - the blocks of order 4 are really exhausted. BUT, what I really can't understand is WHY these processes actually do request such large blocks? We have another, almost identical (by hardware and software) server - its blocks of order 4 are exhausted too - and it feels fine, no order 4 allocation failures! Moreover, this identical server is under MUCH heavier load.

I've searched the web deeply many times for "(high order) allocation failures" symptoms, but nothing seems to be relevant. We've tried to play with various sysctl parameters (e.g. vm.min_free_kbytes, vm.vfs_cache_pressure etc., as suggested by some articles), but nothing helps. Eventually we're rolled back all these changes and now most of sysctl settings are system default. We've also tried echoing to /proc/sys/vm/compact_memory and /proc/sys/vm/drop_caches without any obvious (or prolonged) effect. After some long period of exhaustion, all of a sudden and by itself, everything becomes normal (it seems like memory gets defragmented and order 4 blocks become available, /proc becomes available too), but not for long - after some short period all starts over again. A reboot helps for a longer period (due to the totally unfragmented memory), but eventually everything ends up the same...

In general, the only real troubles (that we're aware of) caused by the described behavior is the inablity to monitor and manage server resources, neither remotely (zabbix), nor locally (ps, top, mpstat etc.).

As far as I understand, the lack of order 4 blocks is a normal regular state of memory under Linux. It's just that the processes shouldn't generally request such blocks (especially processes that DON'T do it on other servers). If anyone has any idea about what could be the cause of such a behavior, what can we check or where to dig - we would really appreciate it! We're ready to supply any additional info on demand.

  • 1
    I just can't see running Ubuntu on bare metal hardware because of issues like this.
    – ewwhite
    Aug 30, 2014 at 1:37

2 Answers 2


https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1319244 suggests it's a kernel bug, and there's a fix for Trusty which has only very recently been released. Sorry I can't solve the problem right now though (affects me also, exactly the same behaviour).

  • Seems like this bug could be the cause of this problem, but as far as I understand there's no way for me to confirm it. Thanks for the link, though, somehow I've missed it myself. Sep 1, 2014 at 3:57

Are you sure this is not a hardware problem? If I were you, I would suspect RAM. Try running memtest or similar.

  • We've thought about it, but frankly none of the web searches on this topic has ever suggested that it could be related to a hardware problem. We've also updated the BIOS just in case (with no effect). This server is part of a group serving a pretty high-loaded web-portal - it's unacceptable to take it down for a period required to memtest its 64GB of RAM (without being warranted it helps or reveals something). Aug 28, 2014 at 7:00
  • Compare the configuration files between servers with diff or some configuration/version management tool you might have in use. If there are no differences but one of the hardware-wise identical servers does work and the other one does not, then you most likely have a HW problem. Linux tends to work predictably if the environment is the same. It's up to you to decide if it's better to have unplanned downtime or a scheduled maintenance window where you try to poke the potential hardware problem. Aug 28, 2014 at 7:29
  • The two servers are almost identical: OS on the 'good' one was installed long before the 'bad' one and hasn't since been updated. Thus versions of software differ (including the kernel), so I cannot fully exclude software influence. Though we've tried to apply the full sysctl dump from 'good' to 'bad' one - it didn't help. I suspect hardware myself, but I have no idea how to identify the particular peace to blame (it could be RAM or mobo, or even RAID controller) and since the server is in production mode there's unfortunately no room for various experiments... Aug 28, 2014 at 9:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .