Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Check the logs:

    httpd: page allocation failure. order:4, mode:0xd0

Call Trace:
[<ffffffff8000f696>] __alloc_pages+0x2ef/0x308
[<ffffffff80017a40>] cache_grow+0x139/0x3c7
[<ffffffff8005bd54>] cache_alloc_refill+0x138/0x188
[<ffffffff800df358>] __kmalloc+0x95/0x9f
[<ffffffff800f0bba>] alloc_fd_array+0x18/0x28
[<ffffffff800306c8>] expand_files+0x124/0x2cb
[<ffffffff800226c5>] dup_fd+0x133/0x281
[<ffffffff80049050>] copy_files+0x47/0x64
[<ffffffff8001f632>] copy_process+0x590/0x15db
[<ffffffff800a11c4>] alloc_pid+0x228/0x2c4
[<ffffffff80030f6c>] do_fork+0x69/0x1c1
[<ffffffff8005d116>] system_call+0x7e/0x83
[<ffffffff8005d427>] ptregscall_common+0x67/0xac

Mem-info:
Node 0 DMA per-cpu:
cpu 0 hot: high 0, batch 1 used:0
cpu 0 cold: high 0, batch 1 used:0
cpu 1 hot: high 0, batch 1 used:0
cpu 1 cold: high 0, batch 1 used:0
cpu 2 hot: high 0, batch 1 used:0
cpu 2 cold: high 0, batch 1 used:0
cpu 3 hot: high 0, batch 1 used:0
cpu 3 cold: high 0, batch 1 used:0
cpu 4 hot: high 0, batch 1 used:0
cpu 4 cold: high 0, batch 1 used:0
cpu 5 hot: high 0, batch 1 used:0
cpu 5 cold: high 0, batch 1 used:0
cpu 6 hot: high 0, batch 1 used:0
cpu 6 cold: high 0, batch 1 used:0
cpu 7 hot: high 0, batch 1 used:0
cpu 7 cold: high 0, batch 1 used:0
Node 0 DMA32 per-cpu:
cpu 0 hot: high 186, batch 31 used:14
cpu 0 cold: high 62, batch 15 used:57
cpu 1 hot: high 186, batch 31 used:29
cpu 1 cold: high 62, batch 15 used:58
cpu 2 hot: high 186, batch 31 used:30
cpu 2 cold: high 62, batch 15 used:57
cpu 3 hot: high 186, batch 31 used:30
cpu 3 cold: high 62, batch 15 used:56
cpu 4 hot: high 186, batch 31 used:26
cpu 4 cold: high 62, batch 15 used:13
cpu 5 hot: high 186, batch 31 used:8
cpu 5 cold: high 62, batch 15 used:42
cpu 6 hot: high 186, batch 31 used:30
cpu 6 cold: high 62, batch 15 used:50
cpu 7 hot: high 186, batch 31 used:157
cpu 7 cold: high 62, batch 15 used:59
Node 0 Normal per-cpu:
cpu 0 hot: high 186, batch 31 used:13
cpu 0 cold: high 62, batch 15 used:50
cpu 1 hot: high 186, batch 31 used:66
cpu 1 cold: high 62, batch 15 used:49
cpu 2 hot: high 186, batch 31 used:3
cpu 2 cold: high 62, batch 15 used:55
cpu 3 hot: high 186, batch 31 used:104
cpu 3 cold: high 62, batch 15 used:55
cpu 4 hot: high 186, batch 31 used:2
cpu 4 cold: high 62, batch 15 used:2
cpu 5 hot: high 186, batch 31 used:87
cpu 5 cold: high 62, batch 15 used:3
cpu 6 hot: high 186, batch 31 used:15
cpu 6 cold: high 62, batch 15 used:55
cpu 7 hot: high 186, batch 31 used:113
cpu 7 cold: high 62, batch 15 used:12
Node 0 HighMem per-cpu: empty
Free pages: 76972kB (0kB HighMem)
Active:3061605 inactive:620399 dirty:68055 writeback:0 unstable:0 free:19243 slab:283635 mapped-file:33024 mapped-anon:2758457 pagetables:91114
Node 0 DMA free:10032kB min:8kB low:8kB high:12kB active:0kB inactive:0kB present:9628kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 3451 16066 16066
Node 0 DMA32 free:53992kB min:3484kB low:4352kB high:5224kB active:2704704kB inactive:367984kB present:3534588kB pages_scanned:268 all_unreclaimable? no
lowmem_reserve[]: 0 0 12615 12615
Node 0 Normal free:12948kB min:12732kB low:15912kB high:19096kB active:9541756kB inactive:2113612kB present:12917900kB pages_scanned:423 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 HighMem free:0kB min:128kB low:128kB high:128kB active:0kB inactive:0kB present:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
Node 0 DMA: 4*4kB 2*8kB 3*16kB 3*32kB 2*64kB 2*128kB 1*256kB 0*512kB 1*1024kB 0*2048kB 2*4096kB = 10032kB
Node 0 DMA32: 0*4kB 59*8kB 45*16kB 82*32kB 0*64kB 6*128kB 5*256kB 0*512kB 1*1024kB 1*2048kB 11*4096kB = 53992kB
Node 0 Normal: 47*4kB 187*8kB 356*16kB 148*32kB 1*64kB 0*128kB 1*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 12948kB
Node 0 HighMem: empty
923737 pagecache pages
Swap cache: add 180089, delete 179928, find 2859/3518, race 0+0
Free swap = 8304596kB
Total swap = 8388600kB
Free swap: 8304596kB
4322816 pages of RAM
230069 reserved pages
14241581 pages shared
117 pages swap cached

My /etc/fstab (SWAP is a file on an SSD):

proc /proc proc defaults 0 0
none /dev/pts devpts gid=5,mode=620 0 0
tmpfs /dev/shm        tmpfs   defaults        0       0
/dev/md0 /boot ext3           defaults 0 0
/dev/md1        /       ext4    rw,noatime,nodiratime,usrjquota=aquota.user,grpjquota=aquota.group,usrquota,grpquota,jqfmt=vfsv0       0       0
/dev/md2        /home   ext4    rw,noatime,nodiratime,usrjquota=aquota.user,grpjquota=aquota.group,usrquota,grpquota,jqfmt=vfsv0       0       0
UUID=b422827c-9f2f-4bc5-ab5a-3cc921924daa /var/lib/mysql ext4 rw,noatime,nodiratime,usrjquota=aquota.user,grpjquota=aquota.group,usrquota,grpquota,jqfmt=vfsv0,  1      1
/var/lib/mysql/swapfile1 swap swap defaults 0 0

Swapiness is set to 20:

cat /proc/sys/vm/swappiness
20

There is only a few RAM left and still using very low SWAP till it crashes and the RAM is freeded and I got another crash in a few hours:

Mem:  16370988k total, 16200724k used,   170264k free,   250340k buffers
Swap:  8388600k total,    83356k used,  8305244k free,   859484k cached
share|improve this question
    
What are your ulimit values for apache? –  David Mar 13 '12 at 1:14
3  
Could you work on getting that accept rate closer to 100%? You've asked 13 questions; surely at least half of them have had acceptable answers. –  Wesley Mar 13 '12 at 2:44
    
pastebin.com/HtUm0DM8 –  Spacedust Mar 13 '12 at 10:31

1 Answer 1

What is /proc//oom_adj ?

How does the memory demand change - a sudden increase might indicate you are hitting the 2Gb limit. A slow increase implies you've definitely got maxclients set too high, and proabably need more hardware. It makes little sense to use big iron as a webserver - use several smaller boxes.

What kernel version - see this bug report (and others).

share|improve this answer
    
2.6.18-308.1.1.el5 #1 SMP Wed Mar 7 04:16:51 EST 2012 x86_64 x86_64 x86_64 GNU/Linux CentOS 5.8 64-bit –  Spacedust Mar 13 '12 at 10:32

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.