Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

My server is running Debian 6 64bit with kernel 2.6.32-5-amd64, a few times a day the load on my server goes very high I have this call trace below can anyone explain to me what the problem is?

    [521565.551017] Call Trace:
[521565.551019]  <IRQ>  [<ffffffff810ba7b3>] ? __alloc_pages_nodemask+0x59b/0x5fc
[521565.551029]  [<ffffffff81248c08>] ? skb_gro_receive+0x17e/0x3cf
[521565.551034]  [<ffffffff810e6b26>] ? new_slab+0x5b/0x1ca
[521565.551037]  [<ffffffff810e6e85>] ? __slab_alloc+0x1f0/0x39b
[521565.551039]  [<ffffffff8124896e>] ? __alloc_skb+0x3e/0x15a
[521565.551041]  [<ffffffff8124896e>] ? __alloc_skb+0x3e/0x15a
[521565.551044]  [<ffffffff810e7261>] ? kmem_cache_alloc_node+0x8b/0x10b
[521565.551046]  [<ffffffff8124896e>] ? __alloc_skb+0x3e/0x15a
[521565.551051]  [<ffffffff8119bf48>] ? swiotlb_map_page+0x0/0xc4
[521565.551053]  [<ffffffff8124996c>] ? __netdev_alloc_skb+0x29/0x45
[521565.551063]  [<ffffffffa0058b00>] ? tg3_alloc_rx_skb+0x96/0x11e [tg3]
[521565.551067]  [<ffffffffa006291e>] ? tg3_poll_work+0x45a/0xb49 [tg3]
[521565.551072]  [<ffffffff8127362c>] ? ip_local_deliver_finish+0x146/0x1e9
[521565.551076]  [<ffffffff8124fc8a>] ? napi_gro_flush+0x1e/0x39
[521565.551080]  [<ffffffffa0063139>] ? tg3_poll+0x8f/0x1a8 [tg3]
[521565.551084]  [<ffffffff811f6f71>] ? mix_pool_bytes_extract+0x57/0x14a
[521565.551087]  [<ffffffff8124fd8e>] ? net_rx_action+0xae/0x1c9
[521565.551091]  [<ffffffff81053d2b>] ? __do_softirq+0xdd/0x1a6
[521565.551095]  [<ffffffff81011cac>] ? call_softirq+0x1c/0x30
[521565.551097]  [<ffffffff8101322b>] ? do_softirq+0x3f/0x7c
[521565.551100]  [<ffffffff81053b9b>] ? irq_exit+0x36/0x76
[521565.551102]  [<ffffffff81012922>] ? do_IRQ+0xa0/0xb6
[521565.551104]  [<ffffffff810114d3>] ? ret_from_intr+0x0/0x11
[521565.551106]  <EOI>  [<ffffffff81017318>] ? mwait_idle+0x72/0x7d
[521565.551110]  [<ffffffff810172c8>] ? mwait_idle+0x22/0x7d
[521565.551113]  [<ffffffff8100fe97>] ? cpu_idle+0xa2/0xda
[521565.551115] Mem-Info:
[521565.551116] Node 0 DMA per-cpu:
[521565.551118] CPU    0: hi:    0, btch:   1 usd:   0
[521565.551120] CPU    1: hi:    0, btch:   1 usd:   0
[521565.551121] CPU    2: hi:    0, btch:   1 usd:   0
[521565.551123] CPU    3: hi:    0, btch:   1 usd:   0
[521565.551124] CPU    4: hi:    0, btch:   1 usd:   0
[521565.551126] CPU    5: hi:    0, btch:   1 usd:   0
[521565.551127] CPU    6: hi:    0, btch:   1 usd:   0
[521565.551129] CPU    7: hi:    0, btch:   1 usd:   0
[521565.551130] Node 0 DMA32 per-cpu:
[521565.551132] CPU    0: hi:  186, btch:  31 usd:   0
[521565.551133] CPU    1: hi:  186, btch:  31 usd:   0
[521565.551135] CPU    2: hi:  186, btch:  31 usd:   0
[521565.551136] CPU    3: hi:  186, btch:  31 usd:   0
[521565.551138] CPU    4: hi:  186, btch:  31 usd:   0
[521565.551139] CPU    5: hi:  186, btch:  31 usd:   0
[521565.551141] CPU    6: hi:  186, btch:  31 usd:   0
[521565.551142] CPU    7: hi:  186, btch:  31 usd:   0
[521565.551143] Node 0 Normal per-cpu:
[521565.551145] CPU    0: hi:  186, btch:  31 usd:   0
[521565.551147] CPU    1: hi:  186, btch:  31 usd:   0
[521565.551148] CPU    2: hi:  186, btch:  31 usd:  30
[521565.551150] CPU    3: hi:  186, btch:  31 usd:   0
[521565.551151] CPU    4: hi:  186, btch:  31 usd:   0
[521565.551153] CPU    5: hi:  186, btch:  31 usd:   1
[521565.551154] CPU    6: hi:  186, btch:  31 usd:   0
[521565.551155] CPU    7: hi:  186, btch:  31 usd:   0
[521565.551159] active_anon:50248 inactive_anon:44179 isolated_anon:0
[521565.551160]  active_file:1929286 inactive_file:1929366 isolated_file:0
[521565.551161]  unevictable:0 dirty:307090 writeback:56879 unstable:0
[521565.551162]  free:18514 slab_reclaimable:68418 slab_unreclaimable:51386
[521565.551163]  mapped:260756 shmem:95 pagetables:4659 bounce:0
[521565.551165] Node 0 DMA free:15852kB min:12kB low:12kB high:16kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15296kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[521565.551173] lowmem_reserve[]: 0 2991 16121 16121
[521565.551175] Node 0 DMA32 free:53496kB min:3012kB low:3764kB high:4516kB active_anon:13668kB inactive_anon:24464kB active_file:1297884kB inactive_file:1298024kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3063648kB mlocked:0kB dirty:290652kB writeback:41076kB mapped:221156kB shmem:16kB slab_reclaimable:71508kB slab_unreclaimable:55600kB kernel_stack:360kB pagetables:1048kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[521565.551184] lowmem_reserve[]: 0 0 13130 13130
[521565.551187] Node 0 Normal free:4708kB min:13224kB low:16528kB high:19836kB active_anon:187324kB inactive_anon:152252kB active_file:6419260kB inactive_file:6419440kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:13445120kB mlocked:0kB dirty:937708kB writeback:186440kB mapped:821868kB shmem:364kB slab_reclaimable:202164kB slab_unreclaimable:149944kB kernel_stack:1480kB pagetables:17588kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[521565.551196] lowmem_reserve[]: 0 0 0 0
[521565.551198] Node 0 DMA: 1*4kB 1*8kB 2*16kB 2*32kB 2*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15852kB
[521565.551204] Node 0 DMA32: 121*4kB 276*8kB 539*16kB 1197*32kB 4*64kB 4*128kB 0*256kB 2*512kB 0*1024kB 1*2048kB 0*4096kB = 53460kB
[521565.551211] Node 0 Normal: 612*4kB 80*8kB 43*16kB 16*32kB 0*64kB 0*128kB 0*256kB 1*512kB 0*1024kB 0*2048kB 0*4096kB = 4800kB
[521565.551217] 3883344 total pagecache pages
[521565.551218] 24509 pages in swap cache
[521565.551220] Swap cache stats: add 96013, delete 71504, find 812823/817958
[521565.551221] Free swap  = 4737200kB
[521565.551222] Total swap = 4878304kB
[521565.594382] 4194304 pages RAM
[521565.594383] 77961 pages reserved
[521565.594384] 2084547 pages shared
[521565.594385] 2278383 pages non-shared
[521565.594387] SLUB: Unable to allocate memory on node -1 (gfp=0x20)
[521565.594390]   cache: kmalloc-256, object size: 256, buffer size: 256, default order: 1, min order: 0
[521565.594393]   node 0: slabs: 500, objs: 15104, free: 8
[521598.698442] swapper: page allocation failure. order:4, mode:0x4020
[521598.698451] Pid: 0, comm: swapper Not tainted 2.6.32-5-amd64 #1
share|improve this question

1 Answer 1

I obviously don't know for certain, but judging from that trace, I'd say it's

  • network related
  • indicates your kernel can't allocate memory to handle whatever is coming in
  • specifically inbound traffic related

I would start by running a traffic trace, to correlate inbound network traffic to the periods of slowness. That might give you some insight as to what is going on.

It could also be entirely unrelated to network traffic and just be a memory issue. Check the output of free -m.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.