0

I have an Asterisk server doing automated calls and I'm noticing an unexplained high load in it. The server is only running Asterisk. Database and other support applications run in a different machine.

What can be causing this high load?

If the load gets too high for too long (over the maximum of 8) there is a drop in calls and general unresponsiveness.

The hardware is an 8 core "Intel Xeon E3-1230 V2 @ 3.30GHz" with 16GB of RAM

I read the other posts about similar problems so I'll post all the information requested in them.

Following are the results of the monitoring tools I have been using. They system is currently handling about 200 channels.

It doesn't scale linearly, at 400 channels the load gets to 8 and things go downhill from there.

"ps auxf" shows nothing in D state.

top - 10:50:28 up 7 days, 23:20,  3 users,  load average: 2.16, 2.44, 1.93
Tasks: 341 total,   1 running, 340 sleeping,   0 stopped,   0 zombie
Cpu(s):  4.3%us,  5.9%sy,  0.0%ni, 89.4%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:  16303732k total,  7179980k used,  9123752k free,   264836k buffers
Swap:  8224760k total,        0k used,  8224760k free,  5759716k cached

2512 root      20   0 4744m 173m  46m S 50.8  1.1 396:55.01 asterisk

A typical iostat -x. sda and sdb are in raid 1 and sdc is an ssd storing some sound files used very often.

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           4.30    0.00    5.94    0.38    0.00   89.38

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz       await  svctm  %util
sda               0.00     6.00    0.00    9.00     0.00   112.00    12.44     0.04        4.33   3.67   3.30
sdc               0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00        0.00   0.00   0.00
sdb               0.00     7.00    0.00    8.00     0.00   112.00    14.00     0.04        5.25   4.62   3.70
md127             0.00     0.00    0.00   15.00     0.00   112.00     7.47     0.00        0.00   0.00   0.00
dm-0              0.00     0.00    0.00   14.00     0.00   112.00     8.00     0.05        3.50   2.64   3.70
dm-1              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00        0.00   0.00   0.00
dm-2              0.00     0.00    0.00    0.00     0.00     0.00     0.00     0.00        0.00   0.00   0.00

cat /proc/interrupts

           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6           CPU7
  0:        127          0          0          0          0          0          0              0   IO-APIC-edge      timer
  1:          2          0          0          0          0          0          0          0   IO-APIC-edge      i8042
  3:          2          0          0          0          0          0          0          0   IO-APIC-edge
  4:          2          0          0          0          0          0          0          0   IO-APIC-edge
  8:          1          0          0          0          0          0          0          0   IO-APIC-edge      rtc0
  9:          0          0          0          0          0          0          0          0   IO-APIC-fasteoi   acpi
 10:          2          0          0          0          0          0          0          0   IO-APIC-edge
 12:          4          0          0          0          0          0          0          0   IO-APIC-edge      i8042
 16:         59          0          0          0          0          0          0          0   IO-APIC-fasteoi   ehci_hcd:usb1
 23:         97          0          0          0          0          0          0         40   IO-APIC-fasteoi   ehci_hcd:usb2
 24:       2298          0          0          0          0          0          0          0  HPET_MSI-edge      hpet2
 25:          0          0          0          0          0          0          0          0  HPET_MSI-edge      hpet3
 26:          0          0          0          0          0          0          0          0  HPET_MSI-edge      hpet4
 27:          0          0          0          0          0          0          0          0  HPET_MSI-edge      hpet5
 28:          0          0          0          0          0          0          0          0  HPET_MSI-edge      hpet6
 29:     412442          0          0         36    1175680          0          0        208   PCI-MSI-edge      ahci
 30:         74  335176384          0          0          0          0          0          0   PCI-MSI-edge      eth0
 31:          5         10     344792          0          0          0          0              0   PCI-MSI-edge      eth1-rx-0
 32:          0          0          0          0          0          0          0          0   PCI-MSI-edge      eth1-tx-0
 33:          3          0          0          0          0          0          0          0   PCI-MSI-edge      eth1
NMI:       7784      14329       4689       5198       7033       7387       6069       6332   Non-maskable interrupts
LOC:   46833697   44931615   30462128   37088906   47922986   44201942   27616867   37813275   Local timer interrupts
SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
PMI:       7784      14329       4689       5198       7033       7387       6069       6332   Performance monitoring interrupts
IWI:          0          0          0          0          0          0          0          0   IRQ work interrupts
RES:     897464     372249     589429     570768     646428     605601     478042     484381   Rescheduling interrupts
CAL:         92        292        281        289        267        292        288        291   Function call interrupts
TLB:  206630086  265955778  173872460  156528749  143771724  221909392  129664286  115580760   TLB shootdowns
TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
THR:          0          0          0          0          0          0          0          0   Threshold APIC interrupts
MCE:          0          0          0          0          0          0          0          0   Machine check exceptions
MCP:       2300       2300       2300       2300       2300       2300       2300       2300   Machine check polls
ERR:          0

MIS: 0

vmstat 1 20

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0      0 9103556 265224 5776272    0    0     1     1    2    8  0  1 99  0  0
 0  0      0 9103356 265224 5776268    0    0     0    24 101813 25156  5  9 86  0  0
 1  0      0 9103224 265224 5776324    0    0     0     2 139955 25205  5 12 83  0  0
 3  0      0 9103968 265224 5776280    0    0     0    54 102550 24442  5  9 85  1  0
 2  0      0 9103084 265224 5776304    0    0     0     0 84384 22729  3  7 90  0  0
 2  0      0 9102116 265224 5776412    0    0     0     0 84072 24705  6  8 87  0  0
 1  0      0 9103432 265224 5776328    0    0     0     0 108438 24144  5  9 86  0  0
 0  0      0 9102924 265224 5776340    0    0     0     2 41961 23168  3  5 92  0  0
 0  0      0 9102608 265224 5776364    0    0     0    90 76298 26135  5  7 87  1  0
 2  0      0 9078068 265224 5776444    0    0     0     0 83315 24891  5  8 87  0  0
 0  0      0 9103344 265224 5776436    0    0     0     0 67256 26539  6  7 87  0  0
 1  0      0 9094300 265224 5776444    0    0     0     0 54944 24834  3  6 91  0  0
 0  0      0 9103352 265224 5776460    0    0     0     2 92988 26388  5  9 86  0  0
 2  0      0 9103592 265224 5776440    0    0     0    46 76231 27186  5  7 87  1  0
 1  0      0 9103744 265224 5776500    0    0     0     0 67153 26006  5  7 88  0  0
 3  0      0 9103056 265224 5776520    0    0     0    76 86165 26895  5  8 87  0  0
 1  0      0 9094384 265224 5776568    0    0     0    84 59498 26179  4  6 90  0  0
 1  0      0 9088632 265224 5776556    0    0     0     0 103184 27236  6  9 85  0  0
 1  0      0 9102532 265224 5776608    0    0     0    40 94010 27663  6  9 85  0  0
 1  0      0 9091052 265224 5776648    0    0     0     0 93813 26675  9  9 82  0  0

mpstat 1

11:05:18 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
11:05:19 AM  all    8.22    0.00   11.88    0.00    0.00    0.63    0.00    0.00   79.27
11:05:20 AM  all    3.05    0.00    4.96    0.00    0.00    0.38    0.00    0.00   91.60
11:05:21 AM  all    5.64    0.00    7.27    0.63    0.00    0.38    0.00    0.00   86.09
11:05:22 AM  all    5.44    0.00    6.96    0.00    0.00    0.25    0.00    0.00   87.34
11:05:23 AM  all    3.76    0.00    7.14    0.00    0.00    0.25    0.00    0.00   88.85
11:05:24 AM  all    4.80    0.00    9.86    0.00    0.00    0.51    0.00    0.00   84.83
11:05:25 AM  all    3.80    0.00    5.58    0.00    0.00    0.38    0.00    0.00   90.24
11:05:26 AM  all    6.58    0.00    7.72    0.51    0.00    0.51    0.00    0.00   84.68
11:05:27 AM  all    6.67    0.00    8.43    0.00    0.00    0.50    0.00    0.00   84.40
11:05:28 AM  all    4.32    0.00    5.97    0.00    0.00    0.25    0.00    0.00   89.45
11:05:29 AM  all    5.04    0.00    7.06    0.00    0.00    0.50    0.00    0.00   87.39
11:05:30 AM  all    3.93    0.00    6.34    0.13    0.00    0.51    0.00    0.00   89.10
11:05:31 AM  all    4.07    0.00    5.60    0.38    0.00    0.25    0.00    0.00   89.69
11:05:32 AM  all    7.08    0.00    9.48    0.00    0.00    0.51    0.00    0.00   82.93
11:05:33 AM  all    4.19    0.00    8.51    0.00    0.00    0.51    0.00    0.00   86.79
11:05:34 AM  all    2.67    0.00    4.45    0.00    0.00    0.25    0.00    0.00   92.63
  • 1
    What would really be useful is if you captured the output from all those tools when the server is under high load. Right now all the output shows a box with plenty of spare resources. – zygis Sep 18 '13 at 16:52
  • The load of 2 is disproportional considering the CPU and IO use. I hoped it would be enough. Can this load be explained by the provided info? – Allan Sep 18 '13 at 16:58
1

Maybe its too late to answer to this, and may be the reason is not the same one as in my case.

But I also abserved such high CPU load when updating my asterisk 1.8 to asterisk 11.5 (on Fedora 14 to Fedora 20), but keeping my old configuration file! In asterisk.conf, the line:

;console = yes ; Run as console (same as -c at startup).

was not commented with a semicolon as soon as the line was commented, and asterisk restarted, the CPU go back to a normal activity !!!!

0

It seems you have a quite high number of context switches and interrupts. Also if you look at the load of 2 and the running queue I think that the scenario could be caused by quick bursts of threads that finish their job very quickly so they don't spend much time in the CPU but they make grow up the load.

P.S. Excuse my english please.

  • Thinking about the application itself this makes a lot of sense. Would you have a suggestion that helps determine where such threads come from? I'm not sure if they are inate to Asterisk or something else Asterisk spawns that I could possibly change. – Allan Sep 18 '13 at 17:17
  • Hi Allan, I don't know Asterisk in deep but I think this behaviour could fit with it. Maybe with "strace" you could try to determine if threads come from Asterisk or not. – Icarokun Sep 19 '13 at 15:27
0

You may not see all processes with simple ps auxf, becasue with these options ps hides threads of multi-threaded processes. Try using ps -eLo pid,stat,comm, this will show all threads where some threads could have R or D states. And load average is average amount of R + D theads (or tasks, in linux terminology) per sample interval (which could be 100 times per second).

0

Only thing that Asterisk spends CPU time is recoding when changing codecs (not your case, but when the phones are ALAW/ULAW, and the trunk is G.729), when doing conferences and mixing audio channels into one, MusicOnHold etc. Try recoding your sound files into ALAW/ULAW. If you record calls, try to record it into *LAW (onto RAM disk) and recode to MP3 or some network storage during off-hours. If you have a voice trunk other than SIP, that also can make a difference.

This CPU is way over-dimensioned for that amount of calls. Check out Asterisk dimensioning and you will see that you can do 1000 simultaneous calls with even worse hardware.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.