Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

So Java has been hanging our systems. Our developers are still looking into this but as an Admin what can be done.

On a dual socket system with altogether 8 cores, Load peaks at 40's.. On average during the day: load average: 17.78, 23.18, 24.71 According to this: http://blog.scoutapp.com/articles/2009/07/31/understanding-load-averages
it's not normal.

other than upgrading the system, what can be done about this? our issue currently are randomly system hangs with "task java:4242 blocked for more than 120 seconds" then panics during high load.

What can be done?

During a load of: load average: 21.40, 21.76, 21.80

iostat:

      avg-cpu:  %user   %nice %system %iowait  %steal   %idle
                13.75    0.00    0.49    0.00    0.00   85.77

      Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn
      xvda              2.16         1.82        32.63     832500   14917166

vmstat:

      procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
      r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
      0  0  21172 1272200 138540 583268   0    0     0     3    9    5 14  0 86  0

top:

top - 14:41:02 up 5 days,  7:26,  2 users,  load average: 23.01, 22.17, 21.82
Tasks: 103 total,   1 running, 102 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.0%us,  0.0%sy,  0.0%ni,100.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   5291864k total,  4017092k used,  1274772k free,   138544k buffers
Swap:  1959924k total,    21172k used,  1938752k free,   583332k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
25637 root      20   0 1691m 559m  13m S    2 10.8   4:46.91 java
  848 root      20   0 3044m 2.3g 9844 S    1 45.4   5521:37 java
27729 root      20   0 19272 1272  948 R    0  0.0   0:00.03 top
    1 root      20   0 23760 1696 1200 S    0  0.0   0:01.49 init
    2 root      20   0     0    0    0 S    0  0.0   0:00.00 kthreadd
    3 root      20   0     0    0    0 S    0  0.0   0:00.29 ksoftirqd/0
    4 root      20   0     0    0    0 S    0  0.0   0:00.00 kworker/0:0
share|improve this question
    
What's the cause of the load? Is it CPU intensive or Disk intensive? –  Jay Jul 10 '12 at 18:12
    
I will update on the main thread. –  Tom G Jul 10 '12 at 18:13
    
Limit the threads that the Java application uses, and don't make it do so much. –  womble Jul 10 '12 at 18:14
    
Try running mpstat to get some visibility into the interrupts on the system. Look at %irq and %soft. Maybe a network or other device is causing a lot of processor overhead. –  HTTP500 Jul 10 '12 at 21:36

2 Answers 2

In short, nice.

http://en.wikipedia.org/wiki/Nice_(Unix)

share|improve this answer
    
That won't help load average; the same number of CPUs will still be competing for the same number of CPU cores. At best, it'll provide a moderate improvement in interactive responsiveness. –  womble Jul 10 '12 at 18:15
1  
ionice often has a bigger impact on how a process loads the system. –  lornix Jul 11 '12 at 2:36

I feel like there's something else going on that you haven't looked at or told us about.

  • You don't have I/O issues, or even a really notable I/O load
  • You're not running on bare hardware. xvda is an indicator of a virtual device.
  • You have "all the time in the world" -- your system is reporting as 85% idle, despite a very high load average showing.

Here are my most likely guesses:

  • You took the demonstrative snapshot at the wrong time. The CPU is actually much more burdened and you should use a tool like sar for reporting. atop is also a favorite of mine.
  • You have a generally low load, but very high instant bursts that are making the overall load average high.
  • You are encountering an insidious bug or hardware issue. You should post the stack traces that are logged for killed processes so we can take a guess where.

As to the topic of the question itself: the only way to lower load average is to reduce the number of processes actively competing for CPU time.

share|improve this answer
    
i have updated the thread with TOP. strange it does show CPU load but nothing using it. what would cause this? this is running under XEN through our provider. –  Tom G Jul 10 '12 at 18:42
    
@TomG top is also showing you just a snapshot in time. Seeing small consecutive slices of time for a long period is what I need to diagnose. –  Jeff Ferland Jul 10 '12 at 18:48
    
thanks for the responses. Doesn't this load show 1, 5, and 15 minute load averages? doing top with a 0.1 second change delay shows it going no more than 10% but the 1 minute average is not changing, it's even going up at times. –  Tom G Jul 10 '12 at 18:55
    
By default, top shows 10 second detail. If you see high 60 second load average, but 0 utilization the 10 second snapshot, then it must be happening somewhere in the other 50 seconds. That's the part I'm trying to find. That or you have underlying hypervisor issues that just aren't showing up in the VM's diagnostic tools. –  Jeff Ferland Jul 10 '12 at 19:05
    
Thanks, I understand top has a delay, but I sent the delay to 0.1 (option 'd') and didn't see any strange spikes. very weird. –  Tom G Jul 10 '12 at 19:12

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.