0

I have a piece of java code running on two different machines, but on one of the linux machines, the code uses a lot of CPU (close to 100% cpu usage). On the other machine the same code uses less cpu (under 3 to 4%). The machine where cpu usage is high, is a more powerful machine, more CPU and and more memory. This has started happening recently and performance on the machine with high cpu usage has degraded significantly. I am wondering if anyone has any ideas why something like this could happen, possible causes behind this etc. any guesses? No recent changes in hardware were made, no recent code updates.

5
  • We need more details. Do they have the same actual workload? How do you know? Apr 14, 2012 at 0:22
  • load is being distributed between the two machines, load is distributed in round robin fashion. Any request going to the machine with high cpu usage takes around 40-50 minutes sometimes even longer than that. Previously the same type of requests were taking a few seconds to a couple of minutes. The other machine is still processing requests in the same timeframe (few seconds to a couple of minutes)
    – user305210
    Apr 14, 2012 at 0:38
  • Is this an in-house application? To what extent can we actually help you apart from "Add timer hooks, find the bottleneck?" Apr 14, 2012 at 0:45
  • Yes, this is an in-house app. I was actually trying to find out if hardware issues can lead to performance degradation all of sudden and if that happens, are there specific things that one should look for.
    – user305210
    Apr 14, 2012 at 0:52
  • Hardware issues can most definitely lead to performance degradation. Some CPUs will underclock themselves to prevent damage from overheating. You could try lm-sensors to see if there's high CPU temperature.
    – Nathan
    May 15, 2012 at 3:53

2 Answers 2

3

If you suspect hardware failure, check dmesg for any output. Depending on the hardware vendor, there may be some kind of IPMI implementation (like Dell's DRAC) with a web interface that will show you failed hardware.

To be honest, it's probably a bug in your application. "Add timer hooks, find the bottleneck."

0
  • Verify that the deployed apps are the exact same on both nodes.
  • Verify that both sets of JVM settings and app configuration are the same where appropriate on both Nodes
  • Verify the hardware configuration and spec is the same on both nodes. (CPU/RAM/DISK)
  • Verify that the LB is indeed providing a round robin service. Review the log files of both apps to check that incoming requests are evenly distributed
  • If none of the above throws up anything then run jProfiler on the slow machine and report that back to the app developers if possible.
  • Monitor all processes on both machines.

LB REQUESTS SENT TO NODE1 | NODE1 HW SPEC1 | NODE1 APP DEPLOYED |


LB REQUESTS SENT TO NODE2 | NODE2 HW SPEC2 | NODE2 APP DEPLOYED |

If indeed it all checks out the you need to consider that there has been a change in the pattern of data that is being sent in the requests in the app.

Double check the scheme the LB uses to distribute calls, verify it is not data dependent.

Run jProfiling on both JVMs, Top or similar on the OS, and monitoring mysql log files.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .