I have a piece of java code running on two different machines, but on one of the linux machines, the code uses a lot of CPU (close to 100% cpu usage). On the other machine the same code uses less cpu (under 3 to 4%). The machine where cpu usage is high, is a more powerful machine, more CPU and and more memory. This has started happening recently and performance on the machine with high cpu usage has degraded significantly. I am wondering if anyone has any ideas why something like this could happen, possible causes behind this etc. any guesses? No recent changes in hardware were made, no recent code updates.
If you suspect hardware failure, check dmesg for any output. Depending on the hardware vendor, there may be some kind of IPMI implementation (like Dell's DRAC) with a web interface that will show you failed hardware.
To be honest, it's probably a bug in your application. "Add timer hooks, find the bottleneck."
LB REQUESTS SENT TO NODE1 | NODE1 HW SPEC1 | NODE1 APP DEPLOYED |
LB REQUESTS SENT TO NODE2 | NODE2 HW SPEC2 | NODE2 APP DEPLOYED |
"One of these kids is not like the other"
Tell us which one.
If indeed it all checks out the you need to consider that there has been a change in the pattern of data that is being sent in the requests in the app.
Double Check the scheme the LB uses to distribute calls, verify it is not data dependent.
Run jProfiling on both JVMs, Top or similar on the OS, and monitoing mysql files.