Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I am using Weblogic 10.3.4 + Oracle DB both on Solaris machines (VM)

Unfortunately last week our app hosted on Weblogic became very very slow not being able to serve new users coming (everything lasted for 1h).

In order to find out the cause is we asked reports from the OS level , DB level.

The reports shown:

  • a very intesive CPU usage that hour. (clearly was a peak from number of users using the system)
  • Connection Pool reached the top limit on Weblogic (200 connections !)
  • AWR stats from Db shown a degradation in response times.

NOW who was first the chicken or the egg ?

A very slow DB can cause WEBlogic to be slow and to reach the max numeber of connections from pool.

However if the CPU is very used , it might be that the threads holding the DB connections to be so busy that they do not return the connection to the pool (because the processor is busy)==> a new client comes a new connection is assgined and so on ...so the connection pool might be exhausted as well in this way ...?

Regards

Cris

share|improve this question
1  
Please provide more details of the setup. By VM, do you mean Virtual Machines? What is the the virtualization technology used? Basically Solaris or VMWare? Are these servers running on same physical host(s)? What kind of capacity you have in the servers? Do you have Sar-statistics available of the situation? The AWR is able to provide you in detail what happened in the database at the time. The "Top 5"-list will give you pointers to start looking deeper from. –  grassroot Jan 26 '13 at 16:44
    
Solaris zones. But DB and Weblogic running on different machines.However my question might make abstraction of the configuration...intensive CPU usage might cause on Weblogic exhaustion of the connection pool ? AWR stats show the worst query had a average of 3s the rest were ms.Thanks. –  Cris Jan 26 '13 at 19:24

1 Answer 1

High CPU usage is often caused by the java garbage collector that is trying to collect free memory in the JVM heap space, perm gen, etc. If this is the case, you need to add more memory, usually to JVM heap. See the -Xmx JVM option.

When the CPU usage is high in a JVM, try to find out which thread(s) is(are) using the CPU. During the problem create 2-3 thread dumps with pkill -3 java. The thread dumps will be located (for Solaris) in /proc/<JVM_PID>/path/1

If the thread pool would reach its limit, you should see errors in the logs. The CPU usage should not rise because of this. It is an event driven environment and no need to do pooling.

Slow DB could actually lower the CPU usage on the application server. Depends on how the application is programmed/configured.

share|improve this answer
    
Thanks ...we can't add more memory...it might create even longer GC's we have 4GB / manages server (4 managed servers). It is a post mortem analysis...which i want to reproduce via stress and load testing. Don;t know know which threads used what...what I have are statistics from Domain health...connection pool was exhausted ...and new requests were coming waiting for a db connection...which never came ==> 400 threads waiting on each managed server might result in high CPU usage no ? –  Cris Feb 8 '13 at 21:42
    
I did not understood what you meant by "It is an event driven environment and no need to do pooling." Weblogic has a pool for DB connections and as well a self tuning pool for its threads. –  Cris Feb 8 '13 at 21:50
    
It depends on the application and on the implementation of the connection pool. Threads waiting = 0% CPU usage. –  Mircea Vutcovici Feb 8 '13 at 21:52
1  
Do you have garbage collector logs? Do you have the STDOUT logs of the JVMs? –  Mircea Vutcovici Feb 8 '13 at 21:55
1  
Check that the processes are running in 64 bit virtual memory space with pldd <JVM_PID> or pmap <JVM_PID>|grep .so –  Mircea Vutcovici Feb 8 '13 at 22:00

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.