Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

When I keep sending http requests to my webserver, it stops responding and I'm stuck in a white screen saying, 'waiting for xxx'. It goes unresponsive. This only happens on Linux.

Also I don't see any exception on stacktrace. I don't see any exception in catalina.out.

I think it has something to do with tomcat not closing socket connections. This only happens when

$ cd /proc/{processID}/fd
$ ls -l | wc -l

returns value over 1000. Almost everytime I hit refresh the return value of 'ls -l | wc -l' goes up !

And if I issue this command

$ ll

I see a lot of records like

lrwx------ 1 root root 64  5월 22 17:49 989 -> socket:[1018368]
lr-x------ 1 root root 64  5월 22 17:41 99 -> socket:[1017430]
lrwx------ 1 root root 64  5월 22 17:49 990 -> socket:[1018369]
lrwx------ 1 root root 64  5월 22 17:49 991 -> socket:[1018370]
lrwx------ 1 root root 64  5월 22 17:49 992 -> socket:[1018371]
lrwx------ 1 root root 64  5월 22 17:49 993 -> socket:[1018384]
lrwx------ 1 root root 64  5월 22 17:49 994 -> socket:[1018373]
lrwx------ 1 root root 64  5월 22 17:49 995 -> socket:[1018374]
lrwx------ 1 root root 64  5월 22 17:49 996 -> socket:[1018375]
lrwx------ 1 root root 64  5월 22 17:49 997 -> socket:[1018377]
lrwx------ 1 root root 64  5월 22 17:49 998 -> socket:[1018378]
lrwx------ 1 root root 64  5월 22 17:49 999 -> socket:[1018379]

I'm pretty sure it's not about my web application not returning(or not closing) its connections to the database.

I'm using MySQL and I've configured my.cfg

[mysqld]
wait-timeout=3
interactive-timeout=60

I can check it in mysql

mysql> select @@session.wait_timeout, @@global.wait_timeout;
+------------------------+-----------------------+
| @@session.wait_timeout | @@global.wait_timeout |
+------------------------+-----------------------+
|                     30 |                     3 |
+------------------------+-----------------------+
1 row in set (0.00 sec)

Also if I issue

This is not a DNS issue. I can open the website on my webserver just fine.

netstat -tulpn
tcp        0      0 ::ffff:127.0.0.1:8005       :::*                        LISTEN      21395/java
tcp        0      0 :::8009                     :::*                        LISTEN      21395/java
tcp        0      0 :::80                       :::*                        LISTEN      21395/java

Also

lsof -p 21395

show a lot of

java    21395 root  359u  sock                0,5      0t0   1379694 can't identify protocol
java    21395 root  360u  sock                0,5      0t0   1379696 can't identify protocol
java    21395 root  361u  sock                0,5      0t0   1379697 can't identify protocol
java    21395 root  362u  sock                0,5      0t0   1379700 can't identify protocol
java    21395 root  363u  sock                0,5      0t0   1379702 can't identify protocol
java    21395 root  364u  sock                0,5      0t0   1379703 can't identify protocol
java    21395 root  365u  sock                0,5      0t0   1379961 can't identify protocol

Where as I can see Tomcat's connections to MySQL like

java    21395 root  390u  IPv6            1380717      0t0       TCP cooingbee.co.kr:40026->cooingbee.co.kr:mysql (CLOSE_WAIT)
java    21395 root  391u  IPv6            1380718      0t0       TCP cooingbee.co.kr:40025->cooingbee.co.kr:mysql (CLOSE_WAIT)
java    21395 root  392u  IPv6            1380721      0t0       TCP cooingbee.co.kr:40028->cooingbee.co.kr:mysql (CLOSE_WAIT)

I've set ulimit also

]# ulimit -n 
4096
]#ulimit -Hn
65536

In my sysctl.conf, I've added these but still no luck.

kernel.shmmni = 4096
kernel.sem = 256 32000 100 128
fs.aio-max-nr = 1048576
fs.file-max = 6815744
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586

Additional Information on my server environment

CentOS 5.x
Java 7
Tomcat6 / 7 (I've tested my webapp on both tomcat 6 and 7)
MySQL 5.5
web application uses Hibernate, c3p0, Spring DI, MVC

I've confirmed from the DBMS side that connections are not leaking by using

show processlist

Question

What are those 'sock can't identify protocol' ?

Why my tomcat goes unresponsive if I have more than a thousand records in /proc/{processID}/fd ?

What should I do to make it work smooth?

EDIT 1 I'll show you the code. FYI, I didn't design nor intended to use Hibernate and Spring this way. I think you'll see why I say this now ;).

part pf my hibernate.cfg.xml

<property name="connection.driver_class">com.mysql.jdbc.Driver</property>
<property name="connection.url">jdbc:mysql://***.***.***.***:3306/****</property>
<property name="connection.username">****</property>
<property name="connection.password">****</property>
<property name="hibernate.c3p0.acquire_increment">5</property> 
<property name="hibernate.c3p0.idle_test_period">100</property>
<property name="hibernate.c3p0.max_size">10000</property> 
<property name="hibernate.c3p0.max_statements">0</property> 
<property name="hibernate.c3p0.min_size">5</property>
<property name="hibernate.c3p0.testConnectionOnCheckout">true</property> 
<property name="hibernate.c3p0.unreturnedConnectionTimeout">70</property>
<property name="hibernate.c3p0.timeout">1</property>
<property name="current_session_context_class">thread</property>
<property name="cache.provider_class">org.hibernate.cache.NoCacheProvider</property>

part of my spring-servlet.xml

<bean id="sessionFactory" class="org.springframework.orm.hibernate3.LocalSessionFactoryBean">
        <!--  
        <property name="dataSource"><ref bean="dataSource"/></property>
        -->
        <property name="configLocation">
            <value>classpath:hibernate.cfg.xml</value>
        </property>

Finally, some Java code

public class ItemDAOImpl extends HibernateDaoSupport implements ItemDAO {

public ItemDetail itemDetail(Integer itemSeq) { 
        ItemDetail itemDetail = null;

        try {
            itemDetail =  (ItemDetail) this.getHibernateTemplate().get("kr.co.sgis.services.web.mobile.item.form.ItemDetail", itemSeq);
            System.out.println("Item Detail~ :: " + itemDetail.getId());
            return itemDetail;
        } catch (Exception ex) {
            System.err.println(ex.getMessage() + "\nThe Stack Trace is:"); 
            ex.printStackTrace();
            return itemDetail;
        }

    }
public Item itemView(Integer itemSeq, Integer mo_id) { 
        Item itemView = null;
        Session session = null;
        try {
            session = getSession(); 
            Criteria item = session.createCriteria(Item.class)
               .add(Restrictions.eq("id", itemSeq))
               .add(Restrictions.eq("mobile_id", mo_id));
            itemView = (Item) item.uniqueResult();

            //itemView =  (ItemHead) this.getHibernateTemplate().get("kr.co.sgis.services.web.mobile.item.form.Item", itemSeq);
            System.out.println("Item View~ :: " + itemView.getId());
            session.flush();
            return itemView;
        } catch (Exception ex) {
            System.err.println(ex.getMessage() + "\nThe Stack Trace is:"); 
            ex.printStackTrace();
            return itemView;
        }finally{
            HibernateUtil.tryCloseAll(session, null);
        }

    }
...
}

As far as I've searched on the internet, getHibernateTemplate() and HibernateDAOSupport are not recommended. Also if I use getHibernateTemplate(), I don't have to manually, explicitly close session but I got desparate so I added 'HibernateUtil.tryCloseAll()'.

Here

public static void tryCloseAll(Session s, Transaction t){

        try{

            if(t!=null){
                if(t.isActive()){
                    //?
                    t.commit();
                }

            }

            if(s!=null){
                if(s.isOpen()){
                    try{
                        s.flush();
                    }catch(Exception e){
                        System.out.println("Tried to close session and failed");
                    }
                }
            }


        }catch(Exception e){

        }
    }
share|improve this question
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.