1

I am having issues with tuning Glassfish 2.1.1 on Ubuntu 10.04 LTS 64-bit (Amazon EC2). As soon as I change the HTTP Service thread count from the default 5 value to 100, I get the following errors in the server.log.

java.net.SocketException: Too many open files

Other errors include:

[#|2011-05-19T15:41:38.034-0500|SEVERE|sun-appserver2.1|javax.enterprise.system.tools.deployment|_ThreadID=16;_ThreadName=Timer-20;_RequestID=1bd7cd3e-0011-4ebc-95e5-487b96c76b20;|"DPL8011: autodeployment failure while deploying the application : null"|#]

[#|2011-05-19T15:41:39.555-0500|WARNING|sun-appserver2.1|javax.enterprise.system.stream.err|_ThreadID=20;_ThreadName=Timer-1;_RequestID=0d9630b5-2752-4ffb-ac7c-1cf51920155a;|
java.lang.NullPointerException
    at com.sun.jbi.management.system.AutoAdminTask.pollAutoDirectory(AutoAdminTask.java:1031)
    at com.sun.jbi.management.system.AutoAdminTask.performAutoInstall(AutoAdminTask.java:329)
    at com.sun.jbi.management.system.AutoAdminTask.performAutoFunctions(AutoAdminTask.java:288)
    at com.sun.jbi.management.system.AdminService.heartBeat(AdminService.java:967)
    at com.sun.jbi.management.system.AdminService.handleNotification(AdminService.java:198)
    at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ListenerWrapper.handleNotification(DefaultMBeanServerInterceptor.java:1732)
    at javax.management.NotificationBroadcasterSupport.handleNotification(NotificationBroadcasterSupport.java:257)
    at javax.management.NotificationBroadcasterSupport$SendNotifJob.run(NotificationBroadcasterSupport.java:322)
    at javax.management.NotificationBroadcasterSupport$1.execute(NotificationBroadcasterSupport.java:307)
    at javax.management.NotificationBroadcasterSupport.sendNotification(NotificationBroadcasterSupport.java:229)
    at javax.management.timer.Timer.sendNotification(Timer.java:1237)
    at javax.management.timer.Timer.notifyAlarmClock(Timer.java:1206)
    at javax.management.timer.TimerAlarmClock.run(Timer.java:1289)
    at java.util.TimerThread.mainLoop(Timer.java:512)
    at java.util.TimerThread.run(Timer.java:462)
|#]

and 

[#|2011-05-19T16:30:40.228-0500|SEVERE|sun-appserver2.1|org.apache.jasper.servlet.JspServlet|_ThreadID=16;_ThreadName=httpWorkerThread-4949-48;_RequestID=63d6908e-cc09-4fa8-aac0-241e7582c42f;|PWC6117: File "/opt/glassfish-v2.1.1-b31g/lib/install/applications/admingui/adminGUI_war/header.jsp" not found|#]

and 

[#|2011-05-19T16:30:40.229-0500|SEVERE|sun-appserver2.1|org.apache.jasper.servlet.JspServlet|_ThreadID=17;_ThreadName=httpWorkerThread-4949-46;_RequestID=869579eb-887d-4dc4-b0fc-edc4e41755a7;|PWC6117: File "/opt/glassfish-v2.1.1-b31g/lib/install/applications/admingui/adminGUI_war/homePage.jsp" not found|#]

Googling I found the follow resources:

http://felipeferreira.net/?p=873 http://www.netadmintools.com/art295.html

My /etc/security/limits.confg has the following configuration. And I have change the tcp settings as noted in http://mariosgaee.blogspot.com/2011/04/glassfish-211-on-linux-performance.html

*       soft    nofile          65535
*       hard    nofile          65535
*       soft    stack           unlimited
*       hard    stack           unlimited

/proc/sys/fs/file-max has a value of 762655 (I did not change this) but I did add to /etc/sysctl.conf as 'fs.file-max = 762655'

ulimit output

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 20
file size               (blocks, -f) unlimited
pending signals                 (-i) 16382
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Any ideas what may be causing this issue? Thanks in advance!

Improve this question
1
  • I would upgrade to 3.1 before you do this further glassfish.java.net/public/downloadsindex.html#top. Usually when you have an error or problem with an old version of some software, you should use the latest stable released version, that might have the problems fixed. Then if you still have the problem, then ask the question again.
    – lsd
    May 19, 2011 at 21:46

3 Answers 3

Reset to default
1

To see actual limits of running process you may use /proc , just get pid of your glassfish/java process and look at cat /proc/$PID_OF_PROCESS/limits It should be "Max open files" there. Also you can monitor number of open files with "lsof -p".

Improve this answer
2
  • Huh, so the "Max open files" value was 1024 and not the value of 65535 displayed by 'ulimit -a' even when I 'su glassfish'. Any idea why? Anyways, I changed my startup script to set 'ulimit -n 65535' and my issues went away. Thanks for the tip!
    – Hoon
    May 20, 2011 at 14:28
  • I think you dealing with PAM there , because it's PAM should apply limits from /etc/security/limits.conf . So it seem when you call su - PAM apply pam_limits.so , and when starting application - not. You can look deeply in /etc/pam.d/* . Usually me personally set limits right in applications init.d files.
    – Ruslan
    May 20, 2011 at 16:10
0

Continuing the previous answer: 1. Did you restart your glassfish server after changing limits.conf? 2. What used does glassfish run under? Do it's limits match what you set in limits.conf? Sometimes startup scripts will call ulimit for you, overriding what you set in limits.conf 3. You mention your run Ubuntu 10.04. Check and see if your pam setup load pam_limits.so, otherwise your limits.conf will be ignored

Improve this answer
2
  • I did restart glassfish which runs under a glassfish user. The ulimits were identical when I did a su glassfish. The pam_limits.so is loaded in the /etc/pam.d/login. Thank you for your answer, however I did figure it out with the help of Ruslan's answer.
    – Hoon
    May 20, 2011 at 14:19
  • Did you check pam.d/su and pam.d/sudo? They might not be loading pam_limits, and as a consequence, your limits.conf is ignored. Try checking out if the start script uses su or sudo and look in the corresponding pam module
    – katriel
    May 20, 2011 at 21:23
0

Try to add the line in /etc/pam.d/common-session : 

session required pam_limits.so

and reboot can be required.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .