1

I am running a SmartMachine on Joyent. I beleive these are some sort of virtual machine running Solaris. We have a web application on this machine running on Apache, PHP, and MySQL. It handles our moderate amount of traffic very well. However every night since we've gone live. The site will begin to return 403 Forbidden errors until Apache is restarted. I quick look at Apache's error log reveals the following:

[Tue Oct 26 23:13:00 2010] [error] server reached MaxClients setting, consider raising the MaxClients setting
[Wed Oct 27 13:09:40 2010] [error] (24)Too many open files: Cannot open SSLSessionCache DBM file `/var/run/ssl_scache' for reading (fetch)
[Wed Oct 27 13:09:40 2010] [error] (24)Too many open files: Cannot open SSLSessionCache DBM file `/var/run/ssl_scache' for writing (store)
[Wed Oct 27 13:09:40 2010] [error] [client 98.25.133.36] PHP Fatal error:  Unknown: Failed opening required '/home/jill/web/content/index.php' (include_path='.:/opt/local/lib/php') in Unknown on line 0, referer: https://[redacted]/presentations/present#
[Wed Oct 27 13:09:42 2010] [error] (24)Too many open files: Cannot open SSLSessionCache DBM file `/var/run/ssl_scache' for reading (fetch)
[Wed Oct 27 13:09:42 2010] [error] (24)Too many open files: Cannot open SSLSessionCache DBM file `/var/run/ssl_scache' for writing (store)
[Wed Oct 27 13:09:42 2010] [crit] [client 68.193.4.75] (24)Too many open files: /home/jill/web/content/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable, referer: https://[redacted]/presentations/present#
[Wed Oct 27 13:09:43 2010] [error] (24)Too many open files: Cannot open SSLSessionCache DBM file `/var/run/ssl_scache' for reading (fetch)
[Wed Oct 27 13:09:43 2010] [error] (24)Too many open files: Cannot open SSLSessionCache DBM file `/var/run/ssl_scache' for writing (store)
[Wed Oct 27 13:09:43 2010] [crit] [client 72.28.224.201] (24)Too many open files: /home/jill/web/content/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable, referer: https://[redacted]/presentations/present#
[Wed Oct 27 13:09:44 2010] [error] (24)Too many open files: Cannot open SSLSessionCache DBM file `/var/run/ssl_scache' for reading (fetch)
[Wed Oct 27 13:09:44 2010] [error] (24)Too many open files: Cannot open SSLSessionCache DBM file `/var/run/ssl_scache' for writing (store)
[Wed Oct 27 13:09:44 2010] [crit] [client 72.28.224.201] (24)Too many open files: /home/jill/web/content/.htaccess pcfg_openfile: unable to check htaccess file, ensure it is readable, referer: https://[redacted]/presentations/present#

The last three lines then repeat for every request made to the server. I'm really at a loss to how to prevent this from happening. I tried increasing the number of files that can be open using prctl, but I must not be using it right because prctl returns 1.02K for basic when I tried to set it to 65.5K. I'm not even sure if that is a sound solution:

prctl -i process -n process.max-file-descriptor `pgrep httpd`
process: 18284: /opt/local/sbin/httpd -k start
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
process.max-file-descriptor
        basic           1.02K       -   deny                             18284
        privileged      65.5K       -   deny                                 -
        system          2.15G     max   deny                                 -

process: 18285: /opt/local/sbin/httpd -k start
NAME    PRIVILEGE       VALUE    FLAG   ACTION                       RECIPIENT
process.max-file-descriptor
        basic           1.02K       -   deny                             18285
        privileged      65.5K       -   deny                                 -
        system          2.15G     max   deny                                 -

So what is the best way to trackdown and fix a problem like this?

UPDATE

here is the output of pfiles for the root httpd process.

[root@fe5txrad ~]# pfiles 18269
18269:  /opt/local/sbin/httpd -k start
  Current rlimit: 1024 file descriptors
   0: S_IFCHR mode:0666 dev:304,8 ino:3020727013 uid:0 gid:3 rdev:13,2
      O_RDONLY
      /dev/null
   1: S_IFCHR mode:0666 dev:304,8 ino:3020727013 uid:0 gid:3 rdev:13,2
      O_WRONLY|O_CREAT|O_TRUNC
      /dev/null
   2: S_IFREG mode:0640 dev:182,65550 ino:362926 uid:0 gid:0 size:20551848
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE
      /var/log/httpd/error.log
   3: S_IFDOOR mode:0444 dev:313,0 ino:38 uid:0 gid:0 size:0
      O_RDONLY|O_LARGEFILE FD_CLOEXEC  door to nscd[18176]
      /var/run/name_service_door
   4: S_IFSOCK mode:0666 dev:311,0 ino:43693 uid:0 gid:0 size:0
      O_RDWR|O_NONBLOCK FD_CLOEXEC
        SOCK_STREAM
        SO_REUSEADDR,SO_KEEPALIVE,SO_SNDBUF(49152),SO_RCVBUF(49152)
        sockname: AF_INET 0.0.0.0  port: 80
   5: S_IFSOCK mode:0666 dev:311,0 ino:42512 uid:0 gid:0 size:0
      O_RDWR|O_NONBLOCK FD_CLOEXEC
        SOCK_STREAM
        SO_REUSEADDR,SO_KEEPALIVE,SO_SNDBUF(49152),SO_RCVBUF(49152)
        sockname: AF_INET 0.0.0.0  port: 443
   6: S_IFIFO mode:0000 dev:301,0 ino:8763127 uid:0 gid:0 size:0
      O_RDWR|O_NONBLOCK FD_CLOEXEC
   7: S_IFIFO mode:0000 dev:301,0 ino:8763127 uid:0 gid:0 size:0
      O_RDWR FD_CLOEXEC
   8: S_IFREG mode:0640 dev:182,65550 ino:362927 uid:0 gid:0 size:1450493
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE FD_CLOEXEC
      /var/log/httpd/access.log
   9: S_IFREG mode:0644 dev:182,65550 ino:369102 uid:1000 gid:1000 size:528239971
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE FD_CLOEXEC
      /home/jill/logs/access_log
  10: S_IFREG mode:0644 dev:182,65550 ino:369102 uid:1000 gid:1000 size:528239971
      O_WRONLY|O_APPEND|O_CREAT|O_LARGEFILE FD_CLOEXEC
      /home/jill/logs/access_log
  11: S_IFREG mode:0644 dev:308,39 ino:3386326219 uid:0 gid:0 size:0
      O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE FD_CLOEXEC
  12: S_IFREG mode:0644 dev:308,39 ino:3088492558 uid:0 gid:0 size:0
      O_WRONLY|O_CREAT|O_EXCL|O_LARGEFILE FD_CLOEXEC
      advisory write lock set by process 7350
  13: S_IFSOCK mode:0666 dev:311,0 ino:6452 uid:0 gid:0 size:0
      O_RDWR FD_CLOEXEC
        SOCK_STREAM
        SO_SNDBUF(16384),SO_RCVBUF(5120)
        sockname: AF_UNIX 
2
  • Do you have lsof?
    – PP.
    Oct 27, 2010 at 13:56
  • lsof is not installed but there is pfiles which is similar. I added the output of pfiles for the root httpd process to my question.
    – joshwbrick
    Oct 27, 2010 at 15:14

6 Answers 6

1

You did not mention how many connections you have as a maximum when you start getting these errors.

You can use tools/commands like:

ps -ef | grep apache | wc -l

lsof -p <apache pid>

netstat -anp | grep 80 | grep -i ESTABLISHED | wc -l

The first command tells you how many apache process on your system. The second command tells you how many opened files/connections apache process has. Of course, you need to replace with the real value. The third command tells you how many established connection you have.

1

I once got burned by something similar. One additional thing that you may want to check - that PHP is not sucking up the files either. If PHP is storing session information on-disk, a sudden overload of requests can suck up all the files too.

2
  • I think you may be on to something here. Our PHP sessions are set to not be garbage collected until after an hour so with the amount of traffic we have and the length of the session we could be having a buildup of open session files.
    – joshwbrick
    Oct 27, 2010 at 15:19
  • Taking a quick look inside the PHP cache directory will tell you if it is filling up too quickly.
    – sybreon
    Oct 27, 2010 at 15:31
1

If you have good memory and CPU's you can increase the no. of opened files limit via ulimit command

ulimit -n 32667 (May be higher than this too)

0

I think the error "Too many open files" might be a misnomer. There are various references to inability to read/write to /var/run which is virtual memory in solaris land.

A few questions:

Have you configured apache correctly to handle the load for the amount of resources available? How many child threads are you allowed to fork? How much memory does each thread take? How much swap have you configured?

A handy one liner to find out how much memory httpd processes are using in total:

for i in `pgrep httpd`; do pmap -x $i | nawk '/total/ {a+=$4} END { print a /1024" MB" }'; done | nawk '{a+=$1} END { print a " MB" }'

I'm sure someone could put this more eloquently, but it works for me. hth.

0

If you have good memory and CPU's you can increase the no. of opened files limit via ulimit command

ulimit -n 32667 (May be higher than this too)

0

I know this is an old question, but on another question there was a very apt response to the problem, I wonder if it can apply to you too:

https://superuser.com/a/829413/196842

Basically, Xdebug is leaking file descriptors for connection listeners (sorry if this is technically incorrect, that's the idea) when the debugging client is not open. To solve the problem, make sure the debugger client is open when the remote debugger starts a debugging session. Of course, a better solution would be a fix to the bug by the developers.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .