2

I'm using CentOS 6.3 64bit, and having trouble with increase FD limitation: SSH not working after changed the limitation, every-times I logged on my server, after typed password, then "connection closed by remote server".

It's a remote server without any "rescue mode" so there's no way to do a local login to get any error log. I also tried ulimit -SHn before this issue happened but not work as well.

Now I have to restore snapshot every-time this issue happened, and still have no idea what's wrong with my server.

Since I'm working on a high performance server handling millions of requests per second, I need a very high FD limitation, here is my configuration:

/etc/security/limits.conf
* hard nofile 2000000
* soft nofile 2000000


/etc/security/limits.d/90-nproc.conf  
*          soft    nproc     1024


/etc/ssh/sshd_config
Protocol 2
SyslogFacility AUTHPRIV
PasswordAuthentication yes
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
X11Forwarding yes
Subsystem       sftp    /usr/libexec/openssh/sftp-server


/etc/pam.d/sshd
auth       required     pam_sepermit.so
auth       include      password-auth
account    required     pam_nologin.so
account    include      password-auth
password   include      password-auth
session    required     pam_selinux.so close
session    required     pam_loginuid.so
session    required     pam_selinux.so open env_params
session    optional     pam_keyinit.so force revoke
session    include      password-auth
session    required     pam_limits.so   


/etc/sysctl.conf
fs.file-max = 2000000
fs.nr_open = 2000000 
4
  • You are perhaps reaching your process limit before reaching the file limit. That is, whatever is opening the files is also forking processes up to the limit and this prevents you from getting a login shell. Raise the process limit. Apr 9, 2013 at 9:36
  • What is "process limit"? I'm not running any process on my server using huge amount of FDs for now.
    – Shane Hou
    Apr 9, 2013 at 9:52
  • Please provide more details about what processes (services/daemons) are running on the server. Apr 9, 2013 at 10:12
  • Just a new deployed system. If I use "* hard nofile 1024 * soft nofile 1024" after reboot everything is fine, so I don't think there's any process/service caused this issue.
    – Shane Hou
    Apr 9, 2013 at 10:16

2 Answers 2

1

2^20 (approximately one million) is the maximum file descriptor limit (NR_OPEN) that's supported under Linux. It would take many changes, including recompiling the kernel, to raise it. It would also break lots of user space applications that rely on the limit being low.

I suspect you're completely barking up the wrong tree though. There's no correlation between request rate and number of file descriptors needed.

3
  • Do you have any evidence or example about "It would also break lots of user space applications that rely on the limit being low."?
    – Shane Hou
    Apr 9, 2013 at 15:36
  • @ReckHou There was a patch to the Linux kernel to raise the limit and it was removed because of issues with user space applications. The main problem is that applications, typically before calling exec, close all descriptors that they might have open. If the limit is raised, this can take a very long time. One python script went from taking minutes to taking hours. This type of code is buried in libraries. (Think of posix_spawn(3p) and system(3).) Apr 9, 2013 at 15:41
  • 2^20 limitation is not the end of world, see my answer below. But you also made a useful point, +1 for your answer.
    – Shane Hou
    Apr 12, 2013 at 8:51
0

Finally I found a way to overcome the 2^20 limitation without modify kernel source code:

In /etc/sysctl.conf:

fs.nr_open = 10485760

Modify fs.nr_open over 2^20 can solve this issue.

Moreover, the 2^20 limit is not the end of world. There's one person I know have successfully build a CentOS server very close to this limit, his record is 1025216 for now. If he use higher mem with correct configure, no doubt he can overcome this limitation easily.

Here is what he has done(written by Chinese but you can see his config):

http://www.blogjava.net/yongboy/archive/2013/04/09/397559.html http://www.blogjava.net/yongboy/archive/2013/04/11/397677.html http://www.blogjava.net/yongboy/archive/2013/04/10/397631.html http://www.blogjava.net/yongboy/archive/2013/04/09/397594.html

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .