Hot answers tagged

50

These limits came from a time where multiple "normal" users (not apps) would share the server, and we needed ways to protect them from using too many resources. They are very low for high performance servers and we generally set them to a very high number. (24k or so) If you need higher numbers, you also need to change the sysctl file-max option (generally ...


7

I tested this in Linux version 2.6.18-164.el5 - Red Hat 4.1.2-46. I could see that the ulimit is applied per process. The parameter is set at user level, but applied for each process. Eg: 1024 was the limit. Multiple processes were started and the files open by each one was counted using ls -l /proc/--$pid--/fd/ | wc -l There were no errors when the ...


7

Your operating system set limits on how many files can be opened by any running application on your host. You can extend the basic values usually 1024 easily by modifying 2 configuration files: # vi /etc/sysctl.conf fs.file-max = 32000 # vi /etc/security/limits.conf youruser soft nofile 10000 youruser hard nofile 30000 The hard ...


6

to see the top 20 file handle using processes: for x in `ps -eF| awk '{ print $2 }'`;do echo `ls /proc/$x/fd 2> /dev/null | wc -l` $x `cat /proc/$x/cmdline 2> /dev/null`;done | sort -n -r | head -n 20 the output is in the format file handle count, pid, cmndline for process example output 701 1216 /sbin/rsyslogd-n-c5 169 11835 postgres: spaceuser ...


6

You could always just cat /proc/sys/fs/file-nr During the 'high load' situation to see how many file descriptors are in use. As to a maximum - it just depends on what you are doing.


5

The file-max limit that you see under proc fs is one value in struct in "./include/linux/fs.h" the struct is: /* And dynamically-tunable limits and defaults: */ struct files_stat_struct { unsigned long nr_files; /* read only */ unsigned long nr_free_files; /* read only */ unsigned long max_files; /* tunable THIS IS OUR VALUE */ }; Now in ./fs/...


4

From the ulimit builtins man page The ulimit builtin provides control over the resources available to the shell and to processes started by it on systems that allow such control. Your lsof command lists all of the open files for all processes for all users on the system. You are not comparing like with like.


4

If the file descriptors are tcp sockets, etc, then you risk using up a large amount of memory for the socket buffers and other kernel objects; this memory is not going to be swappable. But otherwise, no, in principle there should be no problem. Consult the kernel documentation to try to work out how much kernel memory it will use, and/or test it. We run ...


4

On the process take a look at the lsof -p <pid> Notably you're looking for FD 13 i.e. from one of my httpd deployments. httpd 10865 apache 13w REG 8,5 1113187 59310687 /var/log/httpd/some_site.log I'd look at contention with obtaining an exclusive lock on the file, noted in FD 13 in this case, if other processes are contending for ...


4

In Linux, you can see if that option has changed the limits of the process by running: cat /proc/<PID>/limits Where <PID> is the process ID of nginex. Test on a development environment. If that option is not changing the limits or if you have a too restrictive hard limit you need to change the nofile in the /etc/security/limits.conf file.


4

It really depends on where the buffer is: if the application uses its own logging buffer then there's no way of forcing a flush. If the buffering is done by the C library then you can use LD_PRELOAD to disable the buffering. Assuming that the program is using fopen() to open its log file you can do something like this: #define _GNU_SOURCE 1 #include <...


3

Yes, the ulimit facility limits the number of open files and many other things like stack size, core dump size, etc.


3

The ulimit is for filehandles. It applies to files, directories, sockets, pipes epolls, eventfds, timerfds etc etc. At any point during the processes startup the limits might have been changed. Visit /proc/<pid>/limits and see if the values have been altered.


3

I'll answer my own question: the Linux 2.6 kernel sets the open file limit to 10% of the available memory. Source: http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=blob;f=fs/file_table.c 488 void __init files_init(unsigned long mempages) 489 { 490 int n; 491 492 filp_cachep = kmem_cache_create("filp", sizeof(struct ...


2

I am not personally aware of any best practices. It's somewhat subjective depending on system function. Remember that 1024 you're seeing is a per-user limit and not a system-wide limit. Consider how many applications you run on this system. Is this the only one? Is the user that runs this application doing anything else? (IE do you have humans using this ...


2

You have to add the ulimit line in the squid init script as well as increase max_filedescriptors in squid.conf. Those are the two essential steps. You no longer have to compile squid from source to increase this limit. That was something you had to do with really old squid versions.


2

A bash(1) builtin, ulimit defines the maximum number of open files per process. This is not a system-wide setting in any way.


2

grep -lr pam_limits /etc/pam.d? If it's not return common-session, run the following command: echo -e "session\trequired\t\tpam_limits.so" > /etc/pam.d/common-session logging out and in again to see it works.


2

I'd be surprised if your installation of Ubuntu didn't have by default more than 65536 available. Check your current setting with $ sudo sysctl fs.file-max The general rule of thumb is that you can increase the fs.file-max parameter by 64 for every 1MB of RAM. e.g. 2 gigabytes = 2048 * 1MB = 2048 * 64 = 131072 That'...


2

@oligofren I also carried out some testing to determine how "ulimits -Sn" for "open files" was enforced. Like the poster Chosen mentioned in link, the ulimit for "open files" is indeed applied per process. To see what the process's current limits are: cat /proc/__process_id__/limits To determine how many files a process has open, you need to use the ...


2

The problem was in the launching script of daemon. It was using setuidgid to run the daemon under headless user.Looks like setuidgid will not install the resource limits which are set in limits.conf while changing user/group for the process. A daemon should take care of setting resource limits for itself via its launching scripts. By setting max FD limits ...


2

I found the answer to my question of "how to fix this scenario". I don't know all the details of how this came to be, but I know enough to put out an answer. Short answer: unmounting the disk, running chkdsk -f on it, and mounting back again solves and prevents the issue from reoccurring. As an alternative, creating a new disk (remember we're on AWS) and ...


2

In most cases (obviously not in your case) the reason will be that you are running out of iNodes. To check this run df -i: Filesystem Inodes IUsed IFree IUse% Mounted on [...] 25600 25600 0 100% /foo Here you can see the use of iNodes is 100%. Bad news is, according to http://superuser.com/questions/585641/...


2

In the worst case scenario each request to squid server requires three file descriptors; A descriptor for the client-side connection Another for the server-side connection in case it is not cached. Third one for the file to read hit or cache the miss. Then there are overheads including log files, any inter-process communication, e.g., helpers and idling ...


2

In a search to find more "best practice" information, Nasoo was dead on with how to calculate the number of open files you should configure. I found there are intricacies with how browsers handle downloading files in parallel, so 3000 clients actually talk on about 25-30 sockets each to download the full webpage and dynamic content. Some of this is dependent ...


1

2^20 (approximately one million) is the maximum file descriptor limit (NR_OPEN) that's supported under Linux. It would take many changes, including recompiling the kernel, to raise it. It would also break lots of user space applications that rely on the limit being low. I suspect you're completely barking up the wrong tree though. There's no correlation ...


1

You can read/set the resource limits of a running process with prlimit(1), part of util-linux: sudo prlimit --pid PID --nofile 8192:16384


1

You can find the limit for a process with pid in /proc/<pid>/limits and there is an entry Max open files.


1

It seems like your reasoning is something like, "I have to lower that limit so I don't run out of precious descriptors". The truth is exactly the reverse -- if your server ran out of file descriptors, you need to raise that limit from 1,024 to something larger. For a realistic glassfish implementation, 32,768 is reasonable. Personally, I always raise the ...


1

So it seems MongoDB 1.6.5 doesn't work with the limit nofile line. You need to put ulimit -n X (where X is the file descriptors you want) just after the script keyword. script ulimit -n 60000 PORT=27027 ... I guess the method using the pre-start block is for later versions of MongoDB. Also for note, MongoDB has a hard limit of 20,...



Only top voted, non community-wiki answers of a minimum length are eligible