0

We use Tika for PDF text content extraction for searching. I'm seeing heavy use of temporary files that seem to now run us out of file handles.

We're on CentOS 5.5. For our search system, we had previously bumped up open file handles (/etc/sysctl.conf) fs.file-max = 65535

When looking at lsof for my process, I see lots of files marked as DEL (deleted memory mapped files). If I stop submitting new documents to be indexed, these will get cleared out after several minutes.

java 11105 root DEL REG 104,2 1278402 /tmp/+~JF4155000471009101661.tmp

Can I tune the kernel to clear out these DEL files sooner, or should I increase the number of file handles? If so, how high can we safely go? I have sufficient memory to be able to give more to the kernel.

Thanks

David

Improve this question

2 Answers 2

Reset to default
1

I couldn't find a good reference for how increasing fs.file-max impacts the system. Presumably that makes your kernel consume more memory as the structure to track open files gets larger and larger. I suggest increasing to a larger number like 128K and keeping an eye on /proc/sys/fs/file-nr to see how many files are actually used.

Improve this answer
2
  • Not sure about other impact besides bigger kernel structures, but huge values of fs.file-max are pretty common on Oracle servers with no ill effects that I'm aware of (see download.oracle.com/docs/cd/B28359_01/server.111/b32009/… where they suggest a fairly outrageous value of "6815744 or higher" for their example configuration)
    – voretaq7
    Jan 20, 2011 at 19:05
  • Yeah my naive assumption is that the amount of memory used by the kernel for these data structures is fairly small even if the max file size is extremely large. There may however be performance effects of the kernel having to continually traverse a very large open file list.
    – Phil Hollenback
    Jan 20, 2011 at 20:05
0

what does a few minutes of vmstat show during this high time? vmstat will show you if you're I/O bound on CPU, swap, memory. I would start there.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .