3

I have a production squid server that was having some issues serving content and reporting that it was out of file descriptors. I was successfully able to increase it from 1024 (default) to 4096 and it seemed to resolve my errors in the log. I was still seeing response code 0 and 0 bytes received for some calls that were not cached and this leads me to believe that in a peak volume (boot storm) that my file descriptor count is still too low.

I have read some posts already and the setting can be set high to something like 24k, 40k, or even 70k. With this being a dedicated squid box I am not worried about other processes/users competing for file descriptors system-wide, but I'd really like to know what the best practice is for doing a rough calculation of how many file descriptors i should configure for ulimit -n.

In my configuration, I have a maximum of 3000 client-side TCP connections, a maximum of 3000 server-side TCP connections, and a few log files that are configured by default in the squid config (cache.log, squid.log). Is it as simple as saying that I should set my ulimit -n to 3000 + 3000 + 2 + (some overhead amount)? For a lack of documentation on the matter I'll probably set it to 24k just to never have to deal with it, but I prefer having a best practice formula to follow - just like with apache2 you can calculate memory needed for how many requests you want to be able to handle simultaneously.

Edit: Forgot to mention that I am not writing these cached files to disk, they are staying in memory. It's a few hundred files (<5 MB total) website that is the only page that gets loaded through this, so that's why I omitted the disk read/write file descriptors.

2 Answers 2

3

In the worst case scenario each request to squid server requires three file descriptors;

  1. A descriptor for the client-side connection
  2. Another for the server-side connection in case it is not cached.
  3. Third one for the file to read hit or cache the miss.

Then there are overheads including log files, any inter-process communication, e.g., helpers and idling connections. So as a rough estimate you need three file descriptors for each incoming TCP connection and then factor in any overheads to that.

6
  • I totally forgot the disk read/write consideration! I edited my post that I am not caching them to disk, and instead to memory. There shouldn't be much in helpers/inter-process communication should there? I have no authorization/redirection directives in my squid.conf - it's just caching pages. May 9, 2016 at 17:55
  • 1
    Even if you are not using any actual disk, the memory based cache still uses the same number of file descriptors. Yes there should not be many file descriptors needed for IPC. I would still recommend the following formulae. Theoretical Peak Incoming Connections * 3 + 1024 as the number of file descriptors required for smooth performance.
    – Nasoo
    May 9, 2016 at 17:57
  • 1
    I'm not totally sure how the idling connections work. I am grep'ing netstat to see the server-side (~70) and client-side (~700) connections and then when I ask squidclient, i get a number more like 350 when i run this: squidclient -p 80 mgr:info | grep 'Number of file desc currently in use'. Any idea why those would be so different? May 9, 2016 at 17:57
  • 1
    It looks like you have quite a good cache hit ratio? The number 350 and the limited number of server side connections means that the number 3 that I have calculated for you above is more like 2.1-2.4, in your case. But my number was based on the worst case scenario and I think in production server we need to cater for that.
    – Nasoo
    May 9, 2016 at 18:01
  • Yes. Most of my files are static content and I get TCP_HIT. The dynamic calls to the application server are much fewer but get called periodically and I get TCP_MISS. That's why there are much fewer server-side connections compared to client-side connections. I imagine in a boot storm scenario when every client is getting fresh content that it would be more 1:1 for client-side and server-side connections. For the cache hit rate, could they be responded so quickly that it can keep up with the responses and that is why the file descriptor count is so low? I would expect it to include sockets too May 9, 2016 at 18:12
2

In a search to find more "best practice" information, Nasoo was dead on with how to calculate the number of open files you should configure. I found there are intricacies with how browsers handle downloading files in parallel, so 3000 clients actually talk on about 25-30 sockets each to download the full webpage and dynamic content. Some of this is dependent on how the browsers download in parallel as well as how the javascript APIs handle downloading dynamic content.

So while I cannot accurately determine a proper number without tons of additional testing, I also stumbling across a manual that states that 256 file handles can be set up for every 4MB of RAM. So that should be more than enough, and even half of the 8GB of RAM i have for this box would be overkill.

http://www.tldp.org/LDP/solrhe/Securing-Optimizing-Linux-RH-Edition-v1.3/chap6sec72.html

EDIT: I also started doing some logging of file descriptor usage to an RRD file once per minute via cronjob. It's a pretty basic bash script that logs it all out and you can generate pretty handy graphs without a monitoring server or anything. If anyone is interested in it, let me know and I'll make a gist out of it.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .