1

We have a system that uses Jira, and Jira stores attachments to tickets in /opt/jira/jiraattachments. Under that directory is the project name RRT, and under that is a directory of tickets. Thus, ticket RRT1234 has its attachments under:

/opt/jira/jiraattachments/RRT/RRT1234

We have a monitoring system that fires off an alert when there are more than 30,000 items in the /opt/jira/jiraattachments/RRT directory. Considering we have 900,000 Jira tickets, this isn't really a surprise.

On a programming level, I don't really see an issue. Jira doesn't open the entire directory and keep all those directories open. In fact, the structure is arranged, so that Jira can immediately find the directory that contains the attachments.

However, on an OS level, is there a problem with a single directory containing over 32K files? I can see problems writing shell scripts and attempting to parse this many files. I can see problems with ls trying to read and sort all of those files. I know back in the MS-DOS 2.x days, a directory couldn't have more than 512 entries. But we're no longer in the disco era. I can't see an OS stumbling on something like this.

$ uname -r
2.6.18-238.el5

$ df -kT .
Filesystem    Type   1K-blocks      Used Available Use% Mounted on
10.10.136.125:/vol/jira_prod
               nfs    83886080  58621352  25264728  70% /jira_prod
  • Is there an actual problem in there; or are we just spitballing what happens when you exceed reasonable limits? – Chris S Jul 29 '13 at 17:53
  • @ChrisS - I'm not an OS expert. My gut tells me that this alert is useless, but I don't have the expertise to say for sure. Some one added this alert, but we don't know who and why. Do they know something I don't. Is it a good idea to have 32K entries in a single directory? No, but, this is how Jira works. Besides, Jira doesn't sort thru these dirs. It knows the path right to the directory. The question is whether things will crash and burn when there are 32,001 entries. Do we really need this alert? – David W. Jul 29 '13 at 18:26
  • I believe this is defined primarily by the filesystem in use. – Falcon Momot Jul 31 '13 at 9:20
  • @FalconMomot I have found a blog entry of someone who ran into this very problem. Apparently ext3 file systems cannot handle more than 32K links in a directory. I find this hard to believe - even HFS+ and NTFS can handle more than 32K links per directory. – David W. Jul 31 '13 at 14:16
  • ext3 is merely journaled ext2, and ext2 is rather long in the tooth. – Falcon Momot Jul 31 '13 at 19:49
4

I can't fully explain their rationale, but can say that ext3 has a 32000 sub-directory limit. It can easily accommodate 1/4M files in a directory, more depending on your server. Listing/sorting through the direction is obviously costly, but there's no mechanism that avoids higher look-up "costs" even when you know the name of the file (indexes improve performance, but don't solve all problems).

As you'd expect, performance penalties get worse with size. Most recommendations are to keep less than 15-25k files per directory. If you're not seeing any performance problems, I wouldn't worry about it. The file system will not implode, it just gets slower for each file you add.

  • This may be it. We use ext3 on our Linux directories. If the directory can't have more than 32K, it makes sense to warn on 30K and go critical on 32K. – David W. Jul 29 '13 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.