For single project, which uses a lot of images( ± 1k-3k per day) I use custom PHP script to process images and save them to server.

Currenty PHP script processes them and saves in the following folder structure: enter image description here So the max lvl is 5. But currently I think how does following directory structure influences on inodes count? Should I put the lvl down to 2 or 3?

  • Do you have control over the filesystem it is being saved to? (Is it your own server, or provided by someone else?) You can control the number of inodes created when you format an ext3 partition. Or you can look at other filesystems that are better suited to storing huge numbers of small files. – Grant Apr 5 '12 at 14:08

Every directory entry is an inode entry as well.

16**5 = 1,048,576

16**3 = 4,096

One year of images... 2,000 * 365 = 730,000.

That will give you roughly 178 per directory with a depth of 2. With a depth of 5, just over half utilization at all. Three sounds good to me. Maybe if you're in it for a decade I'd go to 4.

| improve this answer | |
  • Way better than what I was going to post. Nice. – Kyle Smith Apr 5 '12 at 13:53

Every file and directory takes up an inode, but I don't think you're at much risk of exhausting inodes on a decent-sized filesystem -- at 5 levels deep, your directory tree will be taking up 16^5 or 1048576 inodes, which is a fair number, but not likely to overwhelm your filesystem. On the other hand, the number of files you store might cause some problems...

It's a bit wasteful to only be using one hex digit per level, though -- I usually go to three digits per level (so ab3/4f5/5e1/...) as that puts 4096 directories at each level, which is well within the realm of sensible performance from a decent filesystem (don't go to 4, as extN only allows 32000 subdirectories in a single directory).

Then, of course, there's the whole world of filesystems without fixed inode limits, and better performance in the face of massive numbers of files per directory, and no limit on the number of subdirectories...

| improve this answer | |
  • Why is ab3/... less wasteful than a/b/3? The only thing I can think of is reducing the total number of bytes used per filename, but the same amount of space is allocated for the name regardless... any savings are "false". – Jeff Ferland Apr 5 '12 at 14:00
  • You consume less directory entries to get the same degree of bucketing. – womble Apr 5 '12 at 14:03
  • @JeffFerland For each directory, the filesystem has to keep track of a bunch of things besides it's filename. Using fewer directories means it has to keep track of fewer things like the permissions on each directory. – Grant Apr 5 '12 at 14:06
  • It's the same number of directories... whether a/b/c/defghijklmno... or abc/def/ghi/jklmno. The length of the prefix doesn't determine the depth of directories you create. Prefix length just determines the number of directory entries at a level. Thus abc/defghijklmo creates the same number of entries as a/b/c/defghijklmno. Either method represents 16**3 entries. – Jeff Ferland Apr 5 '12 at 16:24
  • You're not comparing apples with apples. a/b/c/<file> means you'll have N/16^3 files per bucket (where N is the number of files), whereas with abc/def/ghi/<file> you'll have N/4096^3 files per bucket. If you only want N/16^3 files per bucket, you can just use abc/<file>, which gives you the same level of spread with less inodes chewed up by directories. – womble Apr 5 '12 at 16:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.