Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

I have 3 different folders containing particular information about sales orders. Everything was working fine but a few days ago I started having an issue about the amount of subdirectories in each of those main folders (orders above 32K).

My temporary solution was to move the oldest data to a backup and remove it from the production environment but I would really like to have it there, so my question is:

What options do you recommend to store a structure where I can save incremental subfolders without hitting the maximum? I am on a Ubuntu server box with ext3

It looks something like

-tmp/
--order_1/
--order_2/
...
--order_32000/
...
-imgs/
--order_1/
--order_2/
...
--order_32000/
...
-hd_imgs/
--order_1/
--order_2/
...
--order_32000/

inside each order_xx folder live around 1 to 30 files.

share|improve this question
1  
How large are these files, and what is the meaning of the top level directories ( tmp, imgs, hd_imgs )? –  psusi Feb 15 '12 at 15:52
    
@psusi no real way to say the file size it varies a lot but some files can be up to 200M. tmp - every file that the user uploads before he is logged in which then gets moved to imgs with the propper order number which then gets moved to hd_imgs after we have worked on the file –  Mauro Morales Feb 15 '12 at 16:58
add comment

2 Answers 2

up vote 2 down vote accepted

It sounds to me like you need a (real) database (as opposed to the filesystem), and some development time to make a front end for it. Investigate MongoDB or Postgres.


If you need a faster solution, try breaking up your orders by time: Store them in a hierarchy like [year]/[month]/order_###### (you can keep using serial order numbers if you want, or compose the order number as YYYYMM##### so it's easier to find in the system later without having to do searches within the directory hierarchy).

This will work as long as the number of orders in a month is less than about 30,000 or so. The next limit you will hit is the filesystem inode limit though, and the only solution there is a new filesystem (or splitting your data across several filesystems). Take a look at df -i on your system today, and remember that every file and directory will chew up one more inode. Eventually you'll run out.

share|improve this answer
    
just checked my inodes, have just 1% used so I'm thinking that shouldn't be a problem. Also the year/month/order solution seems very good for me since we don't have that many sales per month. Thanks a lot! –  Mauro Morales Feb 15 '12 at 16:07
1  
If you only have 32K*3(tmp,imgs,hd_imgs) directories and there aren't that many files under them you're not really in danger of hitting the inode limit (it's usually into the hundreds of millions of files) -- More something to be aware of as you scale up (say around the time you may be hitting 32K orders in a month), by which time hopefully you'll have a better database in place :) –  voretaq7 Feb 15 '12 at 16:11
    
thanks for the illustration @voretaq7 ... and agreed, that'd be a good problem to have ^__^ –  Mauro Morales Feb 15 '12 at 16:20
add comment

You could upgrade to ext4 to get around the 32k limit.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.