Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Currently I am trying to come up with a naming convention for files and folders, essentially a method of spreading out 4 million+ images on a server into various directories or whatever.

What I am trying to figure out is my best method of approach here. Not so much in the sense of breaking all my images into groups if need be, or renaming them to prevent collisions down the road, or any of that. I'm not looking for answers on that part; I'll come up with a means to do that somehow . What I am trying to get an answer on is what would be the best way of building out a folder structure to break these images apart, what would be good naming convention to follow in renaming not only the files now but naming them in the future. As well as whats a good naming convention to follow with the folders?

I only ask this cause I am under the impression that taking a lone folder, be it on a single server or spanned across a cluster grid or cloud style (which grid is in the plans, just not on the current budget) isn't the best approach as it will cause extra load on the read/write times, and whatever goes on in to see if a file exists and then serving it, in time.

I know this seems like a broad question. But its ultimately about the naming conventions and storage conventions to maintain an optimized environment.

By naming conventions I'll give an example. Facebook, when you look at its images the file name is something that resembles a GUID but not quite. But I know in that convention is some logic as well. SO again kinda open, cause in the end I have no idea exactly what I am asking here or even if I am asking it right, but I am hoping someone can guide me in the right direction.

share|improve this question

1 Answer 1

up vote 2 down vote accepted

None.

  • Use GUILD's as file names. Use them to also form a folder structure (first 2 bytes, then 32 bytes etc.).
  • Use a database to map names to GUID's.

Then you can move parts of the storage around (the folder hierarchy does not even have to be a hierarchy) and renames cost nothing (name on file stays the same). You can also deal with double names easily - they simply do not happen on the storage side.

At the end, when you manage this amount of items, noone is going to look at them one by one. You will start having ownership, tagging etc. anyway, and that needs a database. And real names are a pain then. Get rid of them, use identity names (I.e. names that just say "this is item reference nubmer X), which GUID's do.

share|improve this answer
    
Oh Im intent on using a database to store such attributes from file ownership, to everything you expressed and more. My concern steams from talking to several linux/windows developers who work with large heaps of data where theres anything from 1 large file thats a terabyte in size, to several million, and from them I have learned there is generalize constraints storage wise that these systems werent meant to handle. Example I have hade 2500 images in a linux directory and only 1500 would list, and if by some shere act of luck it did list everything it took 5 minutes to load. So these are –  chris Sep 6 '12 at 7:19
    
the perdictiments I think, I may be very wrong and all else. that I might be facing, and what I am trying to find out here, and if it is the case then how do I over come said limitations and build with the future in mind as opposed to getting snowballed a couple months from now with problems that arent so easy to fix –  chris Sep 6 '12 at 7:20
1  
Well, point being - once you ahve a structure like i say, it is easy in the database to havea prefix to server + folder mapping, so you can move individual folders to different structure. And at least windows is happ with a million files in a folder as long as you don't use file explorer on that ;) –  TomTom Sep 6 '12 at 18:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.