4million+ images on a single LAMP stack server

Question

Currently I am trying to come up with a naming convention for files and folders, essentially a method of spreading out 4 million+ images on a server into various directories or whatever.

What I am trying to figure out is my best method of approach here. Not so much in the sense of breaking all my images into groups if need be, or renaming them to prevent collisions down the road, or any of that. I'm not looking for answers on that part; I'll come up with a means to do that somehow . What I am trying to get an answer on is what would be the best way of building out a folder structure to break these images apart, what would be good naming convention to follow in renaming not only the files now but naming them in the future. As well as whats a good naming convention to follow with the folders?

I only ask this cause I am under the impression that taking a lone folder, be it on a single server or spanned across a cluster grid or cloud style (which grid is in the plans, just not on the current budget) isn't the best approach as it will cause extra load on the read/write times, and whatever goes on in to see if a file exists and then serving it, in time.

I know this seems like a broad question. But its ultimately about the naming conventions and storage conventions to maintain an optimized environment.

By naming conventions I'll give an example. Facebook, when you look at its images the file name is something that resembles a GUID but not quite. But I know in that convention is some logic as well. SO again kinda open, cause in the end I have no idea exactly what I am asking here or even if I am asking it right, but I am hoping someone can guide me in the right direction.

Answer 1

None.

• Use GUILD's as file names. Use them to also form a folder structure (first 2 bytes, then 32 bytes etc.).
• Use a database to map names to GUID's.

Then you can move parts of the storage around (the folder hierarchy does not even have to be a hierarchy) and renames cost nothing (name on file stays the same). You can also deal with double names easily - they simply do not happen on the storage side.

At the end, when you manage this amount of items, noone is going to look at them one by one. You will start having ownership, tagging etc. anyway, and that needs a database. And real names are a pain then. Get rid of them, use identity names (I.e. names that just say "this is item reference nubmer X), which GUID's do.