Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

We currently store our photos in a structure like this:

folder\1\10000 - 19999.JPG|ORF|TIF (10 000 files)

folder\2\20000 - 29999.JPG|ORF|TIF (10 000 files) etc...

They are stored on 4 different 2TB D-link NASes attached and shared on our office network (\\nas1, \\nas2, and so on...)

Problems:

1) When a client (Windows only, Vista and 7) wishes to browse the let's say \\nas1\folder\1\ folder, performance is quite poor. A problem. List takes a long time to generate in explorer window. Even with icons turned off.

2) Initial access to the NAS itself is sometimes slow. Problem.

SAN disks too expensive for us. Even with iSCSI interface/switch technology.

I've read a lot of tech pages saying that storing 100 000+ files in one single folder shouldn't be a problem. But we don't dare go there now that we experience problems on a 10K level.

All input greatly appreciated,

/T

share|improve this question
    
This doesn't really answer your question, but just wanted to point something out. I spec'd out a "poor mans" SAN for roughly $3000 with 3TB of drive space and iSCSI. Granted I had to build it myself but just wanted to point that out. –  GregD Dec 10 '10 at 14:20
    
What speed is your networking gear running at? –  GregD Dec 10 '10 at 14:22
    
I don't think that a SAN is going to help other than consolidation of storage which will allow more spindles to be in a single array which should improve the performance of the storage. Beyond that you're serving files here to multiple Window clients from the sound of it. So you would still need a single server attached to the SAN serving the files via CIFs. My suggestion is if possible avoid Explorer. Use a database to index your collection and either a web or GUI to let the users find and retrieve the photo they are looking for. –  3dinfluence Dec 10 '10 at 15:05

4 Answers 4

1: yes, Explorer does not like 10.000 items in a directory. 1000 is b etter. This is NOT a file system issue - you can dsafely store 100.000 files in a folder and work with them efficiently from the command line. It is Explorer being slow.

2: Velociraptor 600gb disc. 10.000 RPM. Fast. Need more then get the money for SAS. Or wait till things get cheaper. If i need a truck, I need to pay for it.

share|improve this answer
    
FS will suffer too if too many files - with varying impact. Old novell didn't like >10000 files in one dir, collapsed performance. –  user65297 Jan 4 '11 at 20:45
    
NTFS will gladly handle 100.000 or more files without problems in a folder. Old novell had all kinds of stupdid decisions going back to times discs were small ;) –  TomTom Jan 5 '11 at 7:03

You need to look at digital asset management software. DAM software is built to manage large collections of photos/videos/audio. They are usually backed with database to manage media catalogs. One example is MS iView

share|improve this answer

I'll bet the dlink box is running out of CPU or RAM.

We store millions of new image files a day.

Windows does not like large numbers of files in a folder. We keep the number under 10K per folder on our windows machines.

Linux using SAMBA is a better solution for managing the presentation layer than CIFS on windows. We layer apache on top of it for users to browse the raw files and also have a database for help them get what they need via a simple web drill down app. We wrote our own solution to manage the images as all the versions out there have issues.

Total files on a LUN is around 80 million or 2Tb.

We use rsynch to backup to another disk system then send that to tape once the volume becomes inactive.

You need to be real careful about disks per raid group and manage your IOs carefully all through the technology stack. it is very easy to run out of gas and with a lot of data coming at you, once you fall behind, you will be in a lot of pain.

CPU, RAM, Raid card, spindles, mobo, network - all need to be the fastest you can get.

A lot of spindles and a pci 2.0 LSI card on a LGA1366 Mobo with at least a x5550 is the way to go.

We really like the seagate desktop caviar line as it is fast and cheap.

I'd build a custom box with a lot of spindles.

share|improve this answer

You should be using a DAM system to track these images. I like the idea of keeping them on a NAS, provided you can back them up, but you would want a database driven application to keep tabs on what file is there and if possible provide a thumbnail searchable database. I've had good luck on the low end with Extensis Portfolio.

Check out: http://www.razuna.com/

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.