4

We currently store our photos in a structure like this:

folder\1\10000 - 19999.JPG|ORF|TIF (10 000 files)

folder\2\20000 - 29999.JPG|ORF|TIF (10 000 files) etc...

They are stored on 4 different 2TB D-link NASes attached and shared on our office network (\\nas1, \\nas2, and so on...)

Problems:

1) When a client (Windows only, Vista and 7) wishes to browse the let's say \\nas1\folder\1\ folder, performance is quite poor. A problem. List takes a long time to generate in explorer window. Even with icons turned off.

2) Initial access to the NAS itself is sometimes slow. Problem.

SAN disks too expensive for us. Even with iSCSI interface/switch technology.

I've read a lot of tech pages saying that storing 100 000+ files in one single folder shouldn't be a problem. But we don't dare go there now that we experience problems on a 10K level.

All input greatly appreciated,

/T

3
  • This doesn't really answer your question, but just wanted to point something out. I spec'd out a "poor mans" SAN for roughly $3000 with 3TB of drive space and iSCSI. Granted I had to build it myself but just wanted to point that out.
    – GregD
    Dec 10, 2010 at 14:20
  • What speed is your networking gear running at?
    – GregD
    Dec 10, 2010 at 14:22
  • I don't think that a SAN is going to help other than consolidation of storage which will allow more spindles to be in a single array which should improve the performance of the storage. Beyond that you're serving files here to multiple Window clients from the sound of it. So you would still need a single server attached to the SAN serving the files via CIFs. My suggestion is if possible avoid Explorer. Use a database to index your collection and either a web or GUI to let the users find and retrieve the photo they are looking for. Dec 10, 2010 at 15:05

4 Answers 4

6

1: yes, Explorer does not like 10.000 items in a directory. 1000 is b etter. This is NOT a file system issue - you can dsafely store 100.000 files in a folder and work with them efficiently from the command line. It is Explorer being slow.

2: Velociraptor 600gb disc. 10.000 RPM. Fast. Need more then get the money for SAS. Or wait till things get cheaper. If i need a truck, I need to pay for it.

2
  • FS will suffer too if too many files - with varying impact. Old novell didn't like >10000 files in one dir, collapsed performance.
    – user65297
    Jan 4, 2011 at 20:45
  • NTFS will gladly handle 100.000 or more files without problems in a folder. Old novell had all kinds of stupdid decisions going back to times discs were small ;)
    – TomTom
    Jan 5, 2011 at 7:03
1

You need to look at digital asset management software. DAM software is built to manage large collections of photos/videos/audio. They are usually backed with database to manage media catalogs. One example is MS iView

0

I'll bet the dlink box is running out of CPU or RAM.

We store millions of new image files a day.

Windows does not like large numbers of files in a folder. We keep the number under 10K per folder on our windows machines.

Linux using SAMBA is a better solution for managing the presentation layer than CIFS on windows. We layer apache on top of it for users to browse the raw files and also have a database for help them get what they need via a simple web drill down app. We wrote our own solution to manage the images as all the versions out there have issues.

Total files on a LUN is around 80 million or 2Tb.

We use rsynch to backup to another disk system then send that to tape once the volume becomes inactive.

You need to be real careful about disks per raid group and manage your IOs carefully all through the technology stack. it is very easy to run out of gas and with a lot of data coming at you, once you fall behind, you will be in a lot of pain.

CPU, RAM, Raid card, spindles, mobo, network - all need to be the fastest you can get.

A lot of spindles and a pci 2.0 LSI card on a LGA1366 Mobo with at least a x5550 is the way to go.

We really like the seagate desktop caviar line as it is fast and cheap.

I'd build a custom box with a lot of spindles.

0

You should be using a DAM system to track these images. I like the idea of keeping them on a NAS, provided you can back them up, but you would want a database driven application to keep tabs on what file is there and if possible provide a thumbnail searchable database. I've had good luck on the low end with Extensis Portfolio.

Check out: http://www.razuna.com/

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .