Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

We have a hard disk that is 600 Gigs and nearly full. It's been filled up with 18,501,765 files (mostly small 19k images) and 7,142,132 folders. It's very difficult to find out where exactly all the space has gone too. Our regular cleanup procedures are not clearing up enough space which means we need to look at this drive as a whole and determine what is out there and what can be moved or removed. We've tried several applications and so far they have either blown up or simply ran for an amazing amount of time to complete.

Server Information

  • Operating System: Windows Server 2003
  • File System: NTFS

Solution

Space ObServer was able to read through 18,501,765 files and 7,142,132 folders without taking up hardly any memory. I'm sure this is mostly due to the fact that it uses a SQL backend to store all of the data. It unfortunately the most expensive of all the products at $259.95 per server.

Attempted Solutions

During my research I tried several different solutions both pay and free. I kept a list of the products I tried below for everyone's information.

Free Software

Pay Software

Updates

Update #1: The server I am attempting to analyze has 2 GB of RAM and most products that I try seem to try and keep the file/folder information in memory. This tends to run out much too quickly with 18,501,765 files and 7,142,132 folders.

Update #2: Looks like the developers of WinDirStat got involved enough to tell us that it can compile under 64-bit. That gives it more memory to work with but I'm not sure if it's going to be enough unless they can persist to disk.

share|improve this question
    
What operating system? –  Wesley Jul 28 '10 at 21:05
    
It's Windows Server 2003. I edited the post to show that now. –  Nathan Palmer Jul 28 '10 at 22:36
    
are any of these completely free,shareware, or open source? –  Maslow Jul 29 '10 at 20:21
    
Nice of you to update us in that way about what is failed, running and pending. Gives me good ideas. –  Wesley Jul 29 '10 at 21:53
2  
No problem. I figured it will be good information since I feel like we're dealing with a # of files companies don't usually work with on a daily basis. Also, I just updated with categories for Free and Pay. –  Nathan Palmer Jul 29 '10 at 22:07

13 Answers 13

up vote 5 down vote accepted

Assuming your OS is Windows...

Either way you slice it, tabulating millions of files is always going to take a long time and will be restricted by the I/O of the disk itself. I recommend TreeSize Professional. Or maybe SpaceObServer. You could give the freeware version of TreeSize a try as well.

share|improve this answer
    
I agree. But those two products I haven't tried so I'll take a look. –  Nathan Palmer Jul 28 '10 at 22:36

du -s can be used in a pinch, and will run as long as needed.

share|improve this answer
    
Thanks. Will check that out next time I'm on a *ix system. –  Nathan Palmer Jul 29 '10 at 15:24
    
    
Didn't realize it was available on windows. I'll check it out. Thanks. –  Nathan Palmer Jul 29 '10 at 18:16

I regularly use FolderSizes on several 1TB drives with several million files with no problems.

share|improve this answer
    
Awesome. Didn't know about that one. –  Wesley Jul 28 '10 at 21:08
    
Great. I'll give it a try. –  Nathan Palmer Jul 28 '10 at 22:35

On Windows I use SpaceMonger (or older free version). On OSX I use Disk Inventory X.

share|improve this answer

Definitely try WinDirStat: it gives a fantastic visualization of disk use by depicting each file as a rectangle drawn to scale, color coded by file type. Click on any item in the visualization and you'll see it in the directory tree.

The standard 32-bit build is limited to 10 million files and 2 GB RAM usage, but the source code will build successfully as a 64-bit application. The fact that the server in question has only 2GB of RAM may be problematic in this specific case, but most servers with such large numbers of files will have much more RAM.

Edit #1: I regret to have discovered that, when tested on a 4TB volume containing millions of files, WinDirStat Portable crashed after indexing about 6.5 million files. It may not work for the original question if the drive contains 6+ million files.

Edit #2: Full version of WinDirStat crashes at 10 million files and 1.9GB used

Edit #3: I got in touch with the WinDirStat developers and: (1) they agree that this was caused by memory usage limitations of the x86 architecture, and (2) mentioned that it can be compiled as 64-bit without errors. More soon.

Edit #4: The test of a 64-bit build of WinDirStat was successful. In 44 minutes, it indexed 11.4 million files and consumed 2.7 GB of RAM.

share|improve this answer
    
It might be worth trying the regular version, as it's possible that the portable environment created an unexpected restriction. I'm not in a position to test that myself. windirstat.info –  John Gardeniers Jul 29 '10 at 0:22
    
Indeed, the regular version dies at 10+ million files and 1.9GB RAM usage. I suspect that it is unable to allocate >2GB. I'm surprised that it uses quite so much RAM (nearly 200 bytes per file tallied), but, then again, I grew up in an era when individual bytes were far more precious than they are today... –  Skyhawk Jul 29 '10 at 0:48
    
I use WinDirStat a lot. Unfortunately it just doesn't cut it when you get into a large # of files. –  Nathan Palmer Jul 29 '10 at 5:22
    
I'd be interested to hear if the dev's on WinDirStat come back with anything. RAM is going to be a constraint for me in 32-bit or 64-bit. –  Nathan Palmer Jul 31 '10 at 16:55
1  
There is no official build, but I can send you an unofficial one -- obviously, it would be tricky to roll your own if you don't have Visual Studio! (my.name@gmail.com reaches me) –  Skyhawk Aug 3 '10 at 5:48

+1 for the TreeSize products, but...

Your sentence about "not cleaning enough space" makes me wonder: Could you have run out of NTFS MFT reserved space? If the filesystem grabs more MFT space than is initially allocated, it is not returned to regular filespace, and is not shown in defrag operations.

http://support.microsoft.com/kb/174619

"Volumes with a small number of relatively large files exhaust the unreserved space first, while volumes with a large number of relatively small files exhaust the MFT zone space first. In either case, fragmentation of the MFT starts to take place when one region or the other becomes full. If the unreserved space becomes full, space for user files and directories starts to be allocated from the MFT zone competing with the MFT for allocation. If the MFT zone becomes full, space for new MFT entries is allocated from the remainder of the disk, again competing with other files. "

share|improve this answer
    
That looks like something good to check. Unfortunately we cannot see the MFT size because defrag will not analyze without a CHKDSK and CHKDSK is currently failing with "An unspecified error occurred." –  Nathan Palmer Jul 29 '10 at 19:06

I found a few issues with Spacemonger and in looking for a utility i could easily transfer or run from usb stick - Space Sniffer turned out to be very versatile in that regards, and handled multi-terabyte volumes with ease.

share|improve this answer
    
multi-terabyte volumes with how many files? It seems our major issue is not how much space is used but how many files the program can handle. Most are choking at 10 million. –  Nathan Palmer Jul 29 '10 at 15:24
    
I don't have a server with more than a few million files on to experiement with so I couldnt confidently answer your 10million file question - my only suggestion would be that with these tools you can set the directory depth at which is visualizes - find the happy medium and then go deep in the folder you need to, should save time doing the visualization as well. –  user49672 Jul 30 '10 at 0:33
  1. cd \
  2. dir /s > out.txt
  3. poof! Magic happens; or a perl hacker shows up
  4. Results!

Seriously. I've done this with 5 or 6 million files; not sure exactly what you're looking for but a good scripting language will eat this up.

share|improve this answer
    
Soo... what happens in step #3.. assuming a perl hacker does not show up? –  Nathan Palmer Aug 3 '10 at 16:17
    
Can you post more info on what you need? Largest files? Largest directories? Do you need date/time info? Is it a one-time need, or recurring? –  SqlACID Aug 3 '10 at 16:44
    
For now it's one time. I need to know which directories are the largest (dir + children) but I will need to go a few directories in before that information in valid. Then I will need to have a break-up of files by date so I can view recent vs old files. –  Nathan Palmer Aug 4 '10 at 1:47

Conerning the mft table, from the back of my head I seem to recollect that jkdefrag, the original opensource version, gave a very precise disk view including differet colouring for mft areas. I think i used that once before for a rule of thumb guesstimate of mft size and fragmentation.

Also doesn't care for chdsk.

Might try that?

share|improve this answer
    
I gave it a try. When I run the analyze it suffers the same fate most of these other programs have. Too many files/folders stored in memory. –  Nathan Palmer Aug 1 '10 at 0:21
    
Sorry to read that. Seems like a platform problem. I have another suggestion: Mirror the disk (bit image, Imaging soft or hardware mirror) no matter how, break the mirror and put the copy under forensics on another platform eg. linux/nix. This has consumed enought time to warrant for the cost of a mirror drive concerning the amount of Your working time invested. –  deploymonkey Aug 1 '10 at 18:44

http://www.freshney.org/xinorbis/

Another potential option

share|improve this answer

I'm not usually a Windows user, but I'm aware of Cygwin's existence. :-)

If it works good enough, something like

du -m /your/path | sort -nr | head -n 50

or perhaps in Cygwin

du C:\ | sort -nr | head -n 50

Anyway, those should print you out 50 biggest directories (sizes in megabytes).

share|improve this answer

I've used Disk Usage Analyzer - Baobab on Linux using it's remote scan function on Windows server. I don't know what are it's limits though.

Fedora LiveCD contains Baobab. Boot it on any computer in your LAN.

It's all free — as in beer and as in speech. Also for commercial use.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.