Long time reader, first time asker:
I'm a junior-level Linux admin who's increasingly supporting very large data sets/stores on local & NFS storage. The data is generally in the form of massive folder & file trees with arbitrary names. To give an example of the challenge, I'm working with 20+ TB partitions containing hundreds of thousands of nested directories and tens of millions of files.
As these partitions/exports are shared resources I generally have to play detective when they begin to run out of space or inodes, to determine what folders are the biggest offenders.
I'm discovering that 'du -s' is much too slow and manual to do the job and was hoping to see what tricks other folks are using to determine the largest or most numerous file count dirs are amongst very large partitions.
The only OS I'm concerned with doing these searches on are RHEL 5/6, so any mix of standard CLI Linux tools is fine.
Thanks!