Take the 2-minute tour ×
Server Fault is a question and answer site for system and network administrators. It's 100% free, no registration required.

We have a set of directories containing lucene indexes. Each index is a mix of different file types (differentiated by extension) eg:

0/index/_2z6.frq
0/index/_2z6.fnm
..
1/index/_1sq.frq
1/index/_1sq.fnm
..

(it's about 10 different extensions)

We'd like to get a total by file extension, eg:

.frq     21234
.fnm     34757
..

I've tried various combinations of du/awk/xargs but finding it tricky to do exactly this.

share|improve this question
    
You have the answer for that problem in this post : serverfault.com/questions/183431/… –  Blueicefield Mar 7 '12 at 13:16
    
Do you want to know the total size of each type of file or the total number of each type of file ? –  Iain Mar 7 '12 at 13:23
    
Total file size please. –  barnybug Mar 7 '12 at 14:11

6 Answers 6

up vote 7 down vote accepted

For any given extension you an use

find /path -name '*.frq' -exec ls -l {} \; | awk '{ Total += $5} END { print Total }'

to get the total file size for that type.

And after some thinking

#!/bin/bash

ftypes=$(find . -type f | grep -E ".*\.[a-zA-Z0-9]*$" | sed -e 's/.*\(\.[a-zA-Z0-9]*\)$/\1/' | sort | uniq)

for ft in $ftypes
do
    echo -n "$ft "
    find . -name "*${ft}" -exec ls -l {} \; | awk '{total += $5} END {print total}'
done

Which will output the size in bytes of each file type found.

share|improve this answer
    
Thanks, was looking for something that summarized by any extension (as it'd be handy to then sort for example) –  barnybug Mar 7 '12 at 14:10
    
Check my update. –  Iain Mar 7 '12 at 14:11
    
great thanks. awk produces scientific output for some of the numbers, can this be disabled: .fdt 3.15152e+10 –  barnybug Mar 7 '12 at 14:15
1  
slightly tweaked to just give plain integer numbers: find . -name "*${ft}" -print0 | xargs -0 du -c | grep total | awk '{print $1}' –  barnybug Mar 7 '12 at 14:17
    
Might want to use -iname to make the file extenstion search case insensitive. –  Aaron Copley Mar 7 '12 at 15:16

With bash version4, you just need to call find, ls and awk not necessary:

declare -A ary

while IFS=$'\t' read name size; do 
  ext=${name##*.}
  ((ary[$ext] += size))
done < <(find . -type f  -printf "%f\t%s\n")

for key in "${!ary[@]}"; do 
  printf "%s\t%s\n" "$key" "${ary[$key]}"
done
share|improve this answer
    
thanks for find trick @glenn –  Selman Ulug Mar 7 '12 at 16:19

Every second column splited by . and last part (extension) saved in array.

#!/bin/bash

find . -type f -printf "%s\t%f\n" | awk '
{
 split($2, ext, ".")
 e = ext[length(ext)]
 size[e] += $1
}

END{
 for(i in size)
   print size[i], i
}' | sort -n

then you got every extensions total size in bytes.

60055 gemspec
321991 txt
2075312 html
2745143 rb
13387264 gem
47196526 jar
share|improve this answer

Extending on Iain's script with a faster version for working with a large number of files.

#!/bin/bash

ftypes=$(find . -type f | grep -E ".*\.[a-zA-Z0-9]*$" | sed -e 's/.*\(\.[a-zA-Z0-9]*\)$/\1/' | sort | uniq)

for ft in $ftypes
do
    echo -ne "$ft\t"
    find . -name "*${ft}" -exec du -bcsh '{}' + | tail -1 | sed 's/\stotal//'
done
share|improve this answer

This is the solution :

find . -type f | grep -E ".*\.[a-zA-Z0-9]*$" | sed -e 's/.*\(\.[a-zA-Z0-9]*\)$/\1/' | sort | uniq -c | sort -n

Solution posted originally on this post : Get all extensions and their respective file count in a directory

share|improve this answer
1  
This is a count by number of files, not what I was asking - I'd like a total by size. –  barnybug Mar 7 '12 at 14:10

I solved using this two commands:

FILES=$(find . -name '*.c')
stat -c %s ${FILES[@]} | awk '{ sum += $1 } END { print ".c" " " sum }'
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.