Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

i'm looking for some handy program/script to which i can pump data via stdin and which can present me some basic statistics of input data. for instance - provided with set of values separated by new line character i would like to get:

  • average for all values
  • average for data except 5% smallest and 5% largest values
  • standard deviation

yes - i know, can be done with bash or awk, but maybe you already know something handy?

ps.

i'm perfectly aware of 'big cannons' like octave, r and some other - but i need something much simpler.

thanks

share|improve this question

4 Answers 4

up vote 1 down vote accepted

This little AWK snippet will do part of what you're looking for:

awk '{sum += $0; count++; vals[$0]++} END {mean = sum / count; print "Total: ", sum; print "Mean: ", mean; for (i in vals){ s += vals[i] * ((i - mean) ** 2) }; print "Standard Dev: ", sqrt(s/count)}' datafile

The drop 5% part would be a little more complicated and depend on exactly how you mean it.

I know you're looking for something canned, but short of using R, Octave, SAS or SPSS, I don't know of anything.

Edit: Corrected formula

share|improve this answer
    
thanks.. point is - to pipe from the command line daily/monthly performance logs grepped by custom criteria and get result on the spot. –  pQd Jul 18 '10 at 15:33
    
@pQd: The AWK snippet will do that, as you know, and I imagine that R and Octave could be made to do it, too. I'm not so sure about the others (plus they're not cheap). –  Dennis Williamson Jul 18 '10 at 16:18
    
You're using a very odd definition of Std. Deviation. I don't think that it is the square root of the mean. –  Slartibartfast Jul 18 '10 at 18:12
    
@slartibartfast: Well I'm no statistician, but I believe that's correct for population std. dev. –  Dennis Williamson Jul 18 '10 at 19:43
    
@slartibartfast: Oops, you were right, I left out part of it. I'll post a correction. –  Dennis Williamson Jul 18 '10 at 19:56

you could try something along the lines of;

perl -e 'use List::Util qw(sum);while($r = <>){push (@array, $r)}; print sum(@array) / @array';

to get the average. And you could install the Statistics::Descriptive package http://search.cpan.org/~colink/Statistics-Descriptive-2.6/Descriptive.pm

to do what you need for the other requiremtns. stdev is probably easy, the other one would take a few more lines to sort and filter. (no doubt its possible to do in a single line...;-)

share|improve this answer
    
thanks - yeah - i can script it too without problem, but i was wondering if there is some standardized tool... –  pQd Jul 18 '10 at 13:32

R might be exactly what you are looking for, or it may be a total over kill for your purpose. Hard to tell from your question.

Anyways, check it out http://en.wikipedia.org/wiki/R_(programming_language)

share|improve this answer
    
thx, although i vote here for 'overkill' option :) –  pQd Jul 18 '10 at 16:49
    
"overkill is underrated" (from the new A-Team movie) –  Marcin Jul 27 '10 at 15:02

The first and last items are do-able (I've done them a couple of times) without maintaining the entire set of data in memory and without knowing the total number of items in advance. The middle item (dropping the outliers) is more challenging and requires maintaining the entire list in RAM or at least knowing the total number of items in advance.

I don't know of any simple pre-built tools to do any of those (though Octave and R sound as though they might be such).

share|improve this answer
    
i know that. thanks.. –  pQd Jul 18 '10 at 20:23

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.