say I run a command that outputs space separated values over lines. Some of which are numbers. Is there a utility I can use to calculate the mean, median, standard deviation of these numbers? something like 'cut' but that outputs these statistics.
Simple statistics from the command line interface (CLI), fast. Description
This is a lightweight, fast tool for calculating basic descriptive statistics from the command line. Inspired by https://github.com/nferraz/st, this project differs in that it is written in C++, allowing for faster computation of statistics given larger non-trivial data sets.
Additions include the choice of biased vs unbiased estimators and the option to use the compensated variant algorithm.
Given a file of 1,000,000 ascending numbers, a simple test on a 2.5GHz dual-core MacBook using Bash time showed sta takes less than a second to complete, compared to 14 seconds using st.
You might also consider using clistats. It is a highly configurable command line interface tool to compute statistics for a stream of delimited input numbers.
- Input data can be from a file, standard input, or a pipe
- Output can be written to a file, standard output, or a pipe
- Output uses headers that start with "#" to enable piping to gnuplot
- Signal, end-of-file, or blank line based detection to stop processing
- Comment and delimiter character can be set
- Columns can be filtered out from processing
- Rows can be filtered out from processing based on numeric constraint
- Rows can be filtered out from processing based on string constraint
- Initial header rows can be skipped
- Fixed number of rows can be processed
- Duplicate delimiters can be ignored
- Rows can be reshaped into columns
- Strictly enforce that only rows of the same size are processed
- A row containing column titles can be used to title output statistics
- Summary statistics (Count, Minimum, Mean, Maximum, Standard deviation)
- Least squares offset
- Least squares slope
- Raw data after filtering
NOTE: I'm the author.
The R language would be perfect for this. It runs in the terminal, its free and probably the most popular statistics languages out there. So, it has tons of documentation on how to do neat things, including the ones you are asking for.