I've got a network of static file serving servers. I use nginx to serve the files, and munin to monitor the network traffic. I want to know the output bandwidth of the server that goes to each IP address downloading a file to evaluate the bandwidth each Internet Service Provider in my country is downloading from my servers. The average output bandwidth of servers is about 700MB/s (9 servers, most of them have 4 1Gbits/s ports bonded). How can I do this?
The first approach that comes to mind is to have your web servers simply log the requests they handle and use a log parser to generate usage statistics. The combined log format is a standard commonly supported by log parsing web analytics software.
If your web servers run as cluster rather than have each server maintain their own log files which you would need to merge to get aggregated results, you can use the syslog protocol to have all nodes transmit their log entries to a central logging host and generate usage statistics there.
The relevant settings are documented in the nginx manual.