Measuring cumulative network statistics per user or per process

Question

I've been googling for hours -- Under Linux I want to know the cumulative bytes sent and received by user or by process over all ip protocols. The best I've found in my searches is that it's possible to use iptables to mark packets for a user, for example:

iptables -t mangle -A OUTPUT -p tcp -m owner --uid-owner test -j MARK --set-mark 1

It appears that "tc" can then shape traffic with that but I just want the statistic -- I don't want to shape the traffic. I want something like: "user U has transmitted used XMB since time Y". I can't figure out how to get statistics from these marked packets. Also, I've looked at nethogs but they seem to be measuring the instantaneous flow and I need cumulative counts. Anyone have ideas?

Answer 1

The iptables -m owner trick can only track packets against a user that are sent out bound (by definition). It can't be used to track packets received for that user.

I admit, off the top of my head, I don't see any nice way of doing this. At a stab in the dark, it's going to involve applying a patch at the kernel level to, for example, only allow specific users to bind to particular port ranges on the network stack (similar to the idea that only root can bind to network sockets on port 1024 and lower). Then you could apply iptables traffic logging on these port ranges and know for sure that any traffic is for, and only for, the corresponding user who is allowed to bind to those ports. Downside is, this is going to cause havoc for user applications who are not aware of these limitations, when they then decide to try and bind to a port and the kernel says no.

It may also be possible to do this with SE Linux, but I suspect has the potential to become a sys-admin maintenance nightmare: http://www.linuxquestions.org/questions/linux-server-73/how-can-i-restrict-ports-for-users-to-bind-to-667153/

Answer 2

If you use -j CONNMARK in the OUTPUT chain, and then match with -m connmark in the INPUT chain you will be able to retrieve that information.

Here is an example to track incoming/outgoing traffic for user nobody (id 65534):

# iptables -I OUTPUT -m owner --uid-owner 65534 \
                     -m comment --comment 'out - user nobody - uid 65534' \
                     -j CONNMARK --set-mark 65534
# iptables -I INPUT -m connmark --mark 65534 \
                    -m comment --comment 'in - user nobody - uid 65534'

As a result, you can fetch the counter values (first on the line) by running iptables-save -c:

# iptables-save -c | grep 'uid 65534'
[2585:3797434] -A INPUT -m connmark --mark 0xfffe -m comment --comment "in - user nobody - uid 65534"
[1166:63139] -A OUTPUT -m owner --uid-owner 65534 -m comment --comment "out - user nobody - uid 65534" -j CONNMARK --set-xmark 0xfffe/0xffffffff

Answer 3

For the accounting: you can just leave out -j completely and not provide a target. As the manpage says:

-j, --jump target ... If this option is omitted in a rule (and -g  is
    not used), then matching the rule will have no effect on the
    packet's fate, but the counters on the rule will be incremented.

Answer 4

Perhaps the easiest thing to do, depending on how long you want to keep those data, is to just look at the matched packets/bytes counts for your mark rules with iptables -t mangle -L OUTPUT -v. You'll see two columns at the left of the output indicating these metrics.

If you want a daily count, you can always schedule an iptables -t mangle -Z OUTPUT to clear the counts at midnight.

Answer 5

If you can mark it then it means you can log it.Check the LOG target extension in the iptables man pages. Here a simple introduction: http://www.cyberciti.biz/tips/force-iptables-to-log-messages-to-a-different-log-file.html

I guess you can add a log rotate entry for this and use it to trigger a script that summarizes the log entries and removes the file. maybe hourly?