Traffic shaping with iptables, ipset and tc (--match-set and --set-mark)

Question

I am having some trouble with what I believe should be a fairly straightforward traffic shaping problem.

I have an Ubuntu (16.04) server that is acting as a router/nat. I want to allow most users to use 2mbps internet, whilst throttling some devices to 512kbps.

I have the list of throttled devices in an ipset. I am trying to --mark-set the packets that come from these devices so that tc can target them with traffic shaping.

My current setup is:

tc qdisc del root dev $LAN
tc qdisc add dev $LAN root handle 1: htb default 20
tc class add dev $LAN parent 1: classid 1:1 htb rate 6mbit burst 15k
tc class add dev $LAN parent 1:1 classid 1:10 htb rate 5mbit burst 15k
tc class add dev $LAN parent 1:1 classid 1:20 htb rate 2mbit ceil 2mbit burst 15k
tc class add dev $LAN parent 1:1 classid 1:30 htb rate 512kbit ceil 512kbit burst 15k
tc qdisc add dev $LAN parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $LAN parent 1:20 handle 20: sfq perturb 10
tc qdisc add dev $LAN parent 1:30 handle 30: sfq perturb 10
tc filter add dev $LAN protocol ip parent 1: prio 1 handle 6 fw flowid 1:30


$IPS create throttled hash:mac -exist
$IPS add throttled 00:11:22:33:44:55 -exist
$IPT -A PREROUTING -t mangle -m set --match-set throttled src,dst -j MARK --set-mark 6

This doesn't throttle me down to 512kbps (assuming my mac address is the one provided), it falls back to the default 2mbps

If I remove the --match-set part, it correctly throttles all devices down to 512kbps (which makes me think the tc section is correct)

However, if I DROP the packets that match this set, it correctly drops the packets (which makes me think that the iptables section is correct)

Am I missing something here, or can these two commands not be used together like this?

Any help greatly appreciated.

Thanks,

Answer 1

Hopefully after a year you have figured out your issue, however for the benefit of those that follow, the following is an explanation of at least 1 of your issues.

Based on the ipset example you provided, your iptables rules are invalid. The set type hash:mac only stores a single field. However your iptables statement is testing the set as if it contains data pairs (i.e. two fields such as mac,IP or mac,net). As such you are matching entries within the set using the source MAC address of the packet and an undefined destination field from the packet:

ipset create throttled hash:mac -exist
ipset add throttled 00:11:22:33:44:55 -exist
iptables -A PREROUTING -t mangle -m set --match-set throttled src,dst -j MARK --set-mark 6

However due to the set type used, you can only match against a single field. The following iptables statement shows the correct method to match a source MAC address contained within the set:

ipset create throttled hash:mac -exist
ipset add throttled 00:11:22:33:44:55 -exist
iptables -A PREROUTING -t mangle -m set --match-set throttled src -j MARK --set-mark 6

To illustrate, consider if you had a set with the type of hash:net,port,net on a firewall. The set would store triples (two networks and one port).

Let's assume the firewall has the following ipset entries and iptables rules:

# creates set
ipset create throttled hash:net,port,net
ipset add services 192.168.0.0/16,tcp:80,192.168.1.80/32
ipset add services 192.168.0.0/16,tcp:443,192.168.1.80/32

# clear all rules (to illustrate that conntrack is not being used)
iptables -F FORWARD

# allows traffic to web server
iptables -A FORWARD -m set --match-set services src,dst,dst -j ACCEPT

# allows traffic from web server
iptables -A FORWARD -m set --match-set services dst,src,src -j ACCEPT

# drop all other traffic
iptables -P DROP

The above rules are functionally equivalent to:

# clear all rules (to illustrate that conntrack is not being used)
iptables -F FORWARD

# allows traffic to web server
iptables -A FORWARD -s 192.168.0.0/16 -p tcp --dport 80  -d 192.168.1.80/32 -j ACCEPT
iptables -A FORWARD -s 192.168.0.0/16 -p tcp --dport 443 -d 192.168.1.80/32 -j ACCEPT

# allows traffic from web server
iptables -A FORWARD -d 192.168.0.0/16 -p tcp --sport 80  -s 192.168.1.80/32 -j ACCEPT
iptables -A FORWARD -d 192.168.0.0/16 -p tcp --sport 443 -s 192.168.1.80/32 -j ACCEPT

# drop all other traffic
iptables -P DROP

Though the two above methods are functionally equivalent, the ipset method will perform better with a large number of entries due to utilizing hash table lookups vs linear evaluations of the iptables rules.

When using sets, the set type dictates how many matching fields are used. The iptables statement specifies whether source or destination fields of the packets are used to match each field of the set.

The following are a few mappings to illustrate:

 ipset type        | iptables match-set | Packet fields
 ------------------+--------------------+---------------------------------
 hash:net,port,net | src,dst,dst        | src IP address, dst port, dst IP address
 hash:net,port,net | dst,src,src        | dst IP address, src port, src IP address
 hash:ip,port,ip   | src,dst,dst        | src IP address, dst port, dst IP address
 hash:ip,port,ip   | dst,src,src        | dst IP address, src port, src ip address
 hash:mac          | src                | src mac address
 hash:mac          | dst                | dst mac address
 hash:ip,mac       | src,src            | src IP address, src mac address
 hash:ip,mac       | dst,dst            | dst IP address, dst mac address
 hash:ip,mac       | dst,src            | dst IP address, src mac address

You may have additional issues within your TC/iptables interaction, however this should at least resolve your ipset/iptables issue.

Answer 2

One thought ... your set contains MAC addresses, but I suspect that the PREROUTING rule is trying to match src or dst IP addresses rather than MAC addresses.