IPTables isnt made for this kind of work, where lots and lots of packets need to be analyzed to make these decisions. IPTables is partly the answer though!
The real answer to this is the awesome and underused traffic control facilities in linux. Note that mucking around with this without knowing what is going on may lead to you losing network connectivity to the machine! You have been warned!
Assuming eth0 is the outgoing device you will need to create a class-based traffic control queue which will by default output most traffic throught the 'fast' queue and put a specific list of people into the 'slow' queue.
The beauty of this is you can create a situation whereby you allow lots of outbound traffic for the slow user unless an overriding class wants the bandwidth, but this example does not do this (will always provide 10kbps to the slow users). The queing system will look something like this:
| Class 1:1 |
| Root (all flows)|
| 100mbit |
+----------+ | | +----------+
| 1:11 +-----+ +-----+ 1:12 |
| Default | | Slow |
|100mb-80kb| | 80kb |
To do this, first you'll need to setup the queuing discipline in the kernel. The following will do this for you.. you must run this as one whole script
tc qdisc add dev eth0 parent root handle 1: hfsc default 11
tc class add dev eth0 parent 1: classid 1:1 hfsc sc rate 100mbit ul rate 100mbit
tc class add dev eth0 parent 1:1 classid 1:11 hfsc sc rate 99920kbit ul rate 100000kbit
tc class add dev eth0 parent 1:1 classid 1:12 hfsc sc rate 80kbit ul rate 80kbit
tc qdisc add dev eth0 parent 1:11 handle 11:1 pfifo
tc qdisc add dev eth0 parent 1:12 handle 12:1 pfifo
The "default 11" is important as it tells the kernel what to do with traffic not classified.
Once this is done, you can then setup an iptables rule to classify packets that match a certain criteria. If you plan on putting lots and lots of people into this slow rule an ipset rule is more appropriate (which should be available on rhel6 I believe).
So, create an ipset database to do the matching against...
ipset create slowips hash:ip,port
Then create the iptables rule to do the match..
iptables -t mangle -I OUTPUT -m set --match-set slowips dst,src -j CLASSIFY --set-class 1:12
This instructs the kernel that if you match the destination IP with the source port from the set, classify it into the slow queue you setup with traffic control.
Now, finally whenever you want to slow an IP down you can use the ipset command to add the ip to the set such as this:
ipset add slowips 192.168.1.1,80
ipset add slowips 192.168.1.1,21
You can test it works using the command "tc -s class show dev eth0" and you will see stats in there indicating packets being redirected to the slow queue.
Note the only real downside to this is making it survive reboots. I dont think there are any init scripts available to create the ipsets from dumps on reboot (and they also must be created before iptables rules) and im certain theres no init scripts to resetup traffic control rules on reboot. If your not bothered, you can just recreate the whole thing from invoking a script in rc.local.