Couple of obvious problems I see with your htb config.
First, you don't have a 1:30 handle, and you should, given the default 30. Perhaps you meant default 1? This would attach all traffic to classid 1:1 unless the rule attached traffic to a different class.
Second, you need to set your rate in the htb 1:1 highest level class to something reasonable and less than your line rate. For example, if your line rate is 100Mbit, rate should be maybe 90% that: 90Mbit. Some people recommend more conservative settings like 75% line rate; I play a dangerous game at 90% and it works OK for me -- reduce if you notice dropped packets or improper bandwidth allocation. The latter is indicative of upstream buffering.
Third, your rate-limited class should be configured with a rate set to the guaranteed bandwidth you're willing to allocate the class and the ceil set to the maximum level you'll allow the user. For example, let's say you want to give testuser 400kbit of guaranteed bandwidth, but if the line is otherwise idle, let it grow to line rate. Set rate 400kbit and ceil to whatever you put as rate for 1:1. If you don't set ceil, it defaults to rate.
Fourth, to accomplish your goal of exempting ftp traffic, you'll need to use connmark instead of just mark. Otherwise your related data connection won't get properly exempted from the --set-mark 10. Connmark will properly pick up related connections.
I suggest the following (untested!) rules off the top of my head:
# flush rules out of postrouting so you're not constantly inserting during testing.
iptables -t mangle -F POSTROUTING
iptables -t mangle -X HTB_OUT
# The use of RETURN here is to fall out of our user chain and hit
# -j CONNMARK --save-mark in the POSTROUTING chain.
iptables -t mangle -N HTB_OUT
iptables -t mangle -A HTB_OUT -j MARK --set-mark 30
iptables -t mangle -A HTB_OUT -p tcp --dport 21 -j MARK --set-mark 30
iptables -t mangle -A HTB_OUT -m mark ! --mark 0 -j RETURN
iptables -t mangle -A HTB_OUT -m owner --uid-owner testuser -j MARK --set-mark 10
iptables -t mangle -A HTB_OUT -m mark ! --mark 0 -j RETURN
iptables -t mangle -A POSTROUTING -j CONNMARK --restore-mark
iptables -t mangle -A POSTROUTING -m mark ! --mark 0 -j ACCEPT
iptables -t mangle -A POSTROUTING -j HTB_OUT
iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark
Then for tc, something like the following:
# set a script variable that will represent our line-rate minus some change
CAPRATE=90Mbit
CAP_SUB_400=89Mbit
# clear our qdisc settings for eth0 so we're starting from a clean slate.
tc qdisc del dev eth0 root
tc qdisc add dev eth0 root handle 1: htb default 20
tc class add dev eth0 parent 1: classid 1:1 htb rate ${CAPRATE} burst 5k
# this is our capped class:
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 400kbit ceil ${CAPRATE}
# this is our default, catch-all class:
tc class add dev eth0 parent 1:1 classid 1:20 htb rate ${CAP_SUB_400} ceil ${CAPRATE}
tc qdisc add dev eth0 parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev eth0 parent 1:20 handle 20: sfq perturb 10
tc filter add dev eth0 parent 1:0 prio 0 protocol ip handle 10 fw flowid 1:10
Things to remember: The sum of all rates for the immediate children of any parent should not exceed the parent rate ever. I cheated a little here and rounded 1:20's rate down to 89Mbit instead of 89600kbit. You can under-commit, but you should never over-commit.
iptables rules are evaluated in order. If your policy allows it, the most common matches should appear first; granted most of this is obviated by the POSTROUTING rules before entering the HTB_OUT chain, but it's a good rule of thumb.
So what's SFQ for... it's like stirring the pot. SFQ tries to give every connection (endpoint pair, actually) a fair share of the bandwidth, then every perturb seconds it mixes things up again, in case too many connections ended up in the same internal bucket (which is possible because of how hashing is done on src/dst/portpair). For more info, check lartc or the tc-sfq manpage.
