1. Can I use a packet sniffer here to determine if the user is downloading something?
Yes, basically you only need Putty.
Logon to modem via telnet. Busybox will welcome you.
cat /proc/sys/net/ip_conntrack or
tail -f /proc/sys/net/ip_conntrack | grep ASSURED | grep 'dport=80'.
but first, you need to know how to read & analyze the ip_conntrack file.
try there, i bet you can understand easly.
here is a simple brief to understand ip_conntrack http://www.faqs.org/docs/iptables/theconntrackentries.html
2. Can I determine if the user is downloading a file based on the modem alone.(The lights blink faster)
Relative to first answer. you can look for basic eth statistics on modem's busybox. it depends vary of modem's software. but there is always eth statistics.
Another way is SNMP. activate snmp on modem's admin interface. Then download the program named STG. snmp traffic grapher. Then try to connect to modem via STG. it will show you realtime statistics of down / up bandwidth.
3. Is there an application that I could use for the modem or the hub to limit or avoid direct downloads.
Yes, most interesting & most efective application is already installed in your modem which named as IPTABLES.
just logon to your modem via telnet. to see what is iptables doing
then prepare your rule via documentation rest is only to apply it.
Also iptables will offer you more interesting limitations & blocking technics.