Dropped packets are de rigeur for ISA Server, and 6000 is not a lot for all but the tiniest network (my TMG box chews through that many UPNP packets in an hour, I reckon!)
It's designed to drop packets if it can't work out what to do with them (i.e. if it doesn't match a rule, or if a client goes overboard).
First, check the Reason code if you think the drops are significant. If your dropped packets are dropped with a flood prevention - type reason code, you may need to implement custom IP limits (under Flood Prevention) for the Terminal Server - it'll source a multiple of the number of connections on one box as it has users, so the one-box limit (think it's ~160 TCPs or HTTPs or similar) might need to yield to an insanely-high Custom limit for that IP.
If there's no such warning, or if it's a warning related to an unexpected protocol, troubleshoot that.
If not, you can use the "processing time" column in the log viewer for better visibility on what requests are taking a long time - keep in mind that some requests are by-design amazingly long-lived (facebook chat; OWA; Outlook Anywhere RPC/HTTP) so they'll skew the average.
You can check Event Logs for indications of authentication slowness, as long as your ISA box is up to date (> SP1) - it logs events when DCs or authentication sources are running slowly.
And you can use Perfmon logging (create a log with a 30 second interval, logged to BLG, not one of the slow text formats) to evaluate the rough throughput of the box at any given moment - ensure you capture all ISA-related objects (and all counters - just log the object), then look for slowness, and see if you can correlate with a cause. Of particular interest might be: request speed, requests/sec, backlogged packets, pending DNS resolutions.