TMG doesn't support FTPS (that's FTP + SSL, rather than SFTP, or SSH File Transfer, which generally works fine)
FTPS is Really Hard to do with an FTP filter, because the FTP NAT filter watches for information the client behind NAT sends to the FTP server, and encryption causes that not to be visible. And that annoys NAT inspectors.
If Total Commander supports using HTTP proxies, you can work around this fairly easily by configuring it to use an HTTPS connection instead (via ye olde CONNECT-to-get-a-plain-TCP-channel hack).
For that to work, you'll also need to configure TunnelPortRanges to allow the destination port as if it were SSL.
If (long list here)
- you're using PASV only (so multiple outbound connections)
- your initial connection is on port 21
- you aren't unhappy with the client using an "all protocols" rule
then you could set up
- a protocol definition for TCP outbound, destination port 21, not bound to the FTP filter ("NONFTP FTPS")
- a rule allowing NONFTP FTPS from the client IP to the server IP
- (I'm assuming it needs to be locked down to only one client and server; otherwise "anywhere" is fine if it's fine with you)
- a rule immediately following that, DENYING FTP (the real FTP) with the same source/dest
- a rule following that one that allows all outbound traffic between source and dest IP (or at least, all TCP connections on expected ports)
And this set of rules should be ordered before any other rules allowing the client full access to all protocols and ports.
It's important the DENY FTP rule is
a) immediately after the Special FTP rule that uses the protocol with the application filter unbound, and
b) before any other rules that might allow the client to use Good Old FTP to the target IP (otherwise TMG prefers the protocol definition with the application filter, which is what we don't want here - we want TMG to treat it like straight TCP)
I think this would cover all the FTPS scenarios. Same principle as this.