SSH will only use RAM when that RAM is not needed for any more important purpose. If you're not actually connecting to it, it should be swapped out as soon as there is any better use for memory. You're trying to solve a problem that's already solved by the operating system's memory management system.
The operating system understands that free RAM is free, that is, that there is no cost associated with using it. It makes no attempt to minimize RAM usage, nor does it generally make any sense for it to do so. Don't worry, the OS is smart enough to use your RAM for the most important purpose at any particular time. If it's letting SSH use the RAM, that's because that is what is best.