I've tried different clients because I thought it was client related. All of them eventually disconnect, or stop transferring files - followed by getting a timeout disconnect. After a reconnect it works again for a bit of time. It is really strange, I can not figure out the reason.
I'm on a Mac, and the server is a VPS running Debian. If you need further information, please ask!
I appreciate any tips, because I'm kinda stuck!