There is little difference between allowing connections to ssh from any IP and allowing connections to ssh from any IP for a given user as the user authentication has to happen after the connection - ie, after it's already let the originator connect.
You certainly should be restricting direct ssh access for servers that don't need it in general regardless of source IP.
If you don't wish to have the ssh port open to all and sundry, you can try some of these options:
- You can set up a VPN server and then use a VPN client.
- You can set up "port knocking" where a specific series of connection attempts in order to the correct ports will cause the ssh port to start listening to you long enough to connect.
- You can add the netblocks of the ISPs you commonly connect from to the list of allowed addresses to vastly reduce the number of potential attackers but still retain the conveniences of a "connect almost anywhere" solution.
- You can have your ssh listen on a non-standard port, gaining nothing in defence against a targeted attack but a great way to deter the usual script kiddies running automated probes.
Other options can include:
- Using something like fail2ban blocking attackers after a certain number of incorrect password attempts.
- Switching to ssh keys and setting sshd to not accept password as a valid method of authentication.
- Using a trusted "go between" server that you know is well maintained and secure, then having your server only accept ssh connections from there.
It's always going to be a trade-off between convenience and security, try to find the point that works for you.