Okay so I ssh a lot, mainly from my Macs to my Linux boxes but sometimes also from a Linux box to a mac or from a Mac to a Mac. I would want to implement a secure passwordless way to ssh between these computers so that I don't have to enter my ssh passwords 50 times a day. I know that it can be done by creating a set of SSH-keys without a passphrase but I understand that it's not a safe or secure way to do it.
There are a number of ways of using ssh without actively putting in your password each time you connect. See the range of answers on this serverfault answer.
If you ssh to a remote server and it accepts your connection but you are prompted for a password, you may be the victim of a man-in-the-middle attack (MIMA). If you have pre-entered your password against a separate ssh key using an ssh key, you won't provide the MIMA attacker with your passphrase. The ssh-agent facility provides one with a convenient way of holding your private ssh key and associated password available for any connections you make for which the key and password may be valid.
Using an ssh key is generally considered to provide better security than providing one's password interactively.
One way in which the ssh key and password may be less secure that a keyboard interactive ssh password entry is if you leave your machine accessible to others. In that case someone may be able to use your pre-authenticated details to access a remote machine.
If you intend using ssh-agent, you may wish to ensure that you have locked your machine when you are away from it, or use
ssh-agent -k to kill your ssh-agent process from time to time.