I am running a SSH server and I am still using simple password authentication. Everywhere I read about security I am advised to use Public-Key Authentication. But I don't get the advantages. Using them is, in my eyes, either insecure or a lot of handy work.
Of course, if someone tries to brute-force the login into my SSH server Public-Key is a lot stronger than any password. But aside from that, it's totally insecure.
The advisors mostly argue that you don't have to remember a password. How insecure is that? So if someone hacks into my computer, he doesn't just get my computer, but my server too? If I am using SSH from various different clients, I have to store the public keys one every one of them, which multiplies the possibility that they fall into the false hands. I could save them on a usb-stick which I carry with me, but it can be lost and the finder has access to my server.
Possibly I am better served with Two-Factor Authentication.
Is there any argument I am missing? What is the best way for me?