So far I have disabled PermitRootLogin on SSH and switched the port.
I'm not a big fan of running services on non-standard ports; you make interopability harder for yourself and others, for no real gain. With a decent password (or better: key), random SSH scans have no chance of succeeding anyway, and targeted attacks are not phased by a different port.
But working as root isn't the best thing to do I've heard... So I thought about just granting my unprivileged user sudo rights.
This is a personal preference; there are valid reasons for choosing either approach.
Personally, I prefer to avoid sudo, mainly because it's a lot more complex than su. And this complexity carries a higher security risk, both in sudo's configuration, and sudo itself (there have been several security advisories). sudo can be very useful in allowing an unprivileged user to execute very specific commands without password, though.
That said, switching to root and using sudo in the way you describe is pretty much interchangable, except for a few things:
Sudo can log each command executed through it. When using su, you have to rely on root's shell history for logging. (An attacker can trivially circumvent this though, e.g. by simply doing sudo su)
With sudo, you typically need your user's password, with su you need root's.
With sudo it's easier to interleave privileged and non-privileged commands.
However, I usually find myself either doing administrative tasks for which I need a lot of root, or something non-administrative for which I don't need root. And to avoid confusion, I color my shell prompts to distinguish between root (red) and user (blue).
By default, sudo is limited to certain accounts.
For su, a similar thing can be achieved by using pam_wheel.so to limit su-to-root to members of a certain group. (addgroup --system wheel, adduser YOU wheel, uncomment the pam_wheel line in /etc/pam.d/su)
Should I disable SSH Authentication via password and just use keys? Then it would be difficult for me to access the server if I'm not on my own laptop.
Yes, that's probably one of the biggest security gains.
You can grant access for the SSH keys of your accounts on multiple (trusted) machines.
If the machines aren't trusted, you should consider not logging in from them anyway; they could inject commands or have a keylogger sniffing your passwords.
Some additional tips:
Consider limiting SSH access to user accounts that need SSH access.
I like to do this by addgroup --system allowssh, and then in /etc/ssh/sshd_config setting PubkeyAuthentication no, and adding a Match Group allowssh\n PubkeyAuthentication yes stanza.
I'm sure this could be done through PAM as well, using something like auth required pam_succeed_if.so quiet_success user ingroup allowssh in /etc/pam.d/sshd.
Consider installing a logchecker that reports anomalies. To cut down on the volume of mails, you could consider having successful login attempts mailed, rather than unsuccessful ones.
