I added this to the bottom of the sshd_config file
Match User user
PasswordAuthentication no
Then I restarted SSH, and when I went to go SSH into my server I got the error
Permission denied (publickey)
How do I make it so a specific user can login to the server without entering a password or needing a key?