The manpage for usermod says this about locking a users account:
-L, --lock
Lock a user's password. This puts a ´!´ in front of the encrypted password, effectively disabling the password. You can´t use this option with -p or -U.
I have tried to use this to have a user only log in with the public key, but when I lock the account I get a password prompt. When I unlock the account again the public key works.
Edited: to make it clear: I want to be able to log in using the key still, and somehow locking the user gives me a password prompt when trying to use the key