PAM does not just do authentication, but authorisation and session services. You probably want to keep it on as it adds quite a bit of flexibility.
PAM will be called for a successful pubkey authentication, because session and account services are still checked.
PAM can do things SSH cannot. This list is not exhaustive:
- Deny a user access if SELinux is not in enforcing mode (if thats your thing).
- Set resource limits like max processes and max logins allowed.
- Flexibly deny a user based off of their user and remote source IP (possible in SSH too, but is pretty terse in PAM)
- Setup a series of environment variables you may want to pass.
- Create a home directory for a user if it did not exist.
- Deny users based off of the time/date of their access attempt.
- Deny inactive users.
- Deny users using an invalid shell.
- Setup key logging facilities of input.
auth
entries of your PAM configuration.account
andsession
are still used by sshd in this case. Turning this off would cause inconsistent behavior between your TTY logins and your ssh logins.