Use IdentityFile
in ~/.ssh/config
If you only want it for a specific hosts, include it under the Host
directive.
IdentityFile
Specifies a file from which the user's DSA, ECDSA or DSA authentication identity is read. The default is ~/.ssh/identity for protocol version 1, and ~/.ssh/id_dsa,
~/.ssh/id_ecdsa and ~/.ssh/id_rsa for protocol version 2. Additionally, any identities represented by the authentication agent will be used for authentication. ssh(1) will
try to load certificate information from the filename obtained by appending -cert.pub to the path of a specified IdentityFile.
The file name may use the tilde syntax to refer to a user's home directory or one of the following escape characters: ‘%d’ (local user's home directory), ‘%u’ (local user
name), ‘%l’ (local host name), ‘%h’ (remote host name) or ‘%r’ (remote user name).
It is possible to have multiple identity files specified in configuration files; all these identities will be tried in sequence. Multiple IdentityFile directives will add
to the list of identities tried (this behaviour differs from that of other configuration directives).