Is it possible to set a custom error message for users denied to access a server? In fact, I'd like to only display this message if a user was rejected due to IP filter set in
sshd_config. But one global error message would suffice.
By "error message" I believe you mean a text string being displayed by the client program. The word "message" has some context in the SSH authentication protocol RFC I'll reference later but I don't think that's the connotation you're using it in.
I believe you're thinking of SSH to be something like TELNET, where arbitrary text strings sent by the server daemon will be displayed by the client. SSH uses an authentication protocol that passes tokenized messages in the form of binary values back and forth between the client and server. One of these tokenized messages, "SSH_MSG_USERAUTH_BANNER", can be used to instruct the client to display arbitrary text strings (if it so chooses) but you'd have to modify your SSH daemon to send these at the appropriate time. I don't believe that OpenSSH has any built-in functionality to send SSH_MSG_USERAUTH_BANNER messages during arbitrary points in the authentication process.
I'd question why you'd want to give a remote attacker any additional information about why their authentication has failed. They know it's failed because their SSH client receives and reacts to the tokenized message from the SSH daemon signaling the failure.
EDIT: Evan Anderson types faster than I do, and seems to confirm that the message is not customizable on the server size.
You can specify a banner file in the sshd_config. The banner would be displayed to all users before the login prompt and state that only logins from certain domains and addresses is allowed.
SSH can pass the login off to the unix /bin/login command. Depending on your OS, you may have an option in there to customize the prompts and errors?
Anything else I can think of would probably mean editing code and recompiling. And it's possible that the failure message is not sent from the server to client as a string, but as a code, in order to prevent an attacker from collecting information. (If it's a string, and customizable, someone will set it to "Login Failed. OpenSSH_3.7_p2 21032004")