As @sufferer mentioned in another answer, some Linux distros include monitors to protect from brute force attacks on external visible services like SSH, for example
fail2ban. These monitors check the log files looking for failed attempts and add filters to block IP addresses that have too many failures (the number is configurable and independent from the sshd config).
If your distro includes
fail2ban, which protect services adding rules to the iptables firewall, you could check which services or "jails" are supervised using the command:
sudo fail2ban-client status
The jail for the SSH service is sshd, so to check if there are banned IPs you can use:
sudo fail2ban-client status sshd
and to unban some IP a.b.c.d:
sudo fail2ban-client set sshd unbanip a.b.c.d
If you have
DenyHosts, the banned list is in the file /etc/hosts.deny; you can edit this file directly as root. To grant some IP a.b.c.d permanent access, you could add the line
sshd:a.b.c.d to the file /etc/hosts.allow.
As always, the
man command is your friend:
There should exist other similar utilities, but I only have used these.
Note that increasing the number of retries allowed in the sshd configuration does not free banned IPs, only permits more failures in the same connection. If the number allowed is exceeded, the user/attacker simply reconnects again to try n times more.
Other services had the ban list integrated (as shown in the answer of Rajnesh Thakur about restarting the VNC server).
Note: If you use fail2ban and use a jail.local file for configuring jails, add the following to the top of the file to stop your ip address being banned:
ignoreip = 10.10.1.1 10.0.2.0/24
Restart fail2ban afterward, for example using systemctl restart fail2ban. The first IP allows a single address, the second allows all of subnet 10.0.2.x; adjust as required.