This is not an issue about logging in via SSH, this is a lag during typing input and receiving output while connected to a server via SSH. I tested the speed of the commands using the time command and see that they are running normally but that input to and output from the terminal is lagging. VNC is fast. Pinging remote servers from this server shows normal response times. SSH sessions to other machines from my machine seem fine. openssh-server 5.3p1-84.1.el6
The fact that VNC is fast should be enough, but for info, RAM is fine, no swapping is occurring, CPU usage is low, ethtool shows network at 1 Gigabit.