I'm getting some bizarre behavior with Ubuntu Server 10.04 64bit on two of our new servers (both fresh installs). I have ubuntu server (same version) deployed on 4-5 other servers without this issue.
Initially I cannot ssh into a fresh server install until I manually set the address that the ssh server is listening on in /etc/ssh/sshd_config. Once I've connected, I seem to be kicked out at random intervals with the following error:
Write failed: Broken pipe
Using "ssh -vv" doesn't show any other information. When I'm kicked out in this manner, I cannot reconnect for another seemingly random period of time. Sometimes a few seconds, others a few minutes. If I run "netstat -nap|grep :22", I can see that my connection still exists after the write failed error. I can't seem to re-connect until that connection drops.
After one of these errors, if I hop onto the server from the console, ssh into another machine, and then attempt to ssh back into the server, everything works fine.
Using "-o TCPKeepAlive=yes" client side doesn't seem to effect anything. I've disabled both iptables and ufw on the server. AppArmor is not showing any enforced profiles and SELinux isn't installed.
My logs aren't reporting any errors and I don't have any custom configs. This is a box-stock install. Note that when I try to get back in after the broken pipe error, this is the error I get:
ssh: connect to host 172.22.50.92 port 22: Connection refused
And nmap no longer shows port 22 as being open, though netstat on the server says it's still listening on port 22.
EDIT - I'm not sure if it means anything, but I've installed KVM on these hosts and I can ssh into the guests (ubuntu server 64bit as well) without any issue.
UPDATE - I've tried purging openssh and re-installing with apt. I've also purged and installed openssh from source with no luck. traceroutes and pings overnight show no packet loss whatsoever.
YET ANOTHER UPDATE - Dell seems to think that we've got a bad motherboard in the server. Having that replaced to see if it resolves the issue.