sometimes it seems to be impossible to establish a connection to this server. Later when a connection could then be established I am not able to see any hints what was causing this behaviour. When I am already connected to the server, it sometimes happens that I am not able to do anything since it won´t react (sometimes the connection will be lost after some time, sometimes it can be kept). It seems to be the same times when no connection can be established at all.

For connecting ssh the error message is:

ssh: connect to host myhost port 22: Connection timed out

The server shouldn´t be to busy according to the cpu / memory at any given time. I´ve already checked the memory with MemTest86+ without any errors.

dmesg lists no further messages for this.

Does anyone have a clue, what I might check / look up?

Kind regards

  • My first reaction would be to look at the network between you and that server. Or if you are very familiar with this server, look through its logs find any other network glitches/delays. So: #1 Do you connect to other servers in the same way: if so, do they have the problem too? #2 Get help from your network/hosting/cloud engineer. – DutchUncle Jan 29 '15 at 11:43
  • 1
    @DutchUncle: Thank you for the response! Actually I´m connecting to two other Ubuntu servers in the same way where we never experienced such problems. What I forgot to mention: Pinging the server works just fine. Currently I´m writing a ticket for the hosting service. – bloodyroo7 Jan 29 '15 at 14:59

We see this behavior under either high load conditions or, more surprisingly, big file write conditions. You already ruled-out the high load. Let me explain the second scenario.

This is a real scenario, and happened as recently as a few days ago:

  1. Assume large amounts of RAM, relative to disk write speed (32 GB RAM, 100 MB/sec)

  2. An application causes a quick write of about 20 GB, where the data is either from a cached source or generated, such that the write is buffered into 20 GB of RAM and written in the background.

  3. A "fsync" at the end of this 20 GB write. The application blocks and waits 200 seconds for the write to complete.

Now for the tricky part:

  1. During this 200 seconds of fsync write, you try to login to SSH or even (virtual) console.

  2. The login process tries to fsync log entries about your login.

  3. This fsync is stalled by the previous fsync, waiting for up to 200 seconds before it completes.

  4. The login process times-out with the message you see.

The whole time the machine is pingable. Also, things that do not issue "fsync" generally work correctly.

Please note, this happens on our CentOS 5 servers, I've read that Theodore (Ted) Ts'o made improvements to newer kernels better manage unrelated concurrent fsyncs.

  • When this happens the shell and nano won´t react on input. Would you see the shell as something that triggers fsync? – bloodyroo7 Feb 11 '15 at 14:57
  • Not sure about nano, but vi does occasional automatic saves that will also stall for the duration of the long fsync. – Dan Armstrong Feb 23 '16 at 0:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.