4

I need to login to a system that went into a read-only state. I can ping it just fine but I can't ssh in anymore. Is there some special command line flag/parameter I can pass ssh that lets me login into a system that has gone into read-only mode?

Forgot to add the exact connection error:

OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to 192.168.0.4 [192.168.0.4] port 22.
debug1: Connection established.
debug1: identity file /home/username/.ssh/identity type -1
debug1: identity file /home/username/.ssh/identity-cert type -1
debug1: identity file /home/username/.ssh/id_rsa type -1
debug1: identity file /home/username/.ssh/id_rsa-cert type -1
debug1: identity file /home/username/.ssh/id_dsa type -1
debug1: identity file /home/username/.ssh/id_dsa-cert type -1
ssh_exchange_identification: Connection closed by remote host

I should add that ping is pretty robust to that box:

ping 192.168.0.4
PING 192.168.0.4 (192.168.0.4) 56(84) bytes of data.
64 bytes from 192.168.0.4: icmp_seq=1 ttl=64 time=0.662 ms
64 bytes from 192.168.0.4: icmp_seq=2 ttl=64 time=0.088 ms
64 bytes from 192.168.0.4: icmp_seq=3 ttl=64 time=0.089 ms
Improve this question
31
  • 1
    Not usually, no; this is what remote console access is for.
    – MadHatter
    Apr 24, 2015 at 6:21
  • 2
    I don't positively know that sshd will refuse connections when it has no writable file system. I wouldn't assume that the ssh connection closed error necessarily is due to the file system problem. How, exactly do you know the fs has been remounted read-only? Are you using remote syslog?
    – caskey
    Apr 24, 2015 at 6:35
  • 4
    If the system is read-only, the ssh server can't write agent-related temporary files in /tmp, can't make new wtmp and lastlog entries, and so on. In brief, unless the ssh server has been configured to allow completely footprintless logins to a normally-working system, I think it would be highly-remiss of it to allow logins when those footprints can't be laid down - and even more of a security disaster to have a special flag that allowed the client to request such silent operation!
    – MadHatter
    Apr 24, 2015 at 6:42
  • 1
    That said, htfree, what do you get when you do telnet 192.168.0.4 22?
    – MadHatter
    Apr 24, 2015 at 6:44
  • 1
    @MadHatter Anybody security-aware would not forward an agent unless it was strictly needed. If you don't forward an agent, then those agent related files won't be needed. Logs are not written to disk by the ssh server itself. It sends log entries to syslog which writes them to disk. That means in case syslog is unable to write them, ssh could still go on. Updating wtmp and lastlog might happen within the sshd process, but I don't think a failure to update those would prevent a login.
    – kasperd
    Apr 24, 2015 at 6:53

2 Answers 2

Reset to default
1

You could login by invoking a no-login shell session. For example, you can pass a command to ssh after successfully authenticating:

ssh user@host bash --noprofile --norc

Of course, this uses bash as the shell. Different shells will require appropriate parameters in order not to trigger a wtmp/utmp update, as well as not trying to do things that could fail and log you off early (i.e., before the shell finishes up its usual tasks when normally opening up a login session).

A note of warning: the shell will be rather (very!) limited, usually without prompts and other fanciness. But it is enough for you to getting into the machine and checking what is wrong.

Edited to add: depending on the circumstances of your host, it could be necessary to specify a full path, for example /bin/bash as the command to execute upon successfully authenticating.

Improve this answer
4
  • It doesn't matter what command the OP asks the server to run. The ssh server is dropping the TCP connection immediately after it is made. He's not even authenticating.
    – Kenster
    Apr 28, 2015 at 15:19
  • Now that @Kenster raised the red flag, this could be a case where ssh and sshd are not agreeing on the authentication method. Especifically, auth is key-based, but there are no suitable keys to proceed. Other than this, I'm out of ideas. Perhaps ssh -vvv could give further insight (I'm assuming only ssh -vv was used, as the highest debug was debug2).
    – rnsanchez
    Apr 28, 2015 at 15:56
  • this could be a case where ssh and sshd are not agreeing on the authentication method Like I said, the server is dropping the connection as soon as it is made. No part of the ssh protocol is actually running. The very first thing the server would normally do is to send its software version string to the client in cleartext, and the server isn't even doing that, according to the debug trace.
    – Kenster
    Apr 28, 2015 at 16:02
  • the debug I had posted was already -vvv , also I tried ssh -vvv user@host bash --noprofile --norc and it gives the same exact result.
    – htfree
    Apr 28, 2015 at 23:22
1

the short answer is that your SSH server is probably nonfunctional due to the system problems which you've described. I don't think there is anything different the client could do.

I'll break down your debug trace:

debug1: Connecting to 192.168.0.4 [192.168.0.4] port 22.
debug1: Connection established.

The client made a connection to that address and port. This implies that the sshd process on the server is still running.

debug1: identity file /home/username/.ssh/identity type -1
debug1: identity file /home/username/.ssh/identity-cert type -1
debug1: identity file /home/username/.ssh/id_rsa type -1
debug1: identity file /home/username/.ssh/id_rsa-cert type -1
debug1: identity file /home/username/.ssh/id_dsa type -1
debug1: identity file /home/username/.ssh/id_dsa-cert type -1

Your local ssh client looked for those key files and didn't find them. These are all default keyfile names that it would normally look for. This is only a problem if you expected one of those files to be present. They're also not really relevant here, because the client never got a chance to authenticate.

ssh_exchange_identification: Connection closed by remote host

The remote server closed the TCP connection. This specific message means the server did a "normal" close on the connection. If the server had crashed, you'd see a different message saying "connection reset by peer".

Normally, the first thing an SSH server will do is to send its software version string. If that had happened here, you'd see this in the debug trace:

...
debug1: identity file /home/foo/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1.1
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1.1 pat OpenSSH*

The server isn't even getting this far before closing the connection.

If the server were healthy, the usual explanation for what you're seeing would be that the server is rejecting the client due to TCP wrappers. But in your case, something about the system state is probably preventing sshd from working properly. For example, immediately after accepting a connection the server will call [fork()][2] to create a child process. The child process handles the connection while the parent continues listening for further connections. If the fork fails, then the server will close the connection without sending anything to the client.

Improve this answer
1
  • thanks for the nice breakdown/explanation, I'm still waiting for my friend to install rsh so I can try rsync -e rsh and also MySQL client connection. Likley seems i'll have to go to the datacenter and just pick up the drives.
    – htfree
    Apr 28, 2015 at 23:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .