Is my server being hacked in?

Question

I need help to decide if my server has been hacked in.

The server has sshd turned on. The other day, I tried lsof -i TCP to see someone is connecting to the server before I restart it. Then I noticed the following output:

sshd     19400    root    3u  IPv6 224125      0t0  TCP [2600:3c01::f03c:91ff:fe57:c70b]:ssh->123.456.789.123:59030 (ESTABLISHED)
sshd     19409    me    3u  IPv6 224125      0t0  TCP [2600:3c01::f03c:91ff:fe57:c70b]:ssh->123.456.789.123:59030 (ESTABLISHED)
sshd     10623    root    3u  IPv4 123099      0t0  TCP www.playandlearn.xyz:ssh->221.131.165.86:49085 (ESTABLISHED)
sshd     10624    sshd    3u  IPv4 123099      0t0  TCP www.playandlearn.xyz:ssh->221.131.165.86:49085 (ESTABLISHED)

The first two lines are for my ssh connection (I changed my ip address to a fake one). The user name shown in last line is sshd, not something I expected. I then searched the ip address 221.131.165.86, and found out that some sites listed it as malicious address.

Trying to figure out how this address logged in the server, I grepped it from /var/log/auth.log, but only find (a lot of) lines like:

Mar 23 18:59:42 www sshd[16055]: Received disconnect from 221.131.165.86 port 19152:11:  [preauth]
Mar 23 18:59:42 www sshd[16055]: Disconnected from 221.131.165.86 port 19152 [preauth]

There was no entries for successful login from this address.

After a few hours, the connection was disconnected. Since then, I randomly checked a few times and hasn't seen any connections from the suspicious address.

But today I noticed the same symptoms for another ip address: showed up as user "sshd" in lsof, a lot of unsuccessful tries to connect, but no successful login entry in /var/log/auth.log.

So my questions are:

1. Are those listings in lsof -i TCP evidences that they have connected successfully, or they are just trying to connect. Note that in the first case, the connection seems lasted for a few hours.
2. If they have hacked in, why I can't find relative logs in auth.log?
3. After wrote down all these, I realize that maybe those connections are connections to my web server? One of the pages uses web socket, that might explains the long connection time?

Thanks for your help. Please let know if I need to provide more info.

Answer 1

I don't think so, it seems you are just scanned. That sshd user appears there because sshd daemon - IIUC - has priviledge separation and during initial authentication of a user it spawns new process under sshd user.

You can try yourself:

watch -n 0.5 "lsof -ni TCP | grep :ssh"

See first output:

Every 0.5s: lsof -ni TCP | grep :ssh                                                                                                                                                            localhost.localdomain: Wed Mar 24 22:07:58 2021

sshd    10010    root    3u  IPv4 1532608      0t0  TCP *:ssh (LISTEN)
sshd    10010    root    4u  IPv6 1532610      0t0  TCP *:ssh (LISTEN)

And try to login into non-existing user (I assume you allow password authentication here, thus it waits for user password).

ssh jwwj@localhost
Password:

And you should see something like...

Every 0.5s: lsof -ni TCP | grep :ssh                                                                                                                                                            localhost.localdomain: Wed Mar 24 22:02:57 2021

sshd    10010       root    3u  IPv4 1532608      0t0  TCP *:ssh (LISTEN)
sshd    10010       root    4u  IPv6 1532610      0t0  TCP *:ssh (LISTEN)
ssh     11218       jiri    3u  IPv4 1543472      0t0  TCP 127.0.0.1:38662->127.0.0.1:ssh (ESTABLISHED)
sshd    11219       root    4u  IPv4 1544285      0t0  TCP 127.0.0.1:ssh->127.0.0.1:38662 (ESTABLISHED)
sshd    11220       sshd    4u  IPv4 1544285      0t0  TCP 127.0.0.1:ssh->127.0.0.1:38662 (ESTABLISHED)
sshd    11221       root    4u  IPv4 1544285      0t0  TCP 127.0.0.1:ssh->127.0.0.1:38662 (ESTABLISHED)

See that my (jiri) local ssh process to localhost caused presence of a process with PID 11220.

(BTW this was tested on OpenSUSE Tumbleweed.)

Info about sshd privsep https://security.stackexchange.com/a/115905/199910 and openssh project's presentation https://www.openbsd.org/papers/openssh-measures-asiabsdcon2007-slides.pdf