0

I have a VPS that runs a Java server, a redis server and a PostgreSQL server.

Everything is running well but last night my server "crashed" at 3:15:50 AM. At least that's what I thought.

I looked at my server log and I saw several exceptions caused by the database and then nothing more.

I looked at redis log :

[733 | signal handler] (1429751751) Received SIGTERM scheduling shutdown...
[733] 23 Apr 03:15:51.977 # User requested shutdown...
[733] 23 Apr 03:15:51.977 * Calling fsync() on the AOF file.
[733] 23 Apr 03:15:51.977 * Saving the final RDB snapshot before exiting.
[733] 23 Apr 03:15:51.996 * DB saved on disk
[733] 23 Apr 03:15:51.996 * Removing the pid file.
[733] 23 Apr 03:15:51.996 # Redis is now ready to exit, bye bye...

Ok, a SIGTERM.

I also looked at PostgreSQL's logs :

2015-04-23 03:15:50 CEST LOG:  received fast shutdown request
2015-04-23 03:15:50 CEST LOG:  aborting any active transactions
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST FATAL:  terminating connection due to administrator command
2015-04-23 03:15:50 CEST LOG:  autovacuum launcher shutting down
2015-04-23 03:15:50 CEST LOG:  shutting down
2015-04-23 03:15:50 CEST LOG:  database system is shut down

Mmmh. A shutdown request? A SIGTERM signal in other words, right?

I also looked at several system logs, like daemon.log and syslog :

Apr 23 03:15:47 vps89164 xinetd[12314]: Exiting...
Apr 23 03:15:48 vps89164 named[604]: received control channel command 'stop -p'
Apr 23 03:15:48 vps89164 named[604]: shutting down: flushing changes
Apr 23 03:15:48 vps89164 named[604]: stopping command channel on 127.0.0.1#953
Apr 23 03:15:48 vps89164 named[604]: stopping command channel on ::1#953
Apr 23 03:15:48 vps89164 named[604]: no longer listening on ::#53
Apr 23 03:15:48 vps89164 named[604]: no longer listening on 127.0.0.1#53
Apr 23 03:15:48 vps89164 named[604]: no longer listening on 127.0.0.2#53
Apr 23 03:15:48 vps89164 named[604]: no longer listening on 37.59.110.223#53
Apr 23 03:15:48 vps89164 named[604]: exiting

And I looked at auth.log :

sshd[435]: Received signal 15; terminating.
saslauthd[758]: server_exit     : master exited: 758

Signal 15. SIGTERM in other words.

So, tell me if I'm wrong but it seems that there was no crash at 3:15:50 AM but just a VPS restart right?

But why? Why would a restart appear? I didn't do anything (i was sleeping). How can I know what happened?

I look at my RAM usage (that was often an issue), but it was not even at 100% use (and CPU was only at 3% use).

Here is a capture of my RAM usage for the last 24 hours :

RAM usage for last 24 hours

When I look at auth.log I can also see a lot of connection tries :

Apr 23 03:15:43 vps89164 sshd[23481]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=mail.setac.com.py  user=root
Apr 23 03:15:44 vps89164 sshd[23479]: Failed password for root from 58.218.199.195 port 49241 ssh2
Apr 23 03:15:44 vps89164 sshd[23481]: Failed password for root from 181.40.125.54 port 46047 ssh2
Apr 23 03:15:45 vps89164 sshd[23481]: Received disconnect from 181.40.125.54: 11: Bye Bye [preauth]
Apr 23 03:15:45 vps89164 sshd[23484]: error: Could not load host key: /etc/ssh/ssh_host_ecdsa_key
Apr 23 03:15:46 vps89164 sshd[23479]: Failed password for root from 58.218.199.195 port 49241 ssh2
Apr 23 03:15:46 vps89164 sshd[23479]: Received disconnect from 58.218.199.195: 11:  [preauth]
Apr 23 03:15:46 vps89164 sshd[23479]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=58.218.199.195  user=root

This is only a sample, there are thousands of lines like these ones However no try seem to succeed so I don't think someone accessed the server and restarted it (and why only restarting a server when you have a root access to it and can do whatever you want?)

Now I'm lost, this is not the first time this is happening. This is annoying and I can't figure out why this is happening and how. I don't know how to search know and I'm requesting your help :(

By the way, I'm using an Ubuntu server

4
  • Well, it is a VPS hosted in some data center. Why would you rule out the possibility that the hoster just shut down the machine for management purposes? It probably was meant to restart afterwards, but didn't. I would file a support request with the hosting provider's. – the-wabbit Apr 23 '15 at 7:54
  • @the-wabbit 's suggestion is probably most likely. Your hosting provider may have replaced a hard drive or something in the hypervisor. If it happens again, I'd definitely submit a ticket though. – John Cave Apr 23 '15 at 8:06
  • I just asked my hoster, I'm waiting for an answer. But you may be right because I just noticed another VPS of mine restarted too – Fabien Henon Apr 23 '15 at 8:46
  • My hoster said they didn't do anything on the restart (no restart). How can I know why it restarted ? – Fabien Henon Apr 24 '15 at 10:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.