We have a dedicated remote server hosted by Hetzner. The OS is Debian 9.4 LAMP (Apache, Mysql, PHP installed by default). We use the server to analyze big data using python (a little), R (R-studio server open source) and Clickhouse (column-oriented database). So, these are the only things we use there.
There were situations when R processes took too much memory and we kill them by PID. Sometimes we use reboot to handle this. But last time after launching heavy R-processes server stop responding (when trying to connect via ssh it refuses connection). We rebooted it via hetzner robot-panel. Firstly it was automated reboot. After that we tried to connect via ssh or sftp - server refuses connection (or "network error. connection refused" was the answer). Then we ordered manual hardware reset - but again nothing changed. However, we could successfully ping the server.
Next, we activated rescue system which allows us to get access to the server within other OS so we could look at the logs and other files. The most strange thing that we found out - after first reboot syslog, daemon.log and auth.log haven't been updated. Only the file /var/log/wtmp was updated after the second reboot. When opened it last lines were about two reboots, everything other was again earlier than first reboot (which was performed at 14:27 25.09)
last lines of syslog:
Sep 25 08:48:56 Debian-94-stretch-64-LAMP systemd[17002]: Received SIGRTMIN+24 from PID 28008 (kill).
Sep 25 08:48:56 Debian-94-stretch-64-LAMP systemd[1]: Stopped User Manager for UID 1002.
Sep 25 08:48:56 Debian-94-stretch-64-LAMP systemd[1]: Removed slice User Slice of bor.
Sep 25 08:55:01 Debian-94-stretch-64-LAMP CRON[28072]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 09:05:01 Debian-94-stretch-64-LAMP CRON[28168]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 09:09:01 Debian-94-stretch-64-LAMP CRON[28200]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 25 09:09:20 Debian-94-stretch-64-LAMP systemd[1]: Starting Clean php session files...
Sep 25 09:09:20 Debian-94-stretch-64-LAMP systemd[1]: Started Clean php session files.
Sep 25 09:15:01 Debian-94-stretch-64-LAMP CRON[28320]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 09:17:01 Debian-94-stretch-64-LAMP CRON[28381]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[1]: Created slice User Slice of bor.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[1]: Starting User Manager for UID 1002...
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[1]: Started Session 2480 of user bor.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Listening on GnuPG cryptographic agent (access for web browsers).
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Listening on GnuPG cryptographic agent and passphrase cache.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Listening on GnuPG network certificate management daemon.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Reached target Paths.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Reached target Timers.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Reached target Sockets.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Reached target Basic System.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Reached target Default.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[28426]: Startup finished in 172ms.
Sep 25 09:21:08 Debian-94-stretch-64-LAMP systemd[1]: Started User Manager for UID 1002.
Sep 25 09:25:01 Debian-94-stretch-64-LAMP CRON[28472]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 09:35:01 Debian-94-stretch-64-LAMP CRON[28567]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 09:39:01 Debian-94-stretch-64-LAMP CRON[28610]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 25 09:39:17 Debian-94-stretch-64-LAMP systemd[1]: Starting Clean php session files...
Sep 25 09:39:17 Debian-94-stretch-64-LAMP systemd[1]: Started Clean php session files.
Sep 25 09:40:58 Debian-94-stretch-64-LAMP systemd[1]: Started Session 2485 of user bor.
Sep 25 09:45:01 Debian-94-stretch-64-LAMP CRON[28728]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 09:55:01 Debian-94-stretch-64-LAMP CRON[28844]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 10:01:31 Debian-94-stretch-64-LAMP systemd[1]: Started Session 2488 of user bor.
Sep 25 10:05:03 Debian-94-stretch-64-LAMP CRON[28950]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 10:09:03 Debian-94-stretch-64-LAMP CRON[28986]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 25 10:09:05 Debian-94-stretch-64-LAMP systemd[1]: Starting Clean php session files...
Sep 25 10:09:27 Debian-94-stretch-64-LAMP systemd[1]: Started Clean php session files.
Sep 25 10:15:04 Debian-94-stretch-64-LAMP CRON[29096]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 10:17:04 Debian-94-stretch-64-LAMP CRON[29116]: (root) CMD ( cd / && run-parts --report /etc/cron.hourly)
Sep 25 10:25:01 Debian-94-stretch-64-LAMP CRON[29195]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 10:35:01 Debian-94-stretch-64-LAMP CRON[29286]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 10:39:01 Debian-94-stretch-64-LAMP CRON[29328]: (root) CMD ( [ -x /usr/lib/php/sessionclean ] && if [ ! -d /run/systemd/system ]; then /usr/lib/php/sessionclean; fi)
Sep 25 10:39:02 Debian-94-stretch-64-LAMP systemd[1]: Starting Clean php session files...
Sep 25 10:39:03 Debian-94-stretch-64-LAMP systemd[1]: Started Clean php session files.
Sep 25 10:45:01 Debian-94-stretch-64-LAMP CRON[29450]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 10:55:02 Debian-94-stretch-64-LAMP CRON[29549]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 11:01:15 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libcrypto.so.1.0.2: cannot map zero-fill pages
Sep 25 11:04:24 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libcap-ng.so.0: failed to map segment from shared object
Sep 25 11:05:21 Debian-94-stretch-64-LAMP CRON[29619]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1)
Sep 25 11:05:22 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: /lib/x86_64-linux-gnu/libselinux.so.1: cannot allocate version reference table: Cannot allocate memory
Sep 25 11:06:17 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libgssapi_krb5.so.2: failed to map segment from shared object
Sep 25 11:07:14 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libdl.so.2: failed to map segment from shared object
Sep 25 11:08:11 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libaudit.so.1: failed to map segment from shared object
Sep 25 11:08:17 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libcrypto.so.1.0.2: cannot create shared object descriptor: Cannot allocate memory
Sep 25 11:08:45 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libcrypt.so.1: failed to map segment from shared object
Sep 25 11:09:07 Debian-94-stretch-64-LAMP cron[936]: Authentication failure
Sep 25 11:09:07 Debian-94-stretch-64-LAMP CRON[29635]: Authentication failure
Sep 25 11:09:12 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libpcre.so.3: failed to map segment from shared object
Sep 25 11:09:37 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libcrypt.so.1: cannot map zero-fill pages
Sep 25 11:09:49 Debian-94-stretch-64-LAMP systemd[1]: phpsessionclean.service: Failed to fork: Cannot allocate memory
Sep 25 11:09:50 Debian-94-stretch-64-LAMP systemd[1]: phpsessionclean.service: Failed to run 'start' task: Cannot allocate memory
Sep 25 11:09:50 Debian-94-stretch-64-LAMP systemd[1]: Failed to start Clean php session files.
Sep 25 11:09:54 Debian-94-stretch-64-LAMP systemd[1]: phpsessionclean.service: Unit entered failed state.
Sep 25 11:10:00 Debian-94-stretch-64-LAMP systemd[1]: phpsessionclean.service: Failed with result 'resources'.
Sep 25 11:11:53 Debian-94-stretch-64-LAMP sshd[980]: /usr/sbin/sshd: error while loading shared libraries: libcrypto.so.1.0.2: failed to map segment from shared object
Sep 25 11:15:03 Debian-94-stretch-64-LAMP cron[936]: (CRON) error (can't fork)
Sep 25 11:17:02 Debian-94-stretch-64-LAMP cron[936]: (CRON) error (can't fork)
Sep 25 11:25:03 Debian-94-stretch-64-LAMP cron[936]: (CRON) error (can't fork)
Sep 25 11:57:02 Debian-94-stretch-64-LAMP systemd[28426]: Time has been changed
Sep 25 11:59:34 Debian-94-stretch-64-LAMP cron[936]: (CRON) error (can't fork)
Sep 25 12:03:49 Debian-94-stretch-64-LAMP systemd[1253]: Time has been changed
Sep 25 12:11:51 Debian-94-stretch-64-LAMP systemd[1253]: Time has been changed
Sep 25 12:18:00 Debian-94-stretch-64-LAMP systemd[28426]: Time has been changed
Sep 25 12:25:13 Debian-94-stretch-64-LAMP systemd[28426]: Time has been changed
Sep 25 12:31:17 Debian-94-stretch-64-LAMP systemd[1253]: Time has been changed
Sep 25 12:35:57 Debian-94-stretch-64-LAMP systemd[28426]: Time has been changed
Sep 25 12:41:15 Debian-94-stretch-64-LAMP systemd[1253]: Time has been changed
Sep 25 12:47:28 Debian-94-stretch-64-LAMP systemd[28426]: Time has been changed
Sep 25 12:57:01 Debian-94-stretch-64-LAMP systemd[1253]: Time has been changed
Sep 25 13:04:57 Debian-94-stretch-64-LAMP systemd[28426]: Time has been changed
Sep 25 13:15:07 Debian-94-stretch-64-LAMP systemd[28426]: Time has been changed
Sep 25 13:27:15 Debian-94-stretch-64-LAMP systemd[1253]: Time has been changed
and auth.log:
Sep 25 11:02:41 Debian-94-stretch-64-LAMP sshd[29602]: Disconnected from xx.xxx.xxx.xx port 29479 [preauth]
Sep 25 11:02:41 Debian-94-stretch-64-LAMP sshd[29595]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=xx.xxx.xxx.xx user=root
Sep 25 11:02:41 Debian-94-stretch-64-LAMP sshd[29599]: Connection closed by xx.xxx.xxx.xx port 31275 [preauth]
Sep 25 11:02:45 Debian-94-stretch-64-LAMP sshd[29595]: Failed password for root from xx.xxx.xxx.xx port 39659 ssh2
Sep 25 11:02:45 Debian-94-stretch-64-LAMP sshd[29595]: Received disconnect from xx.xxx.xxx.xx port 39659:11: [preauth]
Sep 25 11:02:45 Debian-94-stretch-64-LAMP sshd[29595]: Disconnected from xx.xxx.xxx.xx port 39659 [preauth]
Sep 25 11:04:33 Debian-94-stretch-64-LAMP sshd[29608]: ssh_dispatch_run_fatal: Connection from xx.xxx.xxx.xx port 15191: Broken pipe [preauth]
Sep 25 11:04:33 Debian-94-stretch-64-LAMP sshd[29609]: ssh_dispatch_run_fatal: Connection from xxx.xx.xxx.xxx port 48834: Broken pipe [preauth]
Sep 25 11:05:21 Debian-94-stretch-64-LAMP CRON[29615]: pam_unix(cron:session): session opened for user root by (uid=0)
Sep 25 11:05:22 Debian-94-stretch-64-LAMP sshd[29606]: Connection closed by xx.xxx.xxx.xx port 22703 [preauth]
Sep 25 11:05:23 Debian-94-stretch-64-LAMP CRON[29615]: pam_unix(cron:session): session closed for user root
Sep 25 11:07:04 Debian-94-stretch-64-LAMP sshd[29614]: Did not receive identification string from xx.xx.xxx.xx port 36140
Sep 25 11:07:22 Debian-94-stretch-64-LAMP sshd[29622]: Connection closed by xx.xxx.xxx.xx port 34280 [preauth]
Sep 25 11:07:45 Debian-94-stretch-64-LAMP sshd[29629]: error: rexec of /usr/sbin/sshd failed: Cannot allocate memory
Sep 25 11:08:22 Debian-94-stretch-64-LAMP sshd[29629]: Connection closed by xx.xxx.xxx.xx port 19048 [preauth]
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM unable to dlopen(pam_unix.so): /lib/security/pam_unix.so: cannot open shared object file: No such file or directory
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM adding faulty module: pam_unix.so
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM unable to dlopen(pam_permit.so): /lib/security/pam_permit.so: cannot open shared object file: No such file or directory
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM adding faulty module: pam_permit.so
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM unable to dlopen(pam_loginuid.so): /lib/security/pam_loginuid.so: cannot open shared object file: No such file or directory
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM adding faulty module: pam_loginuid.so
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM unable to dlopen(pam_env.so): /lib/security/pam_env.so: cannot open shared object file: No such file or directory
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM adding faulty module: pam_env.so
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM unable to dlopen(pam_limits.so): /lib/security/pam_limits.so: cannot open shared object file: No such file or directory
Sep 25 11:09:05 Debian-94-stretch-64-LAMP CRON[29635]: PAM adding faulty module: pam_limits.so
Sep 25 11:10:13 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:10:58 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:11:03 Debian-94-stretch-64-LAMP sshd[29642]: ssh_dispatch_run_fatal: Connection from xx.xxx.xxx.xx port 19637: Broken pipe [preauth]
Sep 25 11:11:31 Debian-94-stretch-64-LAMP sshd[29641]: Connection closed by xx.xxx.xxx.xx port 6478 [preauth]
Sep 25 11:12:53 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:13:03 Debian-94-stretch-64-LAMP sshd[29652]: error: rexec of /usr/sbin/sshd failed: Cannot allocate memory
Sep 25 11:13:30 Debian-94-stretch-64-LAMP sshd[29652]: Connection closed by xx.xxx.xxx.xx port 64145 [preauth]
Sep 25 11:13:51 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:13:52 Debian-94-stretch-64-LAMP sshd[29654]: error: rexec of /usr/sbin/sshd failed: Cannot allocate memory
Sep 25 11:13:55 Debian-94-stretch-64-LAMP sshd[29654]: fatal: fork of unprivileged child failed
Sep 25 11:14:18 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:14:42 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:14:53 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:15:05 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:15:28 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:15:51 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:15:51 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:16:13 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:16:36 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:16:49 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:17:00 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:17:22 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:17:45 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:17:47 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:18:08 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:18:32 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:18:47 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:18:56 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:19:19 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:19:44 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:19:45 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:19:45 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:20:00 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:20:07 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:20:13 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:20:14 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:20:29 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:20:29 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:20:43 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:20:45 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:20:52 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:21:15 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:21:39 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:21:47 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:22:03 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:22:27 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:22:48 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:22:50 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:23:13 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:23:36 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:23:48 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:24:00 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:24:24 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:24:47 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:24:48 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:25:10 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:25:24 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:25:33 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:25:48 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:25:56 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:25:57 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:26:19 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:26:41 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:26:46 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:27:03 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:27:26 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:27:40 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:27:50 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:28:14 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:28:36 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:28:37 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:29:01 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:29:24 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:29:32 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:29:47 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:30:02 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:30:10 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:30:23 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:30:33 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:30:55 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:32:49 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:35:11 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:37:47 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:40:18 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:43:13 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:45:53 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:48:33 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:51:21 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:52:43 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 11:55:22 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 12:08:23 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 12:15:50 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 12:22:35 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 12:28:55 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 12:33:58 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 12:38:48 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 12:44:35 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 12:52:51 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 13:03:08 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 13:10:35 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
Sep 25 13:21:40 Debian-94-stretch-64-LAMP sshd[980]: error: fork: Cannot allocate memory
It seems that R-processes overcommit memory and sshd and other processes coudn't get any of it. But we can't figure out what happened after reboot, why server still refuses the connection and don't update logs.
We performed fast smartctl disk checks (everything ok) and exchanged the server (via Hetzner support) while keeping the drives. So, it seems to be not a hardware problem.
