1

Our database-server sometimes become unresponsive. It runs a fully updated Ubuntu 14.04 LTS. Notable non-vanilla software running on it are Nimbus, TSM and Oracle.

About once a day, it becomes unresponsive, so far at night-time, when a series of maintenance-tasks are done such as backups.

After it becomes unresponsive, it seems to stay this way forever. I'm not able to SSH into it, and it doesen't accept any database connections.

Weird thing is, the server responds to ping. If I use telnet to open port 22(SSH) or port 1521(Oracle), I do get a reply from the server. Port 22 even states something like "This is OpenSSH". But actually using the ssh client or opening a database-connection just hangs.

I've been looking in the log-files, and found absolutely nothing(dmesg, syslog, auth.log etc). It also seem to be suspiciously little activity in the the logfiles during the unresponsive period. After restarting the server, it works again.

My immediate reaction was to run apt-get update and apt-get dist-upgrade, and monitor the max file-descriptor limit being reached. However, the hard limit for Oracle is far from the filesystem max, so it seems weird if thats the case. Anyone else have any ideas what could cause this?

EDIT: Forgot to mention that CPU, memory and disk space was far from reaching 100%. (They were already monitored, and after this happened, I started monitoring open file descriptors as well, but it has yet to happen again). I can also add that I don't expect anyone to call out the exact problem, but any ideas for additional things to monitor would be appreciated.

closed as off-topic by Iain, kasperd, Jenny D, Ward, mdpc Feb 6 '16 at 0:41

This question appears to be off-topic. The users who voted to close gave this specific reason:

If this question can be reworded to fit the rules in the help center, please edit the question.

  • 4
    Collect hardware utilization information. That's always step #1 in troubleshooting performance problems. Once you have empirical data, you can make informed decisions. – EEAA Feb 3 '16 at 14:10
  • Use tools like htop, atop, vmstat. – Khaled Feb 3 '16 at 14:14
  • 1
    @HenrikKjusAlstad You need to collect usage trends it's not enough to take a one-shot view of the resources. Look into something like Munin. – EEAA Feb 3 '16 at 14:19
  • 1
    If SSH is unable to connect, check the DNS too – Dom Feb 3 '16 at 15:02
  • 1
    You need to employ Scientific Method not Shouting Halp at the internets method. – Iain Feb 3 '16 at 15:59
0

All variables looked quite normal. However, I wrote a cronjob to output the date/time and filedescriptors every minute, and found the filedescriptors to be within normal values. However, at 3am, the servers clock suddenly went 2 hours back in time(took me a while to notice that from the logfile), and then it died without any errors in logs.

It turned out to be a problem on the hosting/WMWare-level(which is not my concern). Among other things, the WMWare host had a time that was completely off. After the infrastructure company fixed their WMWare platform, it worked fine again.

Not the answer you're looking for? Browse other questions tagged or ask your own question.