5

On cloud platforms you often hear that because of high load on neighboring VMs, disks over oversubscribed ethernet, backups, or live migration to other hardware, the virtual machine can 'freeze' for a moment.

I have the suspicion that this is happening to one of our Ubuntu virtual machines on a Cloud provider I'm not looking to publicly shame.

Every night it's unavailable to external monitoring services. The machine itself looks healthy in terms of load, traffic, etc. The provider suggests the network is fine though.

I would like to be able to (dis)prove VM freezes are causing these pagers.

One idea I had was to write the date to a log every second, and after an short moment of unavailability see if we skipped a 'beat'.
However that seems flawed because what if the VM maintains its own clock and allows a drift from the Host's hardware.
If our internal clock freezes along with the VM, we'd still have a nice sequence of seconds in that log file, and a clock that's now behind on the real time.

Is there a better way / tool I can use to determine that there are machine freezes?

I would guess real time and our time would be a tell, then again, there are other causes for drifting clocks.

  • If you're not seeing a problem in the metrics you're collecting then either you're collecting the wrong metrics or you alerting is wrong. – Iain Nov 27 '14 at 14:39
  • 2
    So then what would be the right metric to detect a VM freeze? – kvz Nov 27 '14 at 14:56
  • No idea old chap too may possibilities but you do seem to be in possession of information about what it's not so start monitoring other things. – Iain Nov 27 '14 at 16:53
1

I think you're on the right track in with writing the time to a log file every second, but for the reasons you pointed out that may not be reliable. In addition to writing the time to a local disk, why not have your cron process reach out to a known stable system over the network and have that system log the request to disk? Something as simple as wget could work assuming you're doing an http request to a system and that system is logging the requests. Of course, you'd ideally want to have the target system relatively "close" to the system you suspect of being problematic network-wise, but that could help you get some debugging data at least.

0

You could use Nagios, an IT solution for monitoring. With this, you could check the CPU load (and a lot of other things), and receive an alert to your mail or by the web console. You have to install the server in you PC and the remote plugin executor in the Virtual Machine.

Here's a pretty cool tutorial: http://www.tecmint.com/how-to-add-linux-host-to-nagios-monitoring-server/

  • When other VMs have high load and steal resources, it won't show on our graphs. We do get timeouts from external monitoring such as Pingdom though. So the question is, is there a way I could prove to my Cloud provider that VM freezes are what's causing this (vs flaky network etc). I would think there would be a way as relative to real time, not a single thing happened on the server. Would a sudden jump in time be sufficient? Would that only work if the VM syncs its clock with the host? Would it be hard enough evidence as there could be other factors causing a jump in time? – kvz Nov 27 '14 at 20:57
0

I would ping a VM from an external host. If the VM freezes, its network stack should too and it should show in a sequence of logged pings.

  • That's a good idea, but this does not rule out flaky network. – kvz Nov 28 '14 at 8:08
  • @kvz If the issue only touches one of your VMs, then doing the very same logging also with other VMs might provide the required additional evidence if it is network or not (unless flaky virtual network within the cloud is to be suspected)? Also comparing results of VM to VM and external host to VM might give some insight. However, it's hard to imagine a way to conclusively test anything within a container that you can't trust. – zagrimsan Dec 1 '14 at 8:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.