5

On cloud platforms you often hear that because of high load on neighboring VMs, disks over oversubscribed ethernet, backups, or live migration to other hardware, the virtual machine can 'freeze' for a moment.

I have the suspicion that this is happening to one of our Ubuntu virtual machines on a Cloud provider I'm not looking to publicly shame.

Every night it's unavailable to external monitoring services. The machine itself looks healthy in terms of load, traffic, etc. The provider suggests the network is fine though.

I would like to be able to (dis)prove VM freezes are causing these pagers.

One idea I had was to write the date to a log every second, and after an short moment of unavailability see if we skipped a 'beat'.
However that seems flawed because what if the VM maintains its own clock and allows a drift from the Host's hardware.
If our internal clock freezes along with the VM, we'd still have a nice sequence of seconds in that log file, and a clock that's now behind on the real time.

Is there a better way / tool I can use to determine that there are machine freezes?

I would guess real time and our time would be a tell, then again, there are other causes for drifting clocks.

3
  • If you're not seeing a problem in the metrics you're collecting then either you're collecting the wrong metrics or you alerting is wrong.
    – user9517
    Nov 27, 2014 at 14:39
  • 2
    So then what would be the right metric to detect a VM freeze?
    – kvz
    Nov 27, 2014 at 14:56
  • No idea old chap too may possibilities but you do seem to be in possession of information about what it's not so start monitoring other things.
    – user9517
    Nov 27, 2014 at 16:53

3 Answers 3

1

I think you're on the right track in with writing the time to a log file every second, but for the reasons you pointed out that may not be reliable. In addition to writing the time to a local disk, why not have your cron process reach out to a known stable system over the network and have that system log the request to disk? Something as simple as wget could work assuming you're doing an http request to a system and that system is logging the requests. Of course, you'd ideally want to have the target system relatively "close" to the system you suspect of being problematic network-wise, but that could help you get some debugging data at least.

0

You could use Nagios, an IT solution for monitoring. With this, you could check the CPU load (and a lot of other things), and receive an alert to your mail or by the web console. You have to install the server in you PC and the remote plugin executor in the Virtual Machine.

Here's a pretty cool tutorial: http://www.tecmint.com/how-to-add-linux-host-to-nagios-monitoring-server/

1
  • When other VMs have high load and steal resources, it won't show on our graphs. We do get timeouts from external monitoring such as Pingdom though. So the question is, is there a way I could prove to my Cloud provider that VM freezes are what's causing this (vs flaky network etc). I would think there would be a way as relative to real time, not a single thing happened on the server. Would a sudden jump in time be sufficient? Would that only work if the VM syncs its clock with the host? Would it be hard enough evidence as there could be other factors causing a jump in time?
    – kvz
    Nov 27, 2014 at 20:57
0

I would ping a VM from an external host. If the VM freezes, its network stack should too and it should show in a sequence of logged pings.

2
  • That's a good idea, but this does not rule out flaky network.
    – kvz
    Nov 28, 2014 at 8:08
  • @kvz If the issue only touches one of your VMs, then doing the very same logging also with other VMs might provide the required additional evidence if it is network or not (unless flaky virtual network within the cloud is to be suspected)? Also comparing results of VM to VM and external host to VM might give some insight. However, it's hard to imagine a way to conclusively test anything within a container that you can't trust.
    – zagrimsan
    Dec 1, 2014 at 8:50

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .