On cloud platforms you often hear that because of high load on neighboring VMs, disks over oversubscribed ethernet, backups, or live migration to other hardware, the virtual machine can 'freeze' for a moment.
I have the suspicion that this is happening to one of our Ubuntu virtual machines on a Cloud provider I'm not looking to publicly shame.
Every night it's unavailable to external monitoring services. The machine itself looks healthy in terms of load, traffic, etc. The provider suggests the network is fine though.
I would like to be able to (dis)prove VM freezes are causing these pagers.
One idea I had was to write the date to a log every second, and after an short moment of unavailability see if we skipped a 'beat'.
However that seems flawed because what if the VM maintains its own clock and allows a drift from the Host's hardware.
If our internal clock freezes along with the VM, we'd still have a nice sequence of seconds in that log file, and a clock that's now behind on the real time.
Is there a better way / tool I can use to determine that there are machine freezes?
I would guess real time and our time would be a tell, then again, there are other causes for drifting clocks.