5

We are experiencing crashes on our SSD VPSes, all working on KVM: crashes occur for different reasons; in the hurry to restore the service my team uses to reload a previous snapshot of the machine and never saves the logs.

Anyway, among all the different crashes circumstances, a recurring fact is the corruption of in-memory data: our VPS provider told us their hardware is running fine, yet I don't know how to read the poor log I was given.

enter image description here

What is involved when an "in-memory data corruption" is detected? Could it be because of broken RAM, or there are other kinds of memory corruption?

Funny thing: a VPS provider using VMware never gave us troubles, the one using KVM is really driving us crazy because of these crashes.

Edit 1: I, by no means, demand that you people deduce the solution from this miserable log. I'm stuck with this issue where no decent log is provided, memtest is useless since the hardware is emulated and the VPS provider granted that their hardware is fine and no instances of KVM or QEMU crashed. corruption of in-memory data detected is haunting me, and I can't think of any productive approach to investigate further this issue.

Improve this question
3
  • Do you know what is the host OS and configuration the hoster provides? I have seen such issues with a large cloud provider who was using ubuntu 14 for their hosts, and were running KSM non-stop in a loop (because they had no idea how to do it properly).
    – dyasny
    Feb 20, 2018 at 13:51
  • The provider keeps its KVM configurations secret so I don't know what arguments they use with KVM, nor the underlying OS.
    – elmazzun
    Feb 20, 2018 at 13:55
  • See if you can reproduce this on another provider
    – dyasny
    Feb 20, 2018 at 17:57

1 Answer 1

Reset to default
5

"Corruption of in-memory data detected" doesn't necessarily mean that the hardware RAM is bad. It could also indicate a block was read or written incorrectly, the storage flipped a bit or is otherwise failing, possibly filesystem bugs, and a few other causes.

Reverting to a snapshot probably won't resolve the problem if there is some latent filesystem corruption; it'll just show up again later.

Instead, you should xfs_repair the filesystem, but since it's the root filesystem you'll need to boot from installation media or a rescue environment provided by your VPS provider.

If xfs_repair fails to repair the filesystem, you may run xfs_repair -L which will clear the XFS log (which may itself be corrupt) and then try to repair the filesystem again.

Improve this answer
3
  • Knowledgebase of RedHat is unavailable to me since I'm not a customer, but at least you gave a direction. Thanks a lot
    – elmazzun
    Feb 20, 2018 at 8:15
  • @elmazzun I missed your comment for some reason. If you sign up for the free Red Hat subscription you will also get access to their knowledge base.
    – Michael Hampton
    Apr 9, 2019 at 16:16
  • Just ran into this on VMWare ESXi, didn't expect to be able to run a command like this from the emergency shell.. but gosh darn it actually worked.. and it worked well.
    – Josh C
    Aug 11, 2022 at 17:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .