2

We operate about 35,000 machines running kernel version 3.7.0 with udev version 171. The kernel has aufs patches and some other patches related to V4L2.

On about 200 of the machines, some combination of /dev/sdaX and, sometimes, the by-label for it are missing. The vast majority (about 90%) of these are simply something wrong with the swap partition, the remaining 10% are more critical failures that would prevent boot or other important operations.

I've investigated a few and found that some of the following will sometimes resolve the problem.

  1. Sometimes partprobe will cause a missing by-label to reappear. That is, for example, /dev/sda6 exists but /dev/disk/by-label/OVERLAYB does not. Running partprobe restores OVERLAYB.
  2. Sometimes, the same symptoms are observed, but partprobe does nothing.
  3. Sometimes, a partition appears corrupted in a way that the label is lost. For example, /dev/sda3 exists, but /dev/disk/by-label/SWAP is missing and swapon /dev/sda3 fails with read swap header failed: Invalid argument. mkswap corrects this.
  4. Sometimes, a partition and its by-label are missing. A combination of mknod and mkfs restore the expected behavior.
  5. In another case, /dev/sdaX and its by-label were missing. After mknod, mkfs (with the appropriate -L), and a paranoid partprobe, the by-label was still missing.

Of note is that I have never seen /proc/partitions be incorrect, even when a /dev/sdaX is missing.

Also of note, in a few cases, the current root filesystem had a missing /dev/sdaX mode. This would prevent boot! That indicates to me that the node exists on boot but later disappeared. Nothing seemed odd in dmesg.

I know this is a lot of different stuff going on, but I believe that these are all symptoms of some deeper failure that we have not found. I believe that in correcting a problem, I'm only undoing a symptom and that the problem will happen again.

I'm really at a loss of what else to look at. Has anyone seen anything like this before? What could the underlying problem be? How do I even learn more about the situation, to help me narrow down the problem space?

1

Faced the similar issue with Crucial M500 SSD drives. The drives suddenly disappeared but to see them again power off/on was required. Upgrading the drive firmware from MU02 to MU05(the latest) fixes the issue. The mystery can also be related to BIOS and drive controller firmware version.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.