We operate about 35,000 machines running kernel version 3.7.0 with udev version 171. The kernel has aufs patches and some other patches related to V4L2.
On about 200 of the machines, some combination of
/dev/sdaX and, sometimes, the by-label for it are missing. The vast majority (about 90%) of these are simply something wrong with the swap partition, the remaining 10% are more critical failures that would prevent boot or other important operations.
I've investigated a few and found that some of the following will sometimes resolve the problem.
partprobewill cause a missing by-label to reappear. That is, for example,
/dev/disk/by-label/OVERLAYBdoes not. Running
- Sometimes, the same symptoms are observed, but
- Sometimes, a partition appears corrupted in a way that the label is lost. For example,
/dev/disk/by-label/SWAPis missing and
swapon /dev/sda3fails with
read swap header failed: Invalid argument.
- Sometimes, a partition and its by-label are missing. A combination of
mkfsrestore the expected behavior.
- In another case,
/dev/sdaXand its by-label were missing. After
mkfs(with the appropriate
-L), and a paranoid
partprobe, the by-label was still missing.
Of note is that I have never seen
/proc/partitions be incorrect, even when a
/dev/sdaX is missing.
Also of note, in a few cases, the current root filesystem had a missing
/dev/sdaX mode. This would prevent boot! That indicates to me that the node exists on boot but later disappeared. Nothing seemed odd in
I know this is a lot of different stuff going on, but I believe that these are all symptoms of some deeper failure that we have not found. I believe that in correcting a problem, I'm only undoing a symptom and that the problem will happen again.
I'm really at a loss of what else to look at. Has anyone seen anything like this before? What could the underlying problem be? How do I even learn more about the situation, to help me narrow down the problem space?