BusyBox why u no vi?!
Ubuntu 14.04 Server after upgrade first boot, only boots to BusyBox v1.21.1 initramfs with "gave up waiting for root device" due to missing /dev/mapper/xxx-root as unable to find/mount LVM VG due to incorrect exclusion filter in lvm.conf
- only have the single kernel 3.11.0-17 (yes, no fallback!), both normal and recovery go to BusyBox
- only ssh access, or IP KVM (system in remote CoLo in DC over 10,000 miles away)
- no access to bootable media as IP KVM does not load virtual media
- default initramfs configuration (attempt to use vi shows vi: not found)
- LVM PV should be on something like /dev/sda5 (assume this as have many systems with same config and that is what the others are set as)
100% sure issue is incorrect LVM filter (configuring OpenStack Cinder loopback device for LVM poorly), consequently is not scanning for LVM VG. Known working should be default in lvm.conf:
filter = [ "a/.*/" ]
However this is commented out and instead have in lvm.conf:
filter = [ "a/sda1/", "a/sdb/", "r/.*/"]
If I can make the change in lvm.conf by searching for incorrect string and replacing line with correct string in lvm.conf then I believe LVM VG will boot correctly.
initramfs has no editing (vi, vim, nano, etc), only sed have tried testing sed regex, sadly this is incredibly difficult to crack for this regex noob. I have so far:
sed -i '/filter = [ "a/sda1/", "a/sdb/", "r/.*/"]/c\filter = [ "a/.*/" ]' /etc/lvm/lvm.conf
which makes no change to a test file. I have trouble comprehending the complexities of slashes contained within the sed search/replace string in conjunction with the sed command slash syntax, maybe someone can assist here? If I can crack the sed command regex I'm fairly sure the kernel should be able to boot into LVM.
Alternatively is there a force/manual mount I could use (from grub edit or initramfs), despite the incorrect lvm.conf filter, what is this?
As a fallback, I could go down the path of MaaS (Ubuntu's web based PXE/tftp image booting), which I have configured on another server in the DC, but have not yet tested, or used. I am happy to try to blast a new Ubuntu 14.04 Server onto the non-booting system using MaaS.
Correctly editing lvm.conf by cracking sed first in BusyBox is more desired. I'd like to understand this for future, or any other suggestions? Edit at grub, manual LVM VG boot, or any other suggestions that help are appreciated. Thank you.
Follow up edit:
(not sure on serverfault best practice here, whether I should answer my own question or make this edit, anyway..)
A big thank you to mvillar for the sed guidance. OK, here is the final synatax and steps that saved my bacon from an unbootable system (only to BusyBox) due to an error in lvm.conf filter configuration. I first used the following sed command to double check for correct syntax before and after change to '# print only lines which match regular expression (emulates "grep")':
sed -n '/loop/p' /etc/lvm/lvm.conf
So I discovered the actual incorrect line I had in lvm.conf (differing from what I originally stated):
filter = [ "a/sdc5*/","a/loop2*/", "r/.*/" ]
Note: I believe the cause of the booting error was incorrect syntax in the original lvm.conf. I wanted: filter to accept sda5 and loop2 and reject all others. I was incorrectly stating: sdc5 and possibly missing space after comma
sed regex that worked to replace incorrect syntax with 'accept all' filter in lvm.conf:
sed -i "s/filter \= \[ \"a\/sdc5\*\/\",\"a\/loop2\*\/\", \"r\/\.\*\/\" \]/filter = [ \"a\/.\*\/\" ]/" /etc/lvm/lvm.conf
Lessons learned: + escape not needed for spaces, needed for literal string forward slash, asterix and quotes + default Ubuntu 14.04 does NOT include vi or tail, despite them both listed as BusyBox commands in upstream, something awry here with Ubuntu BusyBox default build. I will in future check that all my systems include AT LEAST vi and tail in BusyBox builds! + sed is at first complicated to understand, but a really useful and powerful tool
Steps to view LVM status, recreate LVM vg and lv and boot from BusyBox after sed and checking:
(initramfs) lvm lvm> pvdisplay lvm> vgdisplay lvm> lvdisplay lvm> vgchange -a y lvm> exit (initramfs) exit
Ubuntu then continued normal boot with active root lv and went straight to login. Awesome!