I have a virtual Centos 5.6 on VmWare. SUDDENLY, it stopped working and it seems the file systems went read only.

cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 ro,data=ordered 0 0
...

It seems here that's a known "problem". But I tried the "solution" and I got:

mount -o remount /
mount: block device /dev/VolGroup00/LogVol00 is write-protected, mounting read-only

I don't know exactly what to look for in dmesg...here the only reference to "read-only":

grep -B 10 "read-only" dmesg

Initializing IPsec netlink socket
NET: Registered protocol family 1
NET: Registered protocol family 17
Using IPI No-Shortcut mode
ACPI: (supports S0 S1 S4 S5<6>Time: tsc clocksource has been installed.)
Initalizing network drop monitor service
Freeing unused kernel memory: 228k freed
Write protecting the kernel read-only data: 413k

Of course everything is stopped, web server db, not working etc. Worst problem I ever saw, completely in the dark.

Any suggestion welcome.

Now I tried

fsck /   

too many errors asking confirmation

fsck -a / 

ended with

/dev/VolGroup00/LogVol00: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)

and then

fsck -y /
/dev/VolGroup00/LogVol00: ***** FILE SYSTEM WAS MODIFIED *****
/dev/VolGroup00/LogVol00: ***** REBOOT LINUX *****
/dev/VolGroup00/LogVol00: 76279/11250176 files (3.6% non-contiguous), 1638779/11247616 blocks

hmmm now...it doesnt reboot??? shutdown -r now does nothing....reboot is even better:

reboot: error while loading shared libraries: libxml2.so.2: cannot open shared object  file: No such file or directory

migrated from stackoverflow.com Apr 29 '13 at 21:13

This question came from our site for professional and enthusiast programmers.

  • You probably have a dmesg output full of errors at the end. What's the first one you see when things get ugly? – Zabuzzman Apr 29 '13 at 21:54
  • I wrote up there what I thought it was interesting in dmesg, probably not enough but now I've lost root access and also ssh access. No way to go back in. I have to wait for reboot tomorrow. And I want to understand this fully, because now it happened on the test environment...On the production server it will be several hours downtime, not nice. – Glasnhost Apr 29 '13 at 22:07
  • fsck / seems really strange to me, read-only or not. You should check a root file system from another Linux. Install a small service Linux for such cases if you don't already have one. – Hauke Laging Apr 30 '13 at 11:48
  • @Hauke what is exactly a "small service linux" ? Sound useful but I dont understand what you mean – Glasnhost Apr 30 '13 at 13:31
  • @Glasnhost That's how I call a Linux whose only purpose is to lighten repairing the main installation. This can be a very small (<1GiB) text mode Linux. You just need the file (system) tools. And backup software may be quite useful. :-) I combine this with a static Grub installation (switching between the Grub installations for each Linux) so that the system can still boot from disk when something went wrong with updating the boot loader. If you happen to understand German: hauke-laging.de/software/static-grub – Hauke Laging Apr 30 '13 at 13:46

This is because ext3 on older revisions of EL5 is susceptible to filesystem corruption. Update your kernel/OS at at least EL5.8 to get past the issue.

I didn't notice this until I worked in a hosting environment with a large number of unpatched RHEL5 servers. We had weekly occurrences of this.

When a Guest/VM complains about storage or network issues, always look to the Host/Hypervisor for clues. In this case as you mention there were some bugs in vmware. This could also be caused by problems with the host server or storage.
Make sure your vmware host and vmware guest drivers are all up to date. http://www.experts-exchange.com/Software/VMWare/Q_27806448.html

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.