0

I'm new on this but I'm stuck. I have a small virtual machine running nagios. Today it started to fail and I saw some errors on the messages log.

@someone~]# tail /var/log/messages Aug 31 02:25:57 muspllx043v kernel: sd 2:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK Aug 31 02:25:57 muspllx043v kernel: sd 2:0:1:0: [sdb] CDB: Write(10): 2a 00 05 fe ce d8 00 00 48 00 Aug 31 02:25:57 muspllx043v kernel: end_request: I/O error, dev sdb, sector 100585176 Aug 31 02:25:57 muspllx043v kernel: JBD2: Detected IO errors while flushing file data on dm-2-8 Aug 31 02:25:57 muspllx043v kernel: Aborting journal on device dm-2-8. Aug 31 02:25:57 muspllx043v kernel: EXT4-fs (dm-2): delayed block allocation failed for inode 1576714 at logical offset 303958 with max blocks 1 with error -30 Aug 31 02:25:57 muspllx043v kernel: Aug 31 02:25:57 muspllx043v kernel: This should not happen!! Data will be lost Aug 31 02:25:57 muspllx043v kernel: EXT4-fs error (device dm-2) in ext4_new_inode: Journal has aborted Aug 31 02:25:57 muspllx043v kernel: EXT4-fs error (device dm-2) in ext4_da_writepages: Journal has aborted

I tried to restart Nagios and getting the following error:

@someone~]# service nagios restart Running configuration check... mktemp: failed to create file via template /tmp/.configtest.XXXXXXXX': Read-only file system /etc/init.d/nagios: line 88: : No such file or directory grep: : No such file or directory grep: : No such file or directory /etc/init.d/nagios: line 108: /usr/local/nagios/var/nagios.configtest: Read-only file system /etc/init.d/nagios: line 109: /usr/local/nagios/var/nagios.configtest: Read-only file system chmod: changing permissions of/usr/local/nagios/var/nagios.configtest': Read-only file system chown: changing ownership of `/usr/local/nagios/var/nagios.configtest': Read-only file system cat: : No such file or directory

After some questions with my partners someone suggested me to check the filesystem, so I ran fsck command, but got the following error:

@someone~]# sudo fsck / fsck from util-linux-ng 2.17.2 e2fsck 1.41.12 (17-May-2010) /dev/mapper/vg_muspllx043v-lv_root: recovering journal fsck.ext4: Bad magic number in super-block while trying to re-open /dev/mapper/vg_muspllx043v-lv_root e2fsck: io manager magic bad!

Now I cannot excecute any command on the console and I'm getting Input/output error.

@someone~]# df -h -bash: /bin/df: Input/output error

If I try to open a new session I get: Connection closed.

Is it possible that fsck had messed the file system? I can navigate trhought the directories but I cannot do any action.

@someone scripts]# df -h -bash: /bin/df: Input/output error @someone scripts]# ls -l -bash: /bin/ls: Input/output error @someone scripts]# pwd /home/nagios/scripts @someone scripts]# cd @someone ~]# pwd /root @someone ~]# cd /var/log/ @someone log]# ls -lrt -bash: /bin/ls: Input/output error @someone log]# pwd /var/log @someone log]# @someone ~]#

I hope you can help me with this.

Improve this question
2
  • First, fix the problem with the disk. Only after that, you can run fsck (if necessary).
    – Michael Hampton
    Aug 31, 2016 at 22:25
  • Hampton is very likely right that you have a problem with your underlying block device. This is a VM, so I don't know what your backing that disk with, but there's something wrong with that VM file, the filesystem that provides that file, or the block device that provides that filesystem. Start at the bottom of that chain at the disk / metal layer and work your way up with integrity checks.
    – Spooler
    Sep 1, 2016 at 0:19

1 Answer 1

Reset to default
1

First things first. fsck will not do anything on filesystems that are mounted read-write.

Second... it looks like the filesystem became corrupted... and as a result the OS remounted the disk as read-only. This is sort-of a failsafe to protect itself from further damage. (notice the number of "Read-only file system" errors?)

Third... fsck can make changes to the filesystem... but if there are problems it will ask you to confirm any changes that would potentially cause loss of data before they are applied... UNLESS you override the default behavior (a -y flag for instance) The "fixes" do not guarantee that your OS or data will be safe. It simply means that it will bring the volume into a proper ext2 format. This sometimes means files and directories will be deleted. Even if you didn't run fsck... it is likely that whatever data you might have been able to pull out of those sectors would have been unusable.

Fourth... it looks like a RAID was setup at some point? (I see volume-groups & logical volumes... ) It may not be a RAID... but I don't know why people would setup logical volumes without a RAID. Did you have multiple disks (virtual or physical) fail?

and the last step: recovery. It looks like the damage is done... and it is unlikely that digital forensics would get you very far (even if you paid a company a large chunk-of-change). Your best-bet is to create a new VM for nagios... and attempt to salvage and restore what you can from the old nagios install.

Final parting notes: I hope you learned your lesson. Backups are king. RAIDs fail... doubly-so when you don't monitor them. Keep OS and data on separate disks/volumes. (this makes things easier to backup/restore) Monitor your system.

Improve this answer
1
  • Thanks for your comments and explanations at the end, unfortunatelly I had to reinstall the VM and Nagios from zero. Yes it was a hard lesson.
    – Dan Sanchez
    Sep 5, 2016 at 17:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .