I'm new on this but I'm stuck. I have a small virtual machine running nagios. Today it started to fail and I saw some errors on the messages log.
@someone~]# tail /var/log/messages Aug 31 02:25:57 muspllx043v kernel: sd 2:0:1:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK Aug 31 02:25:57 muspllx043v kernel: sd 2:0:1:0: [sdb] CDB: Write(10): 2a 00 05 fe ce d8 00 00 48 00 Aug 31 02:25:57 muspllx043v kernel: end_request: I/O error, dev sdb, sector 100585176 Aug 31 02:25:57 muspllx043v kernel: JBD2: Detected IO errors while flushing file data on dm-2-8 Aug 31 02:25:57 muspllx043v kernel: Aborting journal on device dm-2-8. Aug 31 02:25:57 muspllx043v kernel: EXT4-fs (dm-2): delayed block allocation failed for inode 1576714 at logical offset 303958 with max blocks 1 with error -30 Aug 31 02:25:57 muspllx043v kernel: Aug 31 02:25:57 muspllx043v kernel: This should not happen!! Data will be lost Aug 31 02:25:57 muspllx043v kernel: EXT4-fs error (device dm-2) in ext4_new_inode: Journal has aborted Aug 31 02:25:57 muspllx043v kernel: EXT4-fs error (device dm-2) in ext4_da_writepages: Journal has aborted
I tried to restart Nagios and getting the following error:
@someone~]# service nagios restart Running configuration check... mktemp: failed to create file via template
/tmp/.configtest.XXXXXXXX': Read-only file system /etc/init.d/nagios: line 88: : No such file or directory grep: : No such file or directory grep: : No such file or directory /etc/init.d/nagios: line 108: /usr/local/nagios/var/nagios.configtest: Read-only file system /etc/init.d/nagios: line 109: /usr/local/nagios/var/nagios.configtest: Read-only file system chmod: changing permissions of
/usr/local/nagios/var/nagios.configtest': Read-only file system chown: changing ownership of `/usr/local/nagios/var/nagios.configtest': Read-only file system cat: : No such file or directory
After some questions with my partners someone suggested me to check the filesystem, so I ran fsck command, but got the following error:
@someone~]# sudo fsck / fsck from util-linux-ng 2.17.2 e2fsck 1.41.12 (17-May-2010) /dev/mapper/vg_muspllx043v-lv_root: recovering journal fsck.ext4: Bad magic number in super-block while trying to re-open /dev/mapper/vg_muspllx043v-lv_root e2fsck: io manager magic bad!
Now I cannot excecute any command on the console and I'm getting Input/output error.
@someone~]# df -h -bash: /bin/df: Input/output error
If I try to open a new session I get: Connection closed.
Is it possible that fsck had messed the file system? I can navigate trhought the directories but I cannot do any action.
@someone scripts]# df -h -bash: /bin/df: Input/output error @someone scripts]# ls -l -bash: /bin/ls: Input/output error @someone scripts]# pwd /home/nagios/scripts @someone scripts]# cd @someone ~]# pwd /root @someone ~]# cd /var/log/ @someone log]# ls -lrt -bash: /bin/ls: Input/output error @someone log]# pwd /var/log @someone log]# @someone ~]#
I hope you can help me with this.