Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

Ubuntu 12.04

File system goes to readonly mode frequently. First of all I have read this question file system is going into read only mode frequently already. But I have to know if it's not caused by something else than dying hard drive. This is server provided by my client and I am just runing there some node.js workers + one node.js server and I am using mongodb.

From time to time (every 20-50h) system suddenly makes filesystem read only, mongodb process fails (due read-only fs) and my node workers/server (which are started by forever) are just killed.

Here is the log from dmesg - I can see there some errors and messages that FS is going to read-only, and there is also some JOURNAL error but I would like to find cause of those errors..

http://speedy.sh/Ux2VV/dmesg.log.txt


edit

smartctl -t long /dev/sda
smartctl 5.41 2011-06-09 r3365 [x86_64-linux-3.5.0-23-generic] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

SMART support is: Unavailable - device lacks SMART capability.
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.

What I am doing wrong? Same is for sda2.

Morover now when I type any command that not exists in shell I get this:

Sorry, command-not-found has crashed! Please file a bug report at:
https://bugs.launchpad.net/command-not-found/+filebug
Please include the following information with the report:

edit2

I just got info that this server is actually VPS and they told me that hard drives are OK and they are on RAID 10. And they told me that "forcing fsck in fstab should help"...


edit3

here is output from mount command:

/dev/sda2 on / type ext4 (rw,errors=remount-ro)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
none on /media/psf type prl_fs (rw,nosuid,nodev,sync,noatime,share,_netdev)

So there is no actually sda drive? Only sda2?


edit4

Output from fsck -N command:

root@ubuntu:~# fsck -N sda
fsck from util-linux 2.20.1
[/sbin/fsck.ext4 (1) -- /] fsck.ext4 sda /dev/sda2 
share|improve this question

2 Answers 2

[26729.124569] Write(10): 2a 00 03 96 5a b0 00 00 08 00
[26729.124576] end_request: I/O error, dev sda, sector 60185264
[26729.125298] Buffer I/O error on device sda2, logical block 4593494
[26729.125986] lost page write due to I/O error on sda2

For me, that's pretty strong evidence that your /dev/sda is on its way out. You could run a smartctl test on it for confirmation (smartctl -t long /dev/sda), but I'd be inclined to replace it as soon as possible.

Edit: the smartctl command I gave is correct as written. Thanks for showing the failure mode in your question; this looks like either you have very old hardware, or there's some kind of translation layer in the way: either virtualisation, or a hardware RAID controller. Can you clarify?

May I repeat my assertion that your HDD is on its way out? Testing's all very well, but getting the hardware replaced before your system packs up and your data are lost should be your priority now. Please, at the very least make sure that your backups are completely up-to-date before wasting any more time on smartctl.

Edit 2: it's certainly worth trying what they've suggested - fscking the file system - but I have little hope that that will fix the problem because your FS isn't dropping to ro mode because of FS inconsistencies, it's dropping to ro mode because of problems talking to the underlying hardware.

If they have confidence that the underlying hardware is fine, then it's an issue between the kernel and the hardware, ie, the virtualisation layer. You should probably get your VPS provider to confirm that the distro, and the exact kernel version, that you're running are fully supported on their VPS system.

share|improve this answer
    
I edited question - I tried this command with sda/sda2 but it does not work. Anyway did you mean sda or sda2 according to [26729.124576] end_request: I/O error, dev sda, sector 60185264 [26729.125298] Buffer I/O error on device sda2, logical block 4593494 –  user606521 Oct 22 '13 at 8:29
    
Thanks, the server is just for this project and it does not have any valuable data stored on it yet. I have only ssh access to it - I will ask client for details. –  user606521 Oct 22 '13 at 8:37
    
Understood. Good luck! –  MadHatter Oct 22 '13 at 8:38
    
I updated question - I got info from client that this is actually VPS and hard drives are on RAID, and they are OK. –  user606521 Oct 22 '13 at 11:30

More perfect way to find the exact error may be during the read only period and run the command dmesg for any bugs/issues. You may also try running the fsck in dry mode to figure out what is the issue. (sorry due to access restriction I am unable to view your attachment. If its during the issue period, I will check it later)

share|improve this answer
    
I used dmesg command when file system was in read-only mode. Now I just rebooted server and for now it works. What do you mean by fsck in dry mode? I have never used this command... –  user606521 Oct 22 '13 at 11:46
    
` fsck -N <partition> ` Don't execute, just show what would be done. –  rootslash Oct 22 '13 at 11:58
    
I edited question and added output from fsck -N sda –  user606521 Oct 22 '13 at 12:06

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.