Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

CentOS 5.5

I just came across an instance where my /storage directory was supposedly a read-only file system. I get the error:

mkdir: cannot create directory `test': Read-only file system

df:

Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                      141G   21G  114G  16% /
/dev/sda1              99M   13M   82M  14% /boot
tmpfs                1001M     0 1001M   0% /dev/shm
/dev/sdb1             1.4T  307G  999G  24% /storage
/dev/sdd1             1.4T  329G  978G  26% /media/hddblack
/dev/sdc1             1.4T  329G  978G  26% /media/hddsilver

mount:

/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sdb1 on /storage type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/sdd1 on /media/hddblack type ext3 (ro)
/dev/sdc1 on /media/hddsilver type ext3 (ro)

/storage is clearly a read-write file system. Next I tried this:

mount -o remount,rw /dev/disk/by-label/storage /storage

And now, the filesystem is writable again. Hooray. Except, why did it do this? Here is the mount now:

/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/sda1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sdb1 on /storage type ext3 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/sdd1 on /media/hddblack type ext3 (ro)
/dev/sdc1 on /media/hddsilver type ext3 (ro)

Pretty much the same thing... any ideas what would cause this or what logs I should look into to find what the problem was?

share|improve this question
add comment

2 Answers 2

up vote 7 down vote accepted

Most likely you had a disk IO error. Check /var/log/messages. Look for messages about aborting the journal and re-mounting read-only.

P.S.: If that is the case you best have backups of your data.

share|improve this answer
    
In which case, you will want to look into fsck. –  Andrew M. Nov 24 '10 at 4:21
    
Bewarned: if you don't have backups try to salvage as much data as possible before the fsck. Fsck on a drive with I/O errors has a good chance of totally hosing the filesystem. –  Mark Wagner Nov 24 '10 at 18:28
add comment

The problem with your diagnostic steps: if your root filesystem is read-only, /etc/mtab is also read-only. Guess what mount reads to get filesystem status :)

The solution is to check /proc/mounts, which is a dynamic kernel file that shows the true filesystem state. If you'd have done so at the time you had this problem, you'd very likely have found your root filesystem was remounted read-only. Know your utilities and where they get their data from.

How's that happen? Most /etc/fstab files will have a line similar to:

/dev/sda1 /               ext3    relatime,errors=remount-ro 0

Note the "errors=remount-ro" option. In the event of any filesystem errors, the root filesystem is remounted in read-only mode.

Assuming /var/logs was writeable, or you had offline syslogging enabled, you should find the event logged as well.

Disk errors capable of triggering a remount-ro are generally a Bad Thing. Hope you swapped your drive.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.