Question marks showing in ls of directory. IO errors too

Question

Has anyone seen this before? I've got a raid 5 mounted on my server and for whatever reason it started showing this:

jason@box2:/mnt/raid1/cra$ ls -alh
ls: cannot access e6eacc985fea729b2d5bc74078632738: Input/output error
ls: cannot access 257ad35ee0b12a714530c30dccf9210f: Input/output error
total 0
drwxr-xr-x 5 root root 123 2009-08-19 16:33 .
drwxr-xr-x 3 root root  16 2009-08-14 17:15 ..
?????????? ? ?    ?      ?                ? 257ad35ee0b12a714530c30dccf9210f
drwxr-xr-x 3 root root  57 2009-08-19 16:58 9c89a78e93ae6738e01136db9153361b
?????????? ? ?    ?      ?                ? e6eacc985fea729b2d5bc74078632738

The md5 strings are actual directory names and not part of the error. The question marks are odd, and any directory with a question mark throws an io error when you attempt to use/delete/etc it.

I was unable to umount the drive due to "busy". Rebooting the server "fixed" it but it was throwing some raid errors on shutdown. I have configured two raid 5 arrays and both started doing this on random files. Both are using the following config:

mkfs.xfs -l size=128m -d agcount=32
mount -t xfs -o noatime,logbufs=8

Nothing too fancy, but part of an optimized config for this box. We're not partitioning the drives and that was suggested as a possible issue. Could this be the culprit?

Answer 1

Take a backup as soon as humanly possible, if only so that if you mess it up further while trying to repair any potential damage you can go back to the original less-broken state. After backing up, you might run fsck to see if it thinks there's any problems.

Answer 2

I had a similar problem because my directory had read (r) but not execute (x) rights. My directory listing showed:

myname@srv:/home$ ls -l service/mail/ ls: cannot access service/mail/001_SERVICE INBOX: Permission denied total 0 -????????? ? ? ? ? ? 001_SERVICE INBOX d????????? ? ? ? ? ? 01_CURRENT SERVICE

The mail directory had the r bit set, but not the x that you need for listing or search and access. Doing 'sudo chmod -R g+x mail' solved this problem.

Answer 3

The question marks in the ls output just indicate that it could not stat() the directory entry. You can also see those if you ls a directory for which you have r(ead) but not x (search) permission. However in that case it would not report I/O error.

In your case it looks like there is a disk error or possibly filesystem corruption. /var/log/messages or dmesg is likely to reveal further details.

Answer 4

May be filenames just contain undisplayable characters. Try to check file names with emacs DirEd:

http://www.cs.utah.edu/dept/old/texinfo/emacs19/emacs%5F32.html

Answer 5

We had a server with a corrupted filesystem (reiserfs) and it generated directory entries with question marks for all the attributes except the filename. In our case, the filenames were unaffected.

Also, the free space was being reported incorrectly. Using du -sh /* we could only account for about 30G, but the drive was being reported as over 200G in use.

Rebooting the server with shutdown -rF now to force a file-system check did not work. I had to reboot into single-user mode and run:

fsck.reiserfs --rebuild-tree /dev/sda3

This almost worked. It got through a few passes, then locked up. Had to reinstall the OS.

Maintain your backups!