69

Has anyone seen this before? I've got a raid 5 mounted on my server and for whatever reason it started showing this:

jason@box2:/mnt/raid1/cra$ ls -alh
ls: cannot access e6eacc985fea729b2d5bc74078632738: Input/output error
ls: cannot access 257ad35ee0b12a714530c30dccf9210f: Input/output error
total 0
drwxr-xr-x 5 root root 123 2009-08-19 16:33 .
drwxr-xr-x 3 root root  16 2009-08-14 17:15 ..
?????????? ? ?    ?      ?                ? 257ad35ee0b12a714530c30dccf9210f
drwxr-xr-x 3 root root  57 2009-08-19 16:58 9c89a78e93ae6738e01136db9153361b
?????????? ? ?    ?      ?                ? e6eacc985fea729b2d5bc74078632738

The md5 strings are actual directory names and not part of the error. The question marks are odd, and any directory with a question mark throws an io error when you attempt to use/delete/etc it.

I was unable to umount the drive due to "busy". Rebooting the server "fixed" it but it was throwing some raid errors on shutdown. I have configured two raid 5 arrays and both started doing this on random files. Both are using the following config:

mkfs.xfs -l size=128m -d agcount=32
mount -t xfs -o noatime,logbufs=8

Nothing too fancy, but part of an optimized config for this box. We're not partitioning the drives and that was suggested as a possible issue. Could this be the culprit?

1
  • I have seen these question marks when autofs had issues mounting a directory. Shutting down autofs allowed me to delete the directory and try mounting manually which pointed to a permissions error on the remote server.
    – Pace
    Apr 25, 2013 at 1:34

10 Answers 10

65

I had a similar problem because my directory had read (r) but not execute (x) rights. My directory listing showed:

myname@srv:/home$ ls -l service/mail/
ls: cannot access service/mail/001_SERVICE INBOX: Permission denied
total 0
-????????? ? ? ? ?                ? 001_SERVICE INBOX
d????????? ? ? ? ?                ? 01_CURRENT SERVICE

The mail directory had the r bit set, but not the x that you need for listing or search and access. Doing sudo chmod -R g+x mail solved this problem.

2
  • 5
    'sudo chmod -R g+x mail' and then chmod +x mail solved my problem, thanks!
    – glebm
    Oct 10, 2011 at 14:41
  • 8
    what if your ls returns no such file or directory with a bunch of ????? in the privileges/rights area? Apr 15, 2014 at 16:39
15

The answers mentioning the read, but not execute or stat() are correct. But there is a common cause of this (other than corruption) that's bitten me a few times and would match your question with the IO errors nicely. If you improperly mount a filesystem, the mount point of that filesystem may show up with question marks. If you're seeing these where you've just tried mounting a new filesystem, try the following before worrying about corruption and fsck.

$ sudo umount /mnt/raid1/cra/257ad35ee0b12a714530c30dccf9210f
$ ls -alh /mnt/raid1/cra

You should see the 257ad35ee0b12a714530c30dccf9210f folder with permissions and attributes rather than question marks. If so, search for other options for your mount command or /etc/fstab file. If not, perhaps it is time to read the other answers, backup what you can, and run a fsck.

2
12

The question marks in the ls output just indicate that it could not stat() the directory entry. You can also see those if you ls a directory for which you have r(ead) but not x (search) permission. However in that case it would not report I/O error.

In your case it looks like there is a disk error or possibly filesystem corruption. /var/log/messages or dmesg is likely to reveal further details.

6

Take a backup as soon as humanly possible, if only so that if you mess it up further while trying to repair any potential damage you can go back to the original less-broken state. After backing up, you might run fsck to see if it thinks there's any problems.

1
  • in my case this already is the backup.
    – Paolo
    Dec 12, 2014 at 18:09
1

May be filenames just contain undisplayable characters. Try to check file names with emacs DirEd:

http://www.cs.utah.edu/dept/old/texinfo/emacs19/emacs_32.html

1

We had a server with a corrupted filesystem (reiserfs) and it generated directory entries with question marks for all the attributes except the filename. In our case, the filenames were unaffected.

Also, the free space was being reported incorrectly. Using du -sh /* we could only account for about 30G, but the drive was being reported as over 200G in use.

Rebooting the server with shutdown -rF now to force a file-system check did not work. I had to reboot into single-user mode and run:

fsck.reiserfs --rebuild-tree /dev/sda3

This almost worked. It got through a few passes, then locked up. Had to reinstall the OS.

Maintain your backups!

1

I've also seen this when running autofs but autofs can't mount the directory. So then to figure out why it couldn't mount the directory I disabled autofs and tried to mount the directory manually (this also allowed me to delete the directory). I tried mounting the directory manually and found that there was a permissions error. After fixing that, the directory returned to normal again.

0
1

Watch out for other running processes on the server for example rsync

[root@server upload]# ls -la
ls: cannot access .3bfb3dc5-cb55-435f-8e23-2afcab2c6873_image4993891600240007749.jpg.bV6VTV: No such file or directory
total 194496
drwxr-x--- 2 gx apache    1382 Jan 11 10:36 .
drwxr-x--- 3 gx apache       3 Jan 11 10:29 ..
-rw-r--r-- 1 gx apache   94850 Dec 10  2015 37d355b9-210d-45df-8061-968ea5cb9f31_mob.jpg
...
-rw-r--r-- 1 gx apache   10864 Jul 24  2015 3bfb23bf-8ff5-4603-aa57-9b23ca498e2c_internet.png
-rw-r--r-- 1 gx apache   10864 Jul 24  2015 .3bfb23bf-8ff5-4603-aa57-9b23ca498e2c_internet.png.nHmIPk
-????????? ? ?  ?            ?            ? .3bfb3dc5-cb65-435f-8e23-2agcab2c6873_image4993891600240007749.jpg.bV6VTV

It generates temporary files that get created and dropped fast which will cause errors if you try to call other simple file management commands like rm, mv etc.

0

Just to give a different perspective - I had this when I was programatically generating directories from a list directories in a file (in ruby).

Of course the line from the file came through as a string with a \n on the end - which looked good and seemed to work. However as I started creating directories rather than being chomped it ended up creating two of each directory: /whatiwanted and /whatiwanted?.

0

I sometimes see this as a transient error when an NFS server is heavily overloaded.

The OP asked about RAID but several answers mention NFS, and in fact that was the search that brought me here.

You must log in to answer this question.