ls returns nothing only in certain directories

Question

I have a raid drive mounted here:

/data/

And certain directories like this one:

/data/somedir/somesubdir/

when I run

ls

w/ or w/o any flags, terminal doesn't return anything. It does not return an empty directory listing. It simply goes to the next line and sits there blank with no prompt coming up. I cannot CTRL-C out of it. I have to close this terminal instance and start over.

At first I thought it was something to do with the ls command, but its pointing to /bin/ls and I can ls other directories just fine.

Also, running this

find /data/somedir/somesubdir

immediately finds all the files just as expected.

UPDATE:

dmesg doesn't appear to be spitting out any errors that I see. My raid is at /dev/sda5 so if I do

dmesg | grep sda

I get

sd 0:0:0:0: [sda] 1953083392 512-byte hardware sectors (999979 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 23 00 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: disabled, doesn't support DPO or FUA
sd 0:0:0:0: [sda] 1953083392 512-byte hardware sectors (999979 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 23 00 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: disabled, doesn't support DPO or FUA
 sda: sda1 sda2 sda3 sda4 <<7>libata version 2.21 loaded.
 sda5 >
sd 0:0:0:0: [sda] Attached SCSI disk
EXT3 FS on sda3, internal journal
EXT3 FS on sda5, internal journal
EXT3 FS on sda2, internal journal
EXT3 FS on sda1, internal journal

Nothing out of the ordinary.

Also, my 3ware raid controller shows all disks as OK status and doesn't show any reports of any issues with them.

There doesn't appear to be any server compromise.

Also:

ps auxww | grep somesubdir

returns only the grep process for that directory.

UPDATE #2:

I did try leaving the command for upwards of an hour and no result. I was at a loss for what to do next so I went ahead and restarted the server. The problem no longer persists although it does take 10 sec or so to get a directory listing in that subdirectory. I have no idea what the problem was before.

Answer 1

Apart from the suggestions above, I've also seen this problem when there is a very large number of files in the directory you're listing, which amy be causing the file system to run out of inodes. If this is a possibility, try just leaving the ls command to see if it completes after a while.

Answer 2

Your disks are probably in trouble. You could try this:

Terminal 1$ ls /data/somedir/somesubdir/
 (hang)
Terminal 2$ ps auxww | grep somesubdir

If the status of your ls is D, that means it's waiting for the disk to do something. Look in dmesg for status messages, and look into SMART if you're not already using it to see how healthy those disks are.

Why does find work ok? Because it doesn't do a stat() on each file. That indicates that the directory is ok, but one or more of the files is on a damaged section of disk (or something similar).

Answer 3

ls without any flags will return nothing if the directory is empty. You need at least ls -a if you want to see anything in the event of the directory being empty.

If the directory isn't empty, it's possible you've got a corrupted filesystem - run fsck to check. If the filesystem is okay and the directory isn't empty, next up is to question ls - perhaps your copy of ls has been replaced in a very primitive form of rootkit?

Answer 4

Try to figure out where it is hanging by running

strace ls $DIR

Answer 5

Maybe there's a file with a name, which contains a sequence of bytes which block your terminal. You can sometimes unblock a blocked terminal sending Ctrl-Q keyboard shortcut (Ctrl-S blocks it).

Try to do /bin/ls -N | less -S and look if there's any file name with suspicious or control characters. Less would show this characters as hex-codes in angle brackets, for example <BF>.

Existence of a file with characters with a code lower than 32 (from <01> to <1F>) may be a sign of system compromise. Or a memory management bug in some program which create files there.