Take the 2-minute tour ×
Server Fault is a question and answer site for professional system and network administrators. It's 100% free, no registration required.

There is a particular directory (/var/www), that when I run ls (with or without some options), the command hangs and never completes. There is only about 10-15 files and directories in /var/www. Mostly just text files. Here is some investigative info:

[me@server www]$ df .
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_dev-lv_root
                       50G   19G   29G  40% /

[me@server www]$ df -i .
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/vg_dev-lv_root
                        3.2M    435K    2.8M   14% /

find works fine. Also I can type in cd /var/www/ and press TAB before pressing enter and it will successfully tab-completion list of all files/directories in there:

[me@server www]$ cd /var/www/
cgi-bin/         create_vhost.sh  html/            manual/          phpMyAdmin/      scripts/         usage/
conf/            error/           icons/           mediawiki/       rackspace        sqlbuddy/        vhosts/
[me@server www]$ cd /var/www/

I have had to kill my terminal sessions several times because of the ls hanging:

[me@server ~]$ ps | grep ls
gdm       6215  0.0  0.0 488152  2488 ?        S<sl Jan18   0:00 /usr/bin/pulseaudio --start --log-target=syslog
root     23269  0.0  0.0 117724  1088 ?        D    18:24   0:00 ls -Fh --color=always -l
root     23477  0.0  0.0 117724  1088 ?        D    18:34   0:00 ls -Fh --color=always -l
root     23579  0.0  0.0 115592   820 ?        D    18:36   0:00 ls -Fh --color=always
root     23634  0.0  0.0 115592   816 ?        D    18:38   0:00 ls -Fh --color=always
root     23740  0.0  0.0 117724  1088 ?        D    18:40   0:00 ls -Fh --color=always -l
me       23770  0.0  0.0 103156   816 pts/6    S+   18:41   0:00 grep ls

kill doesn't seem to have any affect on the processes, even as sudo.

What else should I do to investigate this problem? It just randomly started happening today.

UPDATE

dmesg is a big list of things, mostly related to an external USB HDD that I've mounted too many times and the max mount count has been reached, but that is an un-related problem I think. Near the bottom of dmesg I'm seeing this:

INFO: task ls:23579 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ls            D ffff88041fc230c0     0 23579  23505 0x00000080
 ffff8801688a1bb8 0000000000000086 0000000000000000 ffffffff8119d279
 ffff880406d0ea20 ffff88007e2c2268 ffff880071fe80c8 00000003ae82967a
 ffff880407169ad8 ffff8801688a1fd8 0000000000010518 ffff880407169ad8
Call Trace:
 [<ffffffff8119d279>] ? __find_get_block+0xa9/0x200
 [<ffffffff814c97ae>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff814c964b>] mutex_lock+0x2b/0x50
 [<ffffffff8117a4d3>] do_lookup+0xd3/0x220
 [<ffffffff8117b145>] __link_path_walk+0x6f5/0x1040
 [<ffffffff8117a47d>] ? do_lookup+0x7d/0x220
 [<ffffffff8117bd1a>] path_walk+0x6a/0xe0
 [<ffffffff8117beeb>] do_path_lookup+0x5b/0xa0
 [<ffffffff8117cb57>] user_path_at+0x57/0xa0
 [<ffffffff81178986>] ? generic_readlink+0x76/0xc0
 [<ffffffff8117cb62>] ? user_path_at+0x62/0xa0
 [<ffffffff81171d3c>] vfs_fstatat+0x3c/0x80
 [<ffffffff81258ae5>] ? _atomic_dec_and_lock+0x55/0x80
 [<ffffffff81171eab>] vfs_stat+0x1b/0x20
 [<ffffffff81171ed4>] sys_newstat+0x24/0x50
 [<ffffffff810d40a2>] ? audit_syscall_entry+0x272/0x2a0
 [<ffffffff81013172>] system_call_fastpath+0x16/0x1b

And also, strace ls /var/www/ spits out a whole BUNCH of information. I don't know what is useful here... The last handful of lines:

ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, TIOCGWINSZ, {ws_row=68, ws_col=145, ws_xpixel=0, ws_ypixel=0}) = 0
stat("/var/www/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/var/www/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
fcntl(3, F_GETFD)                       = 0x1 (flags FD_CLOEXEC)
getdents(3, /* 16 entries */, 32768)    = 488
getdents(3, /* 0 entries */, 32768)     = 0
close(3)                                = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 9), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3093b18000
write(1, "cgi-bin  conf  create_vhost.sh\te"..., 125cgi-bin  conf  create_vhost.sh      error  html  icons  manual  mediawiki  phpMyAdmin  rackspace  scripts  sqlbuddy  usage   vhosts
) = 125
close(1)                                = 0
munmap(0x7f3093b18000, 4096)            = 0
close(2)                                = 0
exit_group(0)                           = ?
share|improve this question
    
found this question by the same symptoms. As it turned out, I had a remote filesystem mounted via sshfs with a hanged connection. –  modosansreves Feb 27 '13 at 11:01
    
So what do you do with sshfs? I have the same problem. –  maythesource.com Jun 19 at 9:43

3 Answers 3

up vote 11 down vote accepted

Run strace ls /var/www/ and see what it hangs on. It's certainly hung on I/O -- that's what the D state in your ps output means (and since kill doesn't help, it's one of the uninterruptible I/O syscalls). Most hangs involve an NFS server that's gone to god, but based on your df that isn't the case here. A quick check of dmesg for anything related to filesystems or disks might be worthwhile, just in case.

share|improve this answer
1  
NFS still might be the case. If ls is aliased to something that tries to dereference symlinks to find what theyre pointing at, it could be hanging if the symlink points to a dead NFS mount. –  Patrick Mar 8 '12 at 0:20
    
Gah, didn't notice it was a df . and not a full df. It could definitely be an NFS problem then. –  womble Mar 8 '12 at 6:40
    
There are no NFS mounts here. It's all the local single disk. It's a very simple linux server. One physical drive. –  Jakobud Mar 8 '12 at 15:46
    
strace ls /var/www/ prints out a bunch of stuff. What do I look for? The last line is exit_group(0) = ?. –  Jakobud Mar 8 '12 at 15:47
    
See updated question. –  Jakobud Mar 8 '12 at 16:03

I had a problem with the same symptoms. It turned out that I had a symlink in that directory to an SMB mount over GVFS.

lrwxrwxrwx  1 alex alex        45 Sep 16  2011 foo -> /home/alex/.gvfs/bar on foo/data/

Normally ls would complete instantly whether or not the share was mounted. But in this case I had suspended and resumed the machine, and the mount was performing poorly in general. Remounting the share fixed the problem.

share|improve this answer

Womble's suggestions are excellent, and you should try those first, but if they don't fix it I have had this problem when a filesystem has become self-inconsistent (through flaky hardware, obscure kernel bugs, or even cosmic rays).

If you think it might be that, you can force a fsck on reboot by doing touch /forcefsck; reboot. Watch what it says at boot time, to see if the fsck picks up any inconsistencies.

Warning: this will fsck all the filesystems attached to the machine; do not do it if you also have a multi-petabyte disc array attached, it may take days. fscking filesystems can also lead to data loss; if you really do have inconsistencies in your file system, e2fsck will change it from one that looks right but doesn't quite work, to one that works right but may not contain everything you expect.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.