43

There is a particular directory (/var/www), that when I run ls (with or without some options), the command hangs and never completes. There is only about 10-15 files and directories in /var/www. Mostly just text files. Here is some investigative info:

[me@server www]$ df .
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_dev-lv_root
                       50G   19G   29G  40% /

[me@server www]$ df -i .
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/mapper/vg_dev-lv_root
                        3.2M    435K    2.8M   14% /

find works fine. Also I can type in cd /var/www/ and press TAB before pressing enter and it will successfully tab-completion list of all files/directories in there:

[me@server www]$ cd /var/www/
cgi-bin/         create_vhost.sh  html/            manual/          phpMyAdmin/      scripts/         usage/
conf/            error/           icons/           mediawiki/       rackspace        sqlbuddy/        vhosts/
[me@server www]$ cd /var/www/

I have had to kill my terminal sessions several times because of the ls hanging:

[me@server ~]$ ps | grep ls
gdm       6215  0.0  0.0 488152  2488 ?        S<sl Jan18   0:00 /usr/bin/pulseaudio --start --log-target=syslog
root     23269  0.0  0.0 117724  1088 ?        D    18:24   0:00 ls -Fh --color=always -l
root     23477  0.0  0.0 117724  1088 ?        D    18:34   0:00 ls -Fh --color=always -l
root     23579  0.0  0.0 115592   820 ?        D    18:36   0:00 ls -Fh --color=always
root     23634  0.0  0.0 115592   816 ?        D    18:38   0:00 ls -Fh --color=always
root     23740  0.0  0.0 117724  1088 ?        D    18:40   0:00 ls -Fh --color=always -l
me       23770  0.0  0.0 103156   816 pts/6    S+   18:41   0:00 grep ls

kill doesn't seem to have any affect on the processes, even as sudo.

What else should I do to investigate this problem? It just randomly started happening today.

UPDATE

dmesg is a big list of things, mostly related to an external USB HDD that I've mounted too many times and the max mount count has been reached, but that is an un-related problem I think. Near the bottom of dmesg I'm seeing this:

INFO: task ls:23579 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
ls            D ffff88041fc230c0     0 23579  23505 0x00000080
 ffff8801688a1bb8 0000000000000086 0000000000000000 ffffffff8119d279
 ffff880406d0ea20 ffff88007e2c2268 ffff880071fe80c8 00000003ae82967a
 ffff880407169ad8 ffff8801688a1fd8 0000000000010518 ffff880407169ad8
Call Trace:
 [<ffffffff8119d279>] ? __find_get_block+0xa9/0x200
 [<ffffffff814c97ae>] __mutex_lock_slowpath+0x13e/0x180
 [<ffffffff814c964b>] mutex_lock+0x2b/0x50
 [<ffffffff8117a4d3>] do_lookup+0xd3/0x220
 [<ffffffff8117b145>] __link_path_walk+0x6f5/0x1040
 [<ffffffff8117a47d>] ? do_lookup+0x7d/0x220
 [<ffffffff8117bd1a>] path_walk+0x6a/0xe0
 [<ffffffff8117beeb>] do_path_lookup+0x5b/0xa0
 [<ffffffff8117cb57>] user_path_at+0x57/0xa0
 [<ffffffff81178986>] ? generic_readlink+0x76/0xc0
 [<ffffffff8117cb62>] ? user_path_at+0x62/0xa0
 [<ffffffff81171d3c>] vfs_fstatat+0x3c/0x80
 [<ffffffff81258ae5>] ? _atomic_dec_and_lock+0x55/0x80
 [<ffffffff81171eab>] vfs_stat+0x1b/0x20
 [<ffffffff81171ed4>] sys_newstat+0x24/0x50
 [<ffffffff810d40a2>] ? audit_syscall_entry+0x272/0x2a0
 [<ffffffff81013172>] system_call_fastpath+0x16/0x1b

And also, strace ls /var/www/ spits out a whole BUNCH of information. I don't know what is useful here... The last handful of lines:

ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, TIOCGWINSZ, {ws_row=68, ws_col=145, ws_xpixel=0, ws_ypixel=0}) = 0
stat("/var/www/", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/var/www/", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
fcntl(3, F_GETFD)                       = 0x1 (flags FD_CLOEXEC)
getdents(3, /* 16 entries */, 32768)    = 488
getdents(3, /* 0 entries */, 32768)     = 0
close(3)                                = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 9), ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3093b18000
write(1, "cgi-bin  conf  create_vhost.sh\te"..., 125cgi-bin  conf  create_vhost.sh      error  html  icons  manual  mediawiki  phpMyAdmin  rackspace  scripts  sqlbuddy  usage   vhosts
) = 125
close(1)                                = 0
munmap(0x7f3093b18000, 4096)            = 0
close(2)                                = 0
exit_group(0)                           = ?
4
  • 2
    found this question by the same symptoms. As it turned out, I had a remote filesystem mounted via sshfs with a hanged connection. Feb 27, 2013 at 11:01
  • 2
    So what do you do with sshfs? I have the same problem.
    – Menelaos
    Jun 19, 2014 at 9:43
  • 2
    ls hung for me on getdents() for a certain directory. The issue resolved itself after I unmounted, ran xfs_check, ran xfs_repair, and remounted even though no issues were found.
    – Leo
    Feb 26, 2015 at 18:56
  • I had to use 'kill -9' to clean up stuck ls runs.
    – flickerfly
    Jul 6, 2015 at 16:31

8 Answers 8

29

Run strace ls /var/www/ and see what it hangs on. It's certainly hung on I/O -- that's what the D state in your ps output means (and since kill doesn't help, it's one of the uninterruptible I/O syscalls). Most hangs involve an NFS server that's gone to god, but based on your df that isn't the case here. A quick check of dmesg for anything related to filesystems or disks might be worthwhile, just in case.

8
  • 2
    NFS still might be the case. If ls is aliased to something that tries to dereference symlinks to find what theyre pointing at, it could be hanging if the symlink points to a dead NFS mount.
    – phemmer
    Mar 8, 2012 at 0:20
  • Gah, didn't notice it was a df . and not a full df. It could definitely be an NFS problem then.
    – womble
    Mar 8, 2012 at 6:40
  • There are no NFS mounts here. It's all the local single disk. It's a very simple linux server. One physical drive. Mar 8, 2012 at 15:46
  • strace ls /var/www/ prints out a bunch of stuff. What do I look for? The last line is exit_group(0) = ?. Mar 8, 2012 at 15:47
  • 3
    @Jakobud Try strace -vf ls -l /var/www to see if it stops at a specific file or dir.
    – ott--
    Mar 8, 2012 at 18:03
4

On the hope this will be helpful, I had the above symptoms being caused by using docker and docker compose with the AUFS driver in Ubuntu 14.04. ls <dir> was hanging, and strace ls <dir> showed it was hanging on the getdents call. Stopping all running containers allowed me to begin using the drive as expected.

4
  • I have the same problem. Doing a docker system prune -a had no effect. I literally started from scratch and it still broke. I removed the affected folder from my host machine and created it again on the host machine and from the container by running: mkdir <folder_name> and it still broke both times. Running an strace next.
    – Biketire
    Aug 26, 2021 at 9:28
  • @Biketire were you ever able to figure out a solution to this? Am getting the exact issue with docker on mac. Feb 8 at 5:22
  • @SaroshKhatana no. I think what I did was switching from the Docker app to docker-machine and that worked out okay. It's a hassle to setup though.
    – Biketire
    Feb 9 at 15:43
  • I managed to solve my issue. I'm on mac and apparently docker's mounted volumes on mac do not work well with underscores in folder names. my folder was functional_tests and that was breaking it. Changed to functionaltests and works fine now. Feb 10 at 13:08
3

I had a problem with the same symptoms. It turned out that I had a symlink in that directory to an SMB mount over GVFS.

lrwxrwxrwx  1 alex alex        45 Sep 16  2011 foo -> /home/alex/.gvfs/bar on foo/data/

Normally ls would complete instantly whether or not the share was mounted. But in this case I had suspended and resumed the machine, and the mount was performing poorly in general. Remounting the share fixed the problem.

2

I was experiencing the same problem.

Entering a directory is fine, listing it hangs, find works, tab complete hangs, and some folders beneath do work. Very head-scratchingly-weird.

Reading this thread on Server Fault did lead me on a logic path towards the solution.

It being to do with NAS, and NAS commonly being put as `automount' made me realise that I had recently changed my fstab to 'automount' some usb drives if they were present but carry on as normal when they weren't.

I then proceeded as follows:

  1. Unmount the partition containing the delinquent directory.
  2. Edit fstab and convert all automount to either commented out or without auto.
  3. Reload SystemD if you have it: systemctl --system daemon-reload
  4. mount -a

Try entering the directory again and get that warm fuzzy feeling of having fixed the issue.

1

Womble's suggestions are excellent, and you should try those first, but if they don't fix it I have had this problem when a filesystem has become self-inconsistent (through flaky hardware, obscure kernel bugs, or even cosmic rays).

If you think it might be that, you can force a fsck on reboot by doing touch /forcefsck; reboot. Watch what it says at boot time, to see if the fsck picks up any inconsistencies.

Warning: this will fsck all the filesystems attached to the machine; do not do it if you also have a multi-petabyte disc array attached, it may take days. fscking filesystems can also lead to data loss; if you really do have inconsistencies in your file system, e2fsck will change it from one that looks right but doesn't quite work, to one that works right but may not contain everything you expect.

1

I had the same exact symptoms that you described. To fix the problem all I had to do was fix the DNS server addresses. We had moved the NAS to a new network, which required updating the DNS server addresses. The addresses were statically assigned, but in the QNAP web interface I updated it to automatically assign.

1
  • Do you have any explanation why a wrong DNS entry would cause the problem?
    – RalfFriedl
    Apr 9, 2019 at 16:26
1

This happened to me. The cause ultimately was due to an sshfs mount point in the directory where the SSH server had become unreachable. strace did not give me any clue that ls was hanging on that entry (or maybe I don't understand how to read strace output).

I managed identify the cause by:

  • Noticing that my usual ls alias hanged but not running /bin/ls directly.
  • I then was to iteratively bisect the directory listing with different glob expressions to narrow down which entry was problematic. For example:
    $ ls -d [a-lA-L]* # OK
    $ ls -d [m-zM-Z]* # Hangs
    $ ls -d [m-sM-S]* # Hangs
    ...
    

Curiously, in my case, /bin/ls -F worked but /bin/ls --color did not. (I don't understand why, but that probably deserves its own question.)

1
-2

Running strace ls /var/www/ will give you hind of what is wrong. I had similar issue for / dir and using strace I was able to locate it was a NAS mount which caused it. Unmounting that NAS fixed the issue.

1
  • 4
    -1 : That is just a repeat of the already accepted answer.
    – HBruijn
    Jul 21, 2015 at 18:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.