10

EDIT: I totally forgot about this thread. It turns out I had a bad hard disk. We had to redeploy this server for other needs so I finally got around to replacing the one bad disk and we're back in business.

For a few weeks now I couldn't figure out why I wasn't able to delete this one particular file. As root I can, but my shell script runs as a different user. So I go run ls -la and it's not there. However, if I call it as a parameter, it shows up! Sure enough, the owner is root, hence I'm not able to delete.

Notice, 6535 is missing ...

[root@server]# ls -la 653*
-rw-rw-r--  1 svn svn  24002 Mar 26 01:00 653
-rw-rw-r--  1 svn svn   7114 Mar 26 01:01 6530
-rw-rw-r--  1 svn svn   8653 Mar 26 01:01 6531
-rw-rw-r--  1 svn svn   6836 Mar 26 01:01 6532
-rw-rw-r--  1 svn svn   3308 Mar 26 01:01 6533
-rw-rw-r--  1 svn svn   3918 Mar 26 01:01 6534
-rw-rw-r--  1 svn svn   3237 Mar 26 01:01 6536
-rw-rw-r--  1 svn svn   3195 Mar 26 01:01 6537
-rw-rw-r--  1 svn svn  27725 Mar 26 01:01 6538
-rw-rw-r--  1 svn svn 263473 Mar 26 01:01 6539

Now it shows up if you call it directly.

[root@server]# ls -la 6535
-rw-rw-r--  1 root root 3486 Mar 26 01:01 6535

Here's something interesting. So I caught this issue because in my shell script, it would fail to delete because 6535 is owned by root. The file actually shows up after I run "rm -rf ." I tried it earlier and it failed to remove the directory since it told me the directory isn't empty. I went in and looked and sure enough, file "6535" finally shows up. No idea why it's doing this.

strace says the following

#strace ls -la 653* 2>&1 | grep ^open

open("/etc/ld.so.cache", O_RDONLY)      = 3
open("/lib64/tls/librt.so.1", O_RDONLY) = 3
open("/lib64/libacl.so.1", O_RDONLY)    = 3
open("/lib64/libselinux.so.1", O_RDONLY) = 3
open("/lib64/tls/libc.so.6", O_RDONLY)  = 3
open("/lib64/tls/libpthread.so.0", O_RDONLY) = 3
open("/lib64/libattr.so.1", O_RDONLY)   = 3
open("/etc/selinux/config", O_RDONLY)   = 3
open("/proc/mounts", O_RDONLY)          = 3
open("/usr/lib/locale/locale-archive", O_RDONLY) = 3
open("/proc/filesystems", O_RDONLY)     = 3
open("/usr/share/locale/locale.alias", O_RDONLY) = 3
open("/usr/share/locale/en_US.UTF-8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.utf8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.UTF-8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.utf8/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_TIME/coreutils.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/etc/nsswitch.conf", O_RDONLY)    = 3
open("/etc/ld.so.cache", O_RDONLY)      = 3
open("/lib64/libnss_files.so.2", O_RDONLY) = 3
open("/etc/passwd", O_RDONLY)           = 3
open("/etc/group", O_RDONLY)            = 3
open("/etc/mtab", O_RDONLY)             = 3
open("/proc/meminfo", O_RDONLY)         = 3
open("/etc/localtime", O_RDONLY)        = 3
4
  • 2
    If you figure this out please post an update.
    – einstiien
    Mar 26, 2010 at 18:36
  • Interesting. what gets reported with ls -ab ? Maybe some odd non octal character is in the file name? I would think -a would list it anyway but I'm not sure.
    – egorgry
    Mar 26, 2010 at 18:52
  • 1
    Have you ran a fsck recently? It is remotely possible something is broke on the filesystem.
    – Zoredache
    Mar 26, 2010 at 19:18
  • I'll have to test it again tomorrow, leaving for the day.
    – sdot257
    Mar 26, 2010 at 20:11

7 Answers 7

7

That's a bit worrysome. I'd verify that your ls file wasn't modified by comparing to a known good file. You could use your distribution's package tools to verify the file on an isolated system.

3
  • +1: ls -al should show everything. If it doesn't there is a problem somewhere. Mar 26, 2010 at 17:46
  • 2
    It's quite possible that ls has been modified to hide a specific PID, possibly 6535.
    – MikeyB
    Mar 26, 2010 at 18:38
  • Nope ... nothing ...
    – sdot257
    Mar 26, 2010 at 19:44
6

Sometimes filenames get odd characters in them such as cursor movement sequences. Try this to make sure:

ls -lq

It should show question marks instead of control characters (it's probably the default, but it might not be).

This partially demonstrates the type of problem that may be present:

touch A C
touch B$(tput cuu1)$'\r'
ls -l
ls -lq
ls -l --show-control-chars    # for systems that have that option and default to -q

I would also try:

type -a ls
alias ls
declare -f ls
md5sum /bin/ls    # compare to a known-good identical system

to see if an alias or function is defined or to see if a binary is in an odd place or has been modified.

2
  • 1
    +1 good insight. It's important to note that if ls were modified, the md5sum on the system could have potentially been modified too. You need a known sane environment to verify on to reach a definitive conclusion.
    – Warner
    Mar 26, 2010 at 18:47
  • I've found that even if md5 has been altered to produce bogus results if you do things like 'md5 file' you can still get good results (if the md5 program works at all) by doing something like bzip2 < file | md5 and compare that to the same command elsewhere.
    – chris
    Mar 28, 2010 at 13:02
3

You may want to fsck that volume.

1
  • 13
    thats what she said.
    – einstiien
    Mar 26, 2010 at 22:23
2

I usually do something like this if I believe 'ls' has been modified...

python -c "import os; print os.listdir('.')"

Of course Python, the C Library, the kernel, or the file system could also be modified, but usually it's just the shell utils.

4
  • 2
    Or, you could use the shell's filename expansion to read the directory -- echo * ( and if you want everything, echo * .* )
    – chris
    Mar 26, 2010 at 19:36
  • *.* is only going to show you files that have character(s) followed by a dot followed by character(s). This is definitely not everything on *nix system. I'm not sure echo will show you everything in one command, I was able to do it echo * && echo .*
    – einstiien
    Mar 26, 2010 at 19:47
  • 4
    If you look closely, it is * (space) .* , not *.* Punctuation is not the strong suit of this commenting system... And, echo is perfectly happy to expand as many expressions separated by the "$IFS" as you care to feed it. The boolean && is not necessary, or really even make much sense, because && is a boolean and and will always work because the echo command is always successful.
    – chris
    Mar 26, 2010 at 20:12
  • @chris: my bad, really hard to see that.
    – einstiien
    Mar 26, 2010 at 22:22
2

You can look into exactly what ls is doing by using strace, and that may tell you why it is avoiding showing that filename.

strace ls -la 653* 2>&1 | less

look that through that and see what's going on.

strace ls -la 653* 2>&1 | grep ^open

The output will look like this:

open("/etc/ld.so.cache", O_RDONLY)      = 3
open("/lib/librt.so.1", O_RDONLY)       = 3
open("/lib/libacl.so.1", O_RDONLY)      = 3
open("/lib/libselinux.so.1", O_RDONLY)  = 3
open("/lib/libc.so.6", O_RDONLY)        = 3
open("/lib/libpthread.so.0", O_RDONLY)  = 3
open("/lib/libattr.so.1", O_RDONLY)     = 3
open("/lib/libdl.so.2", O_RDONLY)       = 3
open("/lib/libsepol.so.1", O_RDONLY)    = 3
open("/etc/selinux/config", O_RDONLY|O_LARGEFILE) = 3
open("/proc/mounts", O_RDONLY|O_LARGEFILE) = 3
open("/selinux/mls", O_RDONLY|O_LARGEFILE) = 3

and if you see something like

open("/var/tmp/.../H@ckl1st", O_RDONLY) = 3

be careful, you've been 0wned...

This isn't a conclusive test, but it is a good indicator...

(if you're using solaris or other OSs, you may need to use truss, or some other similar utility instead of strace)

(if you're using a csh/tcsh derived shell, you'll likely need different redirection statements)

2
  • I like this. The strace utility really is a swiss army knife. You get right to the system call level and bypass a whole pile of arbitrary complication. It's one of the first things any system admin. worth a dime ought to dump on a newly installed machine.
    – McJeff
    Mar 26, 2010 at 20:30
  • Yeah -- the two most valuable tools for a systems administrator are truss / strace and tcpdump. With these, you can always look under the covers to see wtf is going on when something is or isn't behaving the way you expect.
    – chris
    Mar 26, 2010 at 20:52
2

Quick update, we had to replace the server for other reasons. It was the filesystem. All is well now!!! Thank you everyone.

3
  • You mean the filesystem was screwed up and this was just a wacky symptom?
    – chris
    Apr 28, 2010 at 20:12
  • yep that was it.
    – sdot257
    May 10, 2010 at 18:52
  • You probably could have stuck in an edit to say there's a solution in the list below, since it's been buried beneath other upvoted (and useful for troubleshooting) answers. Sep 22, 2010 at 11:36
0

The hack theory is interesting, but I have an alternative theory. Unix file deletion semantics will keep the file around until all processes have closed open file handles pointing at it. Perhaps someone has paused an SVN checkout / commit, or a server thread hung up. If restarting the SVN process (or Apache) solves your problem, this is where I'd place the blame.

Perhaps you can identify the process still using this file with lsof | grep 6535?

3
  • yea lsof didn't show anything. The interesting thing is, 6535 is also "missing" in the source. My script does a hotcopy of the original repo to another directory. That's when I ran into issues with not being able to delete this particular file for some reason.
    – sdot257
    Mar 26, 2010 at 20:21
  • A deleted but open file won't keep you from deleting the containing directory because once that directory entry is deleted, the directory entry won't exist so the directory will be empty. You won't get the space back from the file until the process that has it open is killed, but the directory that had that file can now be deleted.
    – chris
    Mar 26, 2010 at 20:24
  • Your alternate theory is interesting. The removal, if successful, would instantly remove the hard link. The inode would likely still contain the data and some file handles may have it cached in memory, but I do not believe this scenario could explain the described behavior. Or, what chris said, heh.
    – Warner
    Mar 26, 2010 at 21:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .