2

I have an array of paths:

paths=(
  /foo/exists1
  /foo/exists2
  /foo/missing1
)

To find those that are missing:

ls "${paths[@]}" 1>/dev/null

Shows:

ls: cannot access '/foo/missing1': No such file or directory

Good. Now I want to clean this up:

ls "${paths[@]}" 1>/dev/null | sed 's/ls: cannot access //' | sed 's/: No such file or directory//''

But I get:

ls: cannot access '/foo/missing1': No such file or directory
/foo/exists1
/foo/exists2

So the sed doesn't work, and existing files are also shown.

Why does that happen (why does it ignore the 1>/dev/null), and how do I fix this?

6
  • My interpretation is that you want to remove ("extract") elements from paths that don't exist; is that correct? Or, similarly, you want to print only the elements from paths that do exist? Commented Apr 19, 2020 at 19:02
  • @JeffSchaller Yeah I'd like to extract missing paths, so I can print them as warnings, e.g. "warning path X does not exist", etc. (Actually, this is a list of paths fed into tar, so I can extract the good paths and use them in tar, and extract the bad paths and log warnings without allowing tar to fail... my solution below works, but I'm not sure it's the best as I've read not to rely on ls, but rather globstars, but it works for now)
    – lonix
    Commented Apr 20, 2020 at 4:47
  • Thank you for the clarification, Ionix! The word "extract" could be read two different ways ("pull it out and hold it up" or "remove it"), so I wanted to make sure I understood. I would suggest a slight edit to your question to add the "tar vs log warnings" context, since that helps people give you answers that are more directly usable. Commented Apr 20, 2020 at 19:01
  • @womble Funny how I can never get it right on StackExchange. If I'd posted it on SO then someone would have closed it and said to post on SF. When I post on SF someone gets irate and closes it and says to post on SO. On SO I'd say that scripting is "programming thing" and someone would tell me that it's a "sysadmin thing" and go to SF.
    – lonix
    Commented Apr 21, 2020 at 7:21
  • it is tricky to get a feel for a site's scope, unfortunately. I myself didn't realize scripting was off-topic here at SF. For questions like this one, where you're trying to solve a problem in a UNIX/Linux environment, and you're having trouble getting it just right, consider the U&L Stack Exchange: unix.stackexchange.com! Commented Apr 21, 2020 at 10:54

2 Answers 2

2

You've already discovered the immediate reason your code didn't do what you expected: errors from ls are reported to stderr (as suggested by POSIX), which is not captured as input by the pipe. You therefore got a mixture of normal output (which passed through unchanged by your sed statements) and stderr (which bypassed them). I do not know why your ls output changed between calls; redirecting stdout to /dev/null should have the effect of removing all "normal" (existing paths) from the output. The fix for this is not to shove stderr into stdout, though.

Post-processing the output from ls is a dangerous idea if you want a reliable script. One good article on the topic is "Why you shouldn't parse the output of ls(1)", available on the wooledge.org site. One in-depth Q/A at the Unix & Linux site goes into some of issues: Why not parse ls (and what to do instead)?. The upshot is that UNIX filenames can contain almost any character, including spaces, tabs, newlines, single quotes, double quotes, escaped single quotes, etc! For some quick examples, consider directories by these names, all of which are perfectly legal:

  • "No such file" (mkdir "No such file")
  • "ls: cannot access 'foo': No such file or directory" (mkdir "ls: cannot access 'foo': No such file or directory")
  • "directory

    with

    embedded

    newlines" (mkdir $'directory\nwith\nembedded\newlines')

The first is an innocent directory that is wrongfully captured (from stdout) by the grep. The second is also wrongfully captured, but then further mangled into a completely different path -- which may or may not exist! -- by the sed statements. The third is one example of what happens when you pass the output of ls into line-oriented programs; if the directory doesn't exist, ls will say so on more than one line, which is probably how you ended up with two separate sed statements!

To distinguish "good paths" -- ones that exist and are readable -- from "bad paths", I would suggest looping over the array and building new arrays of each.

for p in "${paths[@]}"
do
  if [ -r "$p" ]
  then
    goodpaths+=("$p")
  else
    badpaths+=("$p")
  fi
done

You can then do whatever you like with each set:

printf 'Good path: -->%s<--\n' "${goodpaths[@]}"
echo
printf 'Bad path: -->%s<--\n' "${badpaths[@]}"
1
  • Thanks! Your way is much safer. My way works but it might break tomorrow when some weird paths get into my "to-backup" list.
    – lonix
    Commented Apr 21, 2020 at 7:22
0

Problem turned out to be obvious... errors go to file descriptor 2, not stdout.

So must redirect 2 to 1 then grep:

  paths=(
    /foo/exists1
    /foo/exists2
    /foo/missing1
  )
  ls "${paths[@]}" 2>&1 \
    | grep 'No such file' \
    | sed 's/ls: cannot access '\''//' \
    | sed 's/'\'': No such file or directory//'

Not the answer you're looking for? Browse other questions tagged .