Bash: for…in-loop fails, what am I doing wrong?

[Paul Smith]

On Thu, 2013-01-10 at 20:32 +0100, Johnny Rosenberg wrote:
> > In a script, (easier to read…):
> > for File in "$(find ${PWD} -regextype posix-extended -regex
> > '.*/P[A-Z0-9][0-9]{6}\.(jpg|JPG|jpeg)')"
> 
> It's those quotes that cause my problems, isn't it?

Yes.  The quotes cause the shell to interpret the entire output of the
find command as one word, so the loop executes once with the variable
$File set to everything generated by find.

> But if I omit them, paths with spaces would go wrong, wouldn't they?
> So what now…? :(

The short answer is that it's virtually impossible to write shell
scripts which are completely robust in the face of every possible
filename.  For example, in UNIX it's perfectly legal to have a filename
that contains a newline!  In fact, the only characters which can never
appear in a filename are the "/" character and "\0" (nul byte, ASCII 0).

If you really want to write a lot of scripting manipulating files which
may have arbitrarily bizarre filenames, you probably need to give up on
the shell and start learning something more powerful, such as Perl.

Something that will be more reliable than the above would be:

  find -regextype posix-extended -regex '.*/P[A-Z0-9][0-9]{6}\.(jpg|JPG|jpeg)' \
      | while IFS= read -r File; do
          date --reference="${File}" '+%Y%m'
        done

Even this is not completely foolproof (won't work with filenames
containing newlines, for example), but it will handle "straightforward"
filenames, containing spaces/tabs for example.

One thing to remember is that because the while loop is in a pipeline,
any variable assignment inside the while loop will NOT be set anymore
after the loop completes (in many shells, including bash).

There is only one really foolproof method that I can think of, if you
want to work entirely in the shell: you need to create a script that
takes the files to manage on the command line, then invoke find with the
-exec option passing it your script.

For example, suppose you wrote a script like this:

  $ cat myscript
  #!/bin/sh

  for File in "$@"; do
      date=$(date --reference="$File" ....)
      # more stuff here
  done

Then you could invoke it like this:

  find -regextype posix-extended -regex '.*/P[A-Z0-9][0-9]{6}\.(jpg|JPG|jpeg)' -exec myscript '{}' \;

This should be completely reliable, even for very strangely named files,
as long as you're careful to quote all variables in the "myscript"
script.