What does this do in bash: [@]?

[Colin Watson]

On Sat, Jul 30, 2022 at 10:29:30PM +1000, Karl Auer wrote:
> On Sat, 2022-07-30 at 12:16 +0100, Colin Watson wrote:
> > On Sat, Jul 30, 2022 at 10:19:19AM +0100, Peter Flynn wrote:
> > > On 30/07/2022 00:41, Colin Watson wrote:
> > > > This sort of thing is generally much easier in Python,
> > > 
> > > It's actually much easier in pretty much anything except the shell
> > > :-)
> > 
> > Quite!  About the only virtue of the POSIX shell (along with its
> > relatives, e.g. bash) is
> 
> The single biggest virtue is that it is everywhere, dependably, on
> every Linux machine (very niche/specialist distros sometimes excluded).

Yeah, I intended to cover that sort of thing with my "inertia" comment.

> > Beyond that, while it _can_ be used for programming [...] it's
> > perhaps the worst-designed language for that purpose that I use
> > routinely.
> 
> Because it wasn't designed for that? But I too have become adept at
> solving problems with it that would be much better solved in Python or
> Perl - and why? Because it is always there. Nothing has to be
> installed. There are no versions (python or python3?). There are no
> libraries (ooh, do I have to install urllib? urllib3?). There are no
> library paths. It is there and it is ALL there and it is ALWAYS there.

There are indeed cases where having all your program's dependencies
there by default is convenient, and indeed cases where adding any extra
dependencies is difficult if not prohibitive (I worked on the Debian
installer for many years, which had this sort of restriction, and
scripts to bootstrap projects from scratch often use shell for this
reason).  All the same, shell programming has enough actively dangerous
footguns for writing reliable programs that I think there comes a point
where the ethical thing to do is to recommend that people use something
else rather than struggling on with shell.

Bo quite often posts questions along the lines of the post that started
this thread, apparently related to various download scripts.  I'm sure
there were reasons why those scripts ended up this way.  Nevertheles,
while I could be wrong since I have limited context, it doesn't seem as
though these are running in especially dependency-constrained
environments.  We're told that the script looks something like this:

    CURL=$(command -v curl) #Get full path to the curl command
    
    FFMPEG=$(command -v ffmpeg) #Get full path to the ffmpeg command
    
    URL=('https://tokens.xxxxx.com/' -d $'{\n "pid": "2007524",\n "application":
    "Sports",\n "platform": "desktop",\n "cdn": "akamai",\n "url":
    "https://xxxxxxx.akamaized.net/hls/live/723429/Sportsa418c506/clear/master.m3u8"\n}')
    
    INPUT=$("$CURL" -qsS "${URL[@]}"  | jq -Mr '.akamai[] | .tokenizedUrl')
    
    ...
    
    CMD="ffmpeg -hide_banner -i \"${INPUT[@]}\" -vf scale=w=-4:h=480 -c:v libx264
    -preset fast -crf 26 -c:a aac -t ${CAPTURETIME} ${TARGETFILE}"
    
    eval "$CMD"

Now, I don't have the whole script to go on, and for example I don't
know where ${CAPTURETIME} and ${TARGETFILE} are coming from.  But in
Python that might look something like this:

    #! /usr/bin/python3

    import subprocess

    import requests

    data = {
        "pid": "2007524",
        "application": "Sports",
        "platform": "desktop",
        "cdn": "akamai",
        "url": "https://xxxxxxx.akamaized.net/hls/live/723429/Sportsa418c506/clear/master.m3u8",
    }
    response = requests.get("https://tokens.xxxxx.com/", data=data)
    response.raise_for_status()
    input = [item["tokenizedUrl"] for item in response.json()["akamai"]]

    cmd = ["ffmpeg", "-hide_banner"]
    for url in input:
        cmd.extend(["-i", url])
    cmd.extend([
        "-vf", "scale=w=-4:h=480",
        "-c:v", "libx264",
        "-preset", "fast",
        "-crf", "26",
        "-c:a", "aac",
        "-t", capturetime,
        targetfile,
    ])
    subprocess.run(cmd, check=True)

Getting likely criticisms out of the way first: yes, it's a bit longer
(mostly because I tried to space it out sensibly; you could squash it
down if you wanted to sacrifice readability), and I might have got some
details wrong since I wasn't completely sure of everything that was
going on in the original and I don't have everything to hand to be able
to test it.  Executing subprocesses involves a little more typing (some
quoting plus a library call), because, unlike shell, that isn't Python's
primary focus.  And while Python's exception handling normally means
that you don't have to spend a lot of time on error-checking ceremony,
there are a couple of places here where you have to remember to tell
libraries to check certain kinds of errors.

On the other hand, lists are just lists: you don't have to choose
between bash's array syntax and its weird punctuation or squashing
everything down into strings and hoping that you don't get bitten by
something having a space in it when you weren't expecting it.  You can
prepare commands as lists without having to work out how to safely deal
with shell expansion and quoting (if you really need to do that sort of
thing, the shlex module exists, but it's almost never needed).  You
don't have to encode JSON by hand.  And you can walk through JSON
responses as ordinary data structures without having to work out how to
spell your query in jq's language.

As a bonus, this actually involves installing _fewer_ additional
dependencies on typical Ubuntu systems than the original shell
formulation, because python3 and python3-requests are installed out of
the box on most Ubuntu systems (including desktop images of pretty much
all flavours that I can see, as well as cloud images and typical server
installations), while jq isn't.  And sure, the Python 3 transition was
painful, but it's not 2012 any more; unless you're maintaining some kind
of complicated legacy system, you can just use python3 and not worry
about it.

Other languages are available; I'm not going to twist your arm and tell
you that you have to use Python.  But seriously, nearly any sensible
general-purpose scripting language is likely to be better than bash for
this sort of thing, and most people posting questions about shell
minutiae would probably be able to write more reliable programs with one
of them instead after quite a short learning curve.  I don't think we
should be afraid to say so just because there are some situations where
the shell is all you have.

I guess I view all this as a special case of the XY problem
(https://en.wikipedia.org/wiki/XY_problem).  The root problem is
something like constructing command lines out of fairly arbitrary lists
of items, but the attempted solution involves using the shell language's
really quite weak and obtuse support for arrays.  Solving the root
problem without the constraint of using shell would very likely lead to
a better solution.

-- 
Colin Watson (he/him)                              [cjwatson at ubuntu.com]