[lug] Python: Unable to catch subprocess error

[Rob Nagler]

On Sat, Mar 10, 2018 at 6:18 AM, Davide Del Vento wrote:
>
> It must be said that some say the source code itself is the true
> documentation in python, and it's usually true because it can be written
> (and often is) very expressively.
>

Which version? ;-)

Python is idiomatic. Reading the source code only helps when you are
looking for something very specific, and can understand when important code
is missing. In Jed's case, you won't find the reason "shell=True" changes
the semantics of "program not found", because it is code that wasn't
written.

I think Python is written literally, not expressively, and that makes it
easy to read in some sense but much harder in many others. I'll pick on
subprocess.py a bit, just to see if we can figure out figure out why the
semantics change in the shell=True case. Here's one "if shell" section in
subprocess.py:
<https://github.com/python/cpython/blob/be50a7b627d0aa37e08fa8e2d5568891f19903ce/Lib/subprocess.py#L1374>

if shell:

# On Android the default shell is at '/system/bin/sh'.
unix_shell = ('/system/bin/sh' if
        hasattr(sys, 'getandroidapilevel') else '/bin/sh')
args = [unix_shell, "-c"] + args
if executable:
    args[0] = executable


And the other "if shell" section:
<https://github.com/python/cpython/blob/be50a7b627d0aa37e08fa8e2d5568891f19903ce/Lib/subprocess.py#L1140>

if shell:
    startupinfo.dwFlags |= _winapi.STARTF_USESHOWWINDOW
    startupinfo.wShowWindow = _winapi.SW_HIDE
    comspec = os.environ.get("COMSPEC", "cmd.exe")
    args = '{} /c "{}"'.format (comspec, args)

If it were written expressively, these would be functions that name the
semantics. The literal aspect is that these code snippets for handling "if
shell" are both defined in "_execute_child" functions (yes, plural). The
two _execution_child functions are cleverly defined in a 743-line "if
_mswindows" statement inside a class definition. That makes it hard to
search for what's going on, since the POSIX and Windows sections have
mostly identically named functions. However, you can learn lots of
interesting and *non-expressive* Python tricks, e.g. you can have an "if"
in a class declaration to control method declarations. You might expect to
see something like that in C, but it's silly (and wrong imiho) in Python
which supports polymorphism in other, more expressive ways. subprocess.py
is full of silliness like that (lack of feature testing, too long code,
inline constants, etc.). And, it's a microcosm for CPython imiho.

It's why I say programming in Python is like being inside a Monty Python
sketch. You never know what's going to happen next, and none of it makes
any sense, but if you keep your sense of humor about you, it's a barrel of
laughs! How about this for something completely different: Python 2 and 3
have very different string types (ha ha!) so you might use
types.StringTypes to ensure forward compatibility between the two versions
except that types.StringTypes went away in Python 3. In fact the semantics
of the module "types" change completely from Python 2 to 3. That's some
really silly programming!

The relevant point to this thread is that reading the source code doesn't
give a clue as to why shell=True changes the semantics of "program not
found" exception handling in Python. It's actually just lazy programming.
To make the semantics equivalent in the two cases, _execute_child (the
POSIX version) should parse the exit code of "/bin/sh -c" (127 is a known
exit status for program not found
<http://tldp.org/LDP/abs/html/exitcodes.html>). A newbie can't be expected
to see that code to handle this special case is not there.

Rob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.lug.boulder.co.us/pipermail/lug/attachments/20180310/b9f7179d/attachment.html>