14

Which question and answer should be the canonical for all the many well-trodden duplicates/variants on the Python theme "Why is my command not being expanded by subprocess.run/call/Popen* / exec/ shlex/ etc.?", often asked as just "Why isn't my command working?" or "subprocess fails". Meanwhile, please start aggressively closing and stop answering new duplicates.

(The canonical answer is:

  1. subprocess.Popen(..., shell=False) is the default, you can either expand your wildcards with a glob call, or you can enable shell=True if you want, but it's a security hole when called with user input, so it's not recommended
  2. But either way, whichever call you make to expand '*' wildcard, you then need to escape backslashes, and/or significant whitespace within file/directory names (note esp. Windows users). Otherwise that call will now start breaking for those new reasons.
  3. Also (duh!) check that the result of expanding the wildcards is non-empty, otherwise the final subprocess/ exec call will now fail for the reason that one of its filename/directory args is empty.
  4. Other things that shell=True provides other than wildcard expansion (excl. Windows), variable interpolation, and stdio redirection

Here are some of the many variant questions, but the canonical might well be something else, or maybe one of us just has to write one to cover all the many many variants (but a comprehensive answer will be very long, maybe too long to be readable. Thoughts?):

UPDATE:

8
  • Why do you have to escape forward slashes and whitespace for a call to glob.glob?
    – rici
    May 21, 2018 at 17:55
  • 1
    no need to escape fwd slashes. Just backward slashes (windows). And shell=True doesn't necessarily expand, depends on the shell (on windows, it doesn't) May 21, 2018 at 21:20
  • @Jean-FrançoisFabre: thanks for the correction. Please just directly edit this question. Is that true for all of subprocess, glob, shlex?
    – smci
    May 22, 2018 at 0:14
  • glob expands. Others don't. your edit makes sense May 22, 2018 at 6:22
  • 1
    Note: shell=True is always a security hole, even when used with hardcoded commands. See the ShellShock attack which exploits the environment of the process to inject the command to run (by default subprocess.Popen & co inherit the parent environment). Luckily this should have been fixed but there surely are a ton of legacy system with this problem and there may be some other issue. So if you want to be safe when using shell=True you should always provide the env parameter too. May 22, 2018 at 7:40
  • shell=True is for the lazy. It's good to test something/ensure that you can run the command in "dirty" mode, but there's always a solution to avoid it and it's more performant to go that way anyway because it doesn't run a shell, cannot redirect to files, whatever... May 22, 2018 at 15:44
  • 1
    Is this meant to be Python-only, or used across all languages that have some facility for executing programs without the shell? (And, if the former, is there a reason not to put this on sopython.com/canon like all the other canonical dup targets?)
    – abarnert
    May 22, 2018 at 22:16
  • @abarnert: Python. As to the latter, I don't understand you, we don't have any canonical question & answer yet, so there is nothing yet to add to sopython.com/canon. Please help sketch everything that the canonical should address. This is going to take some work.
    – smci
    May 23, 2018 at 0:51

0

You must log in to answer this question.