38

There are a truly massive number of questions on Stack Overflow about Python's module-import system: thousands tagged , thousands more tagged both and , thousands more tagged instead, hundreds more tagged instead (admittedly getting into less popular stuff now), and who knows how much dreck beyond that, not tagged as anything to do with importing. (Of course, those latter queries will contain a lot of false positives, but it gives some sense of the scale of the problem.)

Of course, a huge percentage of these are simply duplicates, and a huge percentage of those are not yet marked as duplicates. More are asked daily; it's almost always the same few common problems.

Of course, policy for that is trivial, right? Just hammer away, right?

The problem is that all our purported canonicals are garbage. Just today, for example, Why does my python program raise a ModuleError only when ran in a unittest? was asked. It's a very simple problem that's easy to understand and explain, and must come up constantly and have been asked and answered countless times before - but I don't know the right canonical for it, despite all the time I've been spending curating the site and despite the special attention I pay to questions about importing (one of my two largest areas of interest, along with text encoding).

The problems are many, and severe:

  • Even among questions with, say, 100+ score, there are just still way too many of them to choose from.
  • Because the questions arose organically, they'll generally have titles that describe an error or symptom, but not a problem. A hammer-wielder can easily read the question, recognize what went wrong - and then struggle to find the right question, because the titles don't allow for matching them up to the problem.
  • Worse yet, the titles will be clickbait.
    • A question titled Python error "ImportError: No module named" has 2.3 million views, and OP's issue was effectively a typo (on a version where __init__.py files were necessary, OP had __init__.py.bin files instead because of a network file copying issue). (Notice the title match to the question I mentioned in the introduction. No, it is not the same problem. At all.)
    • One of the best known questions is literally titled "Relative imports for the billionth time". That seems like a pretty bad look. It definitely doesn't describe the problem very well (the question is about one specific thing that goes wrong with relative imports). There's a comment on the question with almost 700 upvotes (far and away the most I've ever seen) that's just ranting about the relevant feature, without any attempt to understand it (it is fairly simple, despite the continued refusal of so many people to accept that it works the way that it does), and calling upon the designer of the language to change how it works. (Protip: suggestions of this sort will get zero traction in the Stack Overflow comment section; try https://discuss.python.org/c/ideas/6, but prepare to get flamed if you come in guns blazing without a proper understanding of the existing system.)
    • The most popular question I could find on the topic - at 3 million views - set itself up as a debugging question, but didn't even describe a problem. I actually edited in "but it doesn't work" just to make it read sensibly, after another editor took out "and some other various attempts but so far I couldn't manage to import properly.", which was OP's original wording from 2010. So we never got any idea what the "other various attempts" were, nor about what went wrong. But we can't really edit that out and make it a pure how-to question, because most of the answers address the question from a debugging perspective (lacking debugging information, they suggest standard workarounds rather than anything like, you know, approaching the problem with a proper understanding). I have to admit, "Importing files from different folder" sounds like a really useful thing to know about - but the answers are at best a clumsy attempt at addressing that topic, which is really a bit too broad anyway. A question with a title like that ought instead to have a summary of the techniques (absolute, relative and dynamic, plus a historical note for the old relative-falling-back-to-absolute system), and links to technique-specific canonicals.
  • Meanwhile, every question about a particular error or symptom will become a target for answers about every possible cause of that problem, even if they are completely irrelevant to the setup described in the OP (and that's just considering the questions that actually have a proper MRE and didn't get hundreds of upvotes just because they're old and express a common frustration). There's massive redundancy between questions, but no good way to say that one is a duplicate of another.
  • The questions just plain have way too many answers, and the important information will often be spread across multiple answers. In many cases, a lot of the answers are duplicates or near-duplicates, but can't be cleaned up because they were either contemporary to the question or understanding the redundancy requires subject matter expertise. There's also the power-set problem whereby if a problem could be solved by A, B or C, you'll get seven answers mentioning {{A}, {B}, {C}, {A, B}, {B, C}, {A, C}, {A, B, C}}. If you're lucky. If you're not lucky, you get six answers, no points for guessing which one is missing. Plus they keep attracting new garbage answers because the "protect question" feature is underpowered.
  • Speaking of answers, they're severely outdated and full of blatant misinformation - because they were answered by people reporting what "worked for them", and answers were upvoted in cargo-cult fashion. The main issues here are
    • repeated insistence on including __init__.py files everywhere (since 3.3, these are not needed by Python to actually find files to import, and thus not relevant to almost any problem for which they're suggested; they do impact on the import system, but in much subtler ways)
    • a massive proliferation of inappropriate suggestions for sys.path modification hacks, typically not accompanied by any explanation of what sys.path is or how it is used. (These are almost never necessary, and sys.path is manipulated rarely or never by many of the most popular and significant third-party libraries, spanning hundreds of thousands of LOC each. TensorFlow, for example, modifies it once in a Python notebook in a tutorial for the documentation, and also reads the path as part of a big sweep looking for places to import other components dynamically. Keras doesn't at all, AFAICT. Requests has it commented out, once, in a config file.)
    • some answers reference, without a clear versioning indication, a former "implicitly relative" import behaviour which was The Way Things Worked... until 2.5 which offered explicit relative imports and the option to make the other imports explicitly absolute (which became How It Works in 3.0). (Yes, this part could be cleaned up, but it goes towards showing just how little attention has been paid to curation.)
  • Everything outside the "long tail of crap" is upvoted to the sky no matter how bad it is, so it can't easily be pushed out of the way where that's needed.

I could go on, but I already feel my blood pressure rising. The thing is, the underlying topic is really not this complicated and doesn't require this huge pile of Q&A to address everything. Although Python's import system is very flexible and customizable, the default workings are really just not that hard to get right. It's just that the good advice drowns in the bad, and the bad isn't even well organized - and people are eager to reach for heavy-duty tools in cases where the simple ones work just fine, just because of a lack of understanding. And because of this, we get to stare at a huge mass of people complaining about complexity and pain and annoyance that isn't really there.


My conclusion is that the primary root of the problem is the sourcing of questions. People ask questions about import because they have a specific problem they are trying to solve, and it's usually a "something went wrong" problem rather than a "I would like to do XYZ" problem. As a result, nobody is thinking about cleaving the problem space at its joints.

If there were a proper set of canonical questions, asked and titled with foresight and deliberation, then it would be much clearer what answers go where. Dupe-hammerers would easily be able to identify the right targets (because their subject matter expertise allows them to recognize the problem quickly).

After that, closing and dupe-hammering old questions would do wonders in terms of directing people to good information that actually works and away from comment sections inviting them to join a pitchfork mob against the Python dev team.

Aside from gaining consensus on this point, I would like to hear about:

  • suggestions for what such a canonical set might include
  • any other specific complaints you have about extant canonicals
  • any especially good Q&A about Python's import system that you feel goes unrecognized.
13
  • 4
    "I don't know the right canonical for it" - Why can't you just write one then start using it as a canonical yourself? The entire point of a canonical answer, is to make that answer, easier to find by creating linking it to multiple duplicates. The author's of the duplicate questions ultimately receive a question, and the community, gain multiple questions to a single answer. So if a good canonical does not exist, write a good answer to a question, then start using it. You can also link multiple duplicates to a single question if you have a gold badge. Commented Mar 27, 2023 at 21:27
  • 13
    @SecurityHound Yes, I understand that, and have done similarly many times in the past, for simpler cases. This case is different because of the size of the mess and because there is a general topic that needs several canonicals (but not thousands) to address properly. The point of this question is to determine which canonicals are needed and plan that out - so that a) we can have questions that neatly cover the problem space without significant overlaps; b) we don't have multiple people trying to write the same canonical simultaneously. Commented Mar 27, 2023 at 21:34
  • 3
    I'll briefly call out that having a title that's "clickbait" shouldn't be a marker of it being "bad". A title can be both descriptive and clickbait-y, so it's better to use that as an advantage when building out a new canonical.
    – Makoto
    Commented Mar 27, 2023 at 22:31
  • @Makoto agreed; I didn't explain that part as well as I wanted to (need better vocabulary I guess). Commented Mar 27, 2023 at 22:41
  • 3
    I vote for better tutorials. Because I can venture a guess that people who ask these same questions over and over again are following one rather than using a quality source of information. Tutorials are always written as if it is impossible for things to go wrong, they should incorporate at least basic failure scenarios for the people who are too enthusiastic and do everything wrong as a result of it. Like teaching people to recognise and solve a basic import error. Sigh. Dreams are free, but never realistic.
    – Gimby
    Commented Mar 28, 2023 at 10:23
  • 2
    @Gimby I try to write certain of my canonicals in a tutorial-like style, acknowledging that things do go wrong. That does mean that it isn't a linear set of steps. That said, we can't fix off-site tutorials, and we can't write and publish "classical" tutorials in the main Q&A space (and couldn't really have in Documentation either). Commented Mar 28, 2023 at 10:38
  • 2
    Your request for canonicals is being diluted by the rant on how easy the problem should be. Can you conclude with a list of questions you imagine needing canonicals?
    – Passer By
    Commented Mar 30, 2023 at 15:10
  • 3
    @PasserBy I think the list of canonical questions that should be written would be the answer to this meta question
    – Bergi
    Commented Mar 30, 2023 at 15:12
  • 2
    I don't know why you want consensus on what to do here. Meta is where consensus comes to be drowned under the "what ifs" and "but maybes" and "Shog saids" and everyone else who would rather design perfect ivory towers, than build houses that are good enough. You seem to be motivated to do something about this, so... do it! Make your own canonicals, make them good, dupe-flag the bad canonicals that currently exist. You have the power.
    – Ian Kemp
    Commented Mar 30, 2023 at 18:42
  • 3
    @IanKemp I'm trying to figure out which canonicals are needed. Brainstorming isn't much fun solitaire. That said, I very much appreciate that there is a strong show of support for doing something. I will start, and make my best effort to publicize my efforts and look for more specific help as I become able to specify. Commented Mar 31, 2023 at 2:41
  • 5
    @KarlKnechtel It's much easier to get people to buy in and get involved when you've got something concrete for them to make suggestions about, rather than trying to build consensus around relatively abstract ideas and concepts of how things should be done.
    – Ian Kemp
    Commented Mar 31, 2023 at 9:53
  • Whoops, forgot about this one. My point was that the source of your pending madness is offsite, and it is going to stay that way and get only worse as more bad tutorials are written and filmed. So write your canonical if you must, but I fear the next phase is simply going to be you going insane because you're close voting to the same canonical over and over and over. Me, I would stick to downvoting questions that are not useful to be honest. A quick way to close the book and you've still sent out the signal.
    – Gimby
    Commented Apr 13, 2023 at 12:02
  • 1
    @Gimby repeatedly closing questions doesn't bother me at all. What drives me mad is not having a good canonical for something that clearly ought to have one. Commented Apr 13, 2023 at 14:42

0

You must log in to answer this question.

Browse other questions tagged .