There are a truly massive number of questions on Stack Overflow about Python's module-import system: thousands tagged python-import, thousands more tagged both python and import, thousands more tagged importerror instead, hundreds more tagged modulenotfounderror instead (admittedly getting into less popular stuff now), and who knows how much dreck beyond that, not tagged as anything to do with importing. (Of course, those latter queries will contain a lot of false positives, but it gives some sense of the scale of the problem.)
Of course, a huge percentage of these are simply duplicates, and a huge percentage of those are not yet marked as duplicates. More are asked daily; it's almost always the same few common problems.
Of course, policy for that is trivial, right? Just hammer away, right?
The problem is that all our purported canonicals are garbage. Just today, for example, Why does my python program raise a ModuleError only when ran in a unittest? was asked. It's a very simple problem that's easy to understand and explain, and must come up constantly and have been asked and answered countless times before - but I don't know the right canonical for it, despite all the time I've been spending curating the site and despite the special attention I pay to questions about importing (one of my two largest areas of interest, along with text encoding).
The problems are many, and severe:
- Even among questions with, say, 100+ score, there are just still way too many of them to choose from.
- Because the questions arose organically, they'll generally have titles that describe an error or symptom, but not a problem. A hammer-wielder can easily read the question, recognize what went wrong - and then struggle to find the right question, because the titles don't allow for matching them up to the problem.
- Worse yet, the titles will be clickbait.
- A question titled
Python error "ImportError: No module named"
has 2.3 million views, and OP's issue was effectively a typo (on a version where__init__.py
files were necessary, OP had__init__.py.bin
files instead because of a network file copying issue). (Notice the title match to the question I mentioned in the introduction. No, it is not the same problem. At all.) - One of the best known questions is literally titled "Relative imports for the billionth time". That seems like a pretty bad look. It definitely doesn't describe the problem very well (the question is about one specific thing that goes wrong with relative imports). There's a comment on the question with almost 700 upvotes (far and away the most I've ever seen) that's just ranting about the relevant feature, without any attempt to understand it (it is fairly simple, despite the continued refusal of so many people to accept that it works the way that it does), and calling upon the designer of the language to change how it works. (Protip: suggestions of this sort will get zero traction in the Stack Overflow comment section; try https://discuss.python.org/c/ideas/6, but prepare to get flamed if you come in guns blazing without a proper understanding of the existing system.)
- The most popular question I could find on the topic - at 3 million views - set itself up as a debugging question, but didn't even describe a problem. I actually edited in "but it doesn't work" just to make it read sensibly, after another editor took out "and some other various attempts but so far I couldn't manage to import properly.", which was OP's original wording from 2010. So we never got any idea what the "other various attempts" were, nor about what went wrong. But we can't really edit that out and make it a pure how-to question, because most of the answers address the question from a debugging perspective (lacking debugging information, they suggest standard workarounds rather than anything like, you know, approaching the problem with a proper understanding). I have to admit, "Importing files from different folder" sounds like a really useful thing to know about - but the answers are at best a clumsy attempt at addressing that topic, which is really a bit too broad anyway. A question with a title like that ought instead to have a summary of the techniques (absolute, relative and dynamic, plus a historical note for the old relative-falling-back-to-absolute system), and links to technique-specific canonicals.
- A question titled
- Meanwhile, every question about a particular error or symptom will become a target for answers about every possible cause of that problem, even if they are completely irrelevant to the setup described in the OP (and that's just considering the questions that actually have a proper MRE and didn't get hundreds of upvotes just because they're old and express a common frustration). There's massive redundancy between questions, but no good way to say that one is a duplicate of another.
- The questions just plain have way too many answers, and the important information will often be spread across multiple answers. In many cases, a lot of the answers are duplicates or near-duplicates, but can't be cleaned up because they were either contemporary to the question or understanding the redundancy requires subject matter expertise. There's also the power-set problem whereby if a problem could be solved by A, B or C, you'll get seven answers mentioning {{A}, {B}, {C}, {A, B}, {B, C}, {A, C}, {A, B, C}}. If you're lucky. If you're not lucky, you get six answers, no points for guessing which one is missing. Plus they keep attracting new garbage answers because the "protect question" feature is underpowered.
- Speaking of answers, they're severely outdated and full of blatant misinformation - because they were answered by people reporting what "worked for them", and answers were upvoted in cargo-cult fashion. The main issues here are
- repeated insistence on including
__init__.py
files everywhere (since 3.3, these are not needed by Python to actually find files to import, and thus not relevant to almost any problem for which they're suggested; they do impact on the import system, but in much subtler ways) - a massive proliferation of inappropriate suggestions for
sys.path
modification hacks, typically not accompanied by any explanation of whatsys.path
is or how it is used. (These are almost never necessary, andsys.path
is manipulated rarely or never by many of the most popular and significant third-party libraries, spanning hundreds of thousands of LOC each. TensorFlow, for example, modifies it once in a Python notebook in a tutorial for the documentation, and also reads the path as part of a big sweep looking for places to import other components dynamically. Keras doesn't at all, AFAICT. Requests has it commented out, once, in a config file.) - some answers reference, without a clear versioning indication, a former "implicitly relative" import behaviour which was The Way Things Worked... until 2.5 which offered explicit relative imports and the option to make the other imports explicitly absolute (which became How It Works in 3.0). (Yes, this part could be cleaned up, but it goes towards showing just how little attention has been paid to curation.)
- repeated insistence on including
- Everything outside the "long tail of crap" is upvoted to the sky no matter how bad it is, so it can't easily be pushed out of the way where that's needed.
I could go on, but I already feel my blood pressure rising. The thing is, the underlying topic is really not this complicated and doesn't require this huge pile of Q&A to address everything. Although Python's import system is very flexible and customizable, the default workings are really just not that hard to get right. It's just that the good advice drowns in the bad, and the bad isn't even well organized - and people are eager to reach for heavy-duty tools in cases where the simple ones work just fine, just because of a lack of understanding. And because of this, we get to stare at a huge mass of people complaining about complexity and pain and annoyance that isn't really there.
My conclusion is that the primary root of the problem is the sourcing of questions. People ask questions about import
because they have a specific problem they are trying to solve, and it's usually a "something went wrong" problem rather than a "I would like to do XYZ" problem. As a result, nobody is thinking about cleaving the problem space at its joints.
If there were a proper set of canonical questions, asked and titled with foresight and deliberation, then it would be much clearer what answers go where. Dupe-hammerers would easily be able to identify the right targets (because their subject matter expertise allows them to recognize the problem quickly).
After that, closing and dupe-hammering old questions would do wonders in terms of directing people to good information that actually works and away from comment sections inviting them to join a pitchfork mob against the Python dev team.
Aside from gaining consensus on this point, I would like to hear about:
- suggestions for what such a canonical set might include
- any other specific complaints you have about extant canonicals
- any especially good Q&A about Python's import system that you feel goes unrecognized.