I received a notification that one of my questions Why does Python print unicode characters when the default encoding is ASCII? has been marked as a possible duplicate of a more recent effort that is meant as a canonical explanation. I've been asked whether the new thread answers my question, and if not to provide my question with more details as to how it is different.
I do not believe the matter to be this clear cut. Although the new answer may be more expansive in its topic coverage (mainly because it mentions python 3) I find that my question and my answer still are superior in their pedagogical attempt to be clear, accessible, and to the point. Granted this is purely subjective, but that latest attempt is not what I would like to stumble upon when looking for an answer to my question. Because of this, I would prefer to keep them separate, they will certainly be linked by StackOverflow's algorithm as being related anyway, all for the better. The goals might be similar, but the methodology to get there is clearly different. I am very much concerned with pedagogy over mere factual completeness, which is why I took pain in providing a summary breakdown of how unicode and encoding work conceptually as part of my answer and omitted a bunch of other details that, in my opinion, distract the reader's understanding of the framed behavior. I also took care not to write a "wikipedia" style article that simply dumps a bunch of trees in the reader's lap and leaves it as an assignment to find the forest. I believe that the upvotes and subsequent comments congratulating this attempt speak for themselves as to the usefulness of that contribution in people's effort to capture the potentially eluding topic.
Being the "canonical" answer to anything is an ambitious project especially considering that having seen other canonical answers, they're never enough for everybody. It's not a phenomenon unique to Stackoverflow. Even canonical books have sometimes been found to be lacking by some readers, who will later find the light from a youtube video that hits the nail for them in under two minutes.
I saw that there's a project to close down a bunch of these questions in the name of "unifying" them under this "one true answer". Before dismissing all these as mere duplicate of the new thing, give it some time to confirm that it's actually also hitting the right spots. This is especially true of highly upvoted and bookmarked questions.
Although "canonical answers" are nice, I believe that there's a place in StackOverflow for pointed questions and answers especially when they are well framed and explained. After all the goal should be quality of information, not just expansiveness of the coverage.
As for my question and answer, I'll take some time to update them and further frame them as being specific to Python 2.x. I second the suggestion to do the same for the new question and place it specifically under Python 3 only. For one, the community is increasingly migrating toward it and soon fewer people will encounter the Python 2 behavior. Also, it makes it a clearer, less distracting answer.
encode
anstr
, it must bedecode
d intounicode
first. Python 2 tries to do that implicitly (<effective code>
), with the result that you see. The right way is to always use national characters in Unicode strings, avoiding this problem altogether. Otherwise, you need todecode
the string yourself first, with the right encoding which, per (reference), is the source file's encoding."