9

There are about 2500 questions combined about UnicodeDecodeError and UnicodeEncodeError.

Virtually all of them are caused by Python doing implicit transcodings -- which (like the Spanish Inquisition) no-one ever expects.

Today, someone asked a question formulated so conveniently that I felt entitled to cover this topic once an for all1: when, where and how Python implicitly applies encodings to strings (and what those encodings are).

Even though they ask about Py2 specifically, I took the liberty to cover Py3 as well while I'm at it.

Now, the answer doesn't give immediate practical solutions on every UnicodeError situation (that would make it completely incomprehensible, and is simply off topic for the question), but can be referred to when giving more specialized answers.


Now, apart from possible input on the content, if any, I'm specifically interested in whether there's some better place for this info that it perhaps could be merged into.

E.g. What encoding do normal python strings use? is the first in Google on "python default string encoding" but its focus is different, to the point that the title is actually non-indicative (its focus is more like: "why don't Python 2 strings are Unicode, and how to fix that?").


1Until the next major release of Python, at least.

4
  • 1
    Python - 'ascii' codec can't decode byte is my choice when dupe-closing those.
    – wim
    Commented Apr 21, 2018 at 20:33
  • @wim That's a good example of a "specialized answer" that I spoke of. Referring to my post for the general picture would make it much more comprehensible: "As per (reference), it's actually impossible to encode an str, it must be decoded into unicode first. Python 2 tries to do that implicitly (<effective code>), with the result that you see. The right way is to always use national characters in Unicode strings, avoiding this problem altogether. Otherwise, you need to decode the string yourself first, with the right encoding which, per (reference), is the source file's encoding." Commented Apr 21, 2018 at 21:22
  • 1
    Why don't you ask a self-answered question about the topic? This answer seems to fit a completely general question very well Commented Apr 22, 2018 at 16:40
  • "Virtually all of them are caused by Python doing implicit transcodings"—then Python 3 isn't relevant at all, because Python 3 doesn't do those implicit transcodings. There are, of course, Unicode errors in Python 3—Windows console errors, cp1252 source files without coding declarations, Linux locale issues, etc.—but none of them are the same as the most common Python 2 issue. (Also, 3.4, 3.6, and 3.7 each eliminate a large class of them, while Python 2's errors will only ever be eliminated by Python 2 dying in a year and a half.)
    – abarnert
    Commented Apr 24, 2018 at 2:21

3 Answers 3

2

I think it's worth having a canonical Python 3 answer.

But I think it should be a separate question and answer from the (existing) Python 2 one, rather than trying to merge both into a single question and answer.

The details are very different, and a whole lot simpler in Python 3. Anyone who has a Python 3 question, unless it's actually about porting from Python 2 or writing dual-version code, shouldn't have to slog through all the Python 2 complexities that just make it harder for them to process the answer.

That's already probably the majority of askers today (despite the fact that Unicode errors don't come up as often in Python 3 in the first place), and that will only be more true in the future.


For example: If a Windows user saves a source file with non-ASCII characters in cp1252 via Notepad, they're going to get an error, even in Python 3. But if half the answer is about implicit transcoding between str and unicode in Python 2—which doesn't even have an analog in Python 3—the answer is just going to confuse them. And anyone not using Python 2 in 2018 is probably never going to use it, so learning enough to get past all that confusion isn't even going to buy them anything.

5
  • I was covering only Py2 initially when writing, and extending to Py3 just happened naturally somehow. I guess it's because there are so many parallels -- explaining them side by side gives a strong visualization of similarities and differences, and demonstrates why things work like they do and why the specific changes were made. Commented Apr 24, 2018 at 2:25
  • @ivan_pozdeev For someone who wants to know the history behind the design decisions in Python 3 (some of which turned out to be slightly wrong and have since been changed…), sure, knowing about Python 2 is important. But for someone trying to debug a problem with Unicode in Python 3, the vast majority of the time, it isn't.
    – abarnert
    Commented Apr 24, 2018 at 2:26
  • Note the "why things work like they do" part. Seeing things in perspective helps to better understand either part. Commented Apr 24, 2018 at 2:32
  • @ivan_pozdeev Yes, that's the history of the "history behind the design decisions". But in this case, the history is so overwhelmingly much more complicated than the current design that an answer that focuses on that is unlikely to be as useful as one that doesn't. It's like answering a question about how to get the last value in a list by explaining the difference between Lisp cons-based lists and Python array-based lists—sure, understanding linked lists is useful, but it gets in the way of finding out about negative indexes.
    – abarnert
    Commented Apr 24, 2018 at 2:40
  • Another point -- which would explain why it "happened naturally". There are important parts common for both (e.g. the bytes vs characters and encoded/decoded). If splitting, I would have to duplicate them word for word. Not only would that make both posts look like plagiarism or cheap repetition (damaging the reception). Critical explanations of fundamental concepts like this mustn't be duplicated whenever possible 'cuz it's just too hard and too important to get them "just right" (invent formulations that are both clear and concise, make it fit natually into the narrative's big picture). Commented Apr 24, 2018 at 18:16
1

I received a notification that one of my questions Why does Python print unicode characters when the default encoding is ASCII? has been marked as a possible duplicate of a more recent effort that is meant as a canonical explanation. I've been asked whether the new thread answers my question, and if not to provide my question with more details as to how it is different.

I do not believe the matter to be this clear cut. Although the new answer may be more expansive in its topic coverage (mainly because it mentions python 3) I find that my question and my answer still are superior in their pedagogical attempt to be clear, accessible, and to the point. Granted this is purely subjective, but that latest attempt is not what I would like to stumble upon when looking for an answer to my question. Because of this, I would prefer to keep them separate, they will certainly be linked by StackOverflow's algorithm as being related anyway, all for the better. The goals might be similar, but the methodology to get there is clearly different. I am very much concerned with pedagogy over mere factual completeness, which is why I took pain in providing a summary breakdown of how unicode and encoding work conceptually as part of my answer and omitted a bunch of other details that, in my opinion, distract the reader's understanding of the framed behavior. I also took care not to write a "wikipedia" style article that simply dumps a bunch of trees in the reader's lap and leaves it as an assignment to find the forest. I believe that the upvotes and subsequent comments congratulating this attempt speak for themselves as to the usefulness of that contribution in people's effort to capture the potentially eluding topic.

Being the "canonical" answer to anything is an ambitious project especially considering that having seen other canonical answers, they're never enough for everybody. It's not a phenomenon unique to Stackoverflow. Even canonical books have sometimes been found to be lacking by some readers, who will later find the light from a youtube video that hits the nail for them in under two minutes.

I saw that there's a project to close down a bunch of these questions in the name of "unifying" them under this "one true answer". Before dismissing all these as mere duplicate of the new thing, give it some time to confirm that it's actually also hitting the right spots. This is especially true of highly upvoted and bookmarked questions.

Although "canonical answers" are nice, I believe that there's a place in StackOverflow for pointed questions and answers especially when they are well framed and explained. After all the goal should be quality of information, not just expansiveness of the coverage.

As for my question and answer, I'll take some time to update them and further frame them as being specific to Python 2.x. I second the suggestion to do the same for the new question and place it specifically under Python 3 only. For one, the community is increasingly migrating toward it and soon fewer people will encounter the Python 2 behavior. Also, it makes it a clearer, less distracting answer.

0

Convenient as it is, Which encoding is used for strings in Python 2.x? prominently and specifically asks about Py2 only -- thus is unusable as a general reference link for both Py2 and Py3.

It also has other, some highly upvoted, answers, specifically about Py2. This means that I cannot expand the original question to cover Py3 as well without invalidating other answers.

So, I followed @Luca's advice and moved the answer to a dedicated Q&A at Python string default/implicit encodings and conversions . Edited the old post to only leave a reference and a concise reciting of the Py2 part.

Since the old post has been edited, everyone who voted are free to retract their votes from it if they wish to.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .