46

There is a steady stream of these questions. To protect the innocent, I am paraphrasing:

Halp, my program crashes because it cannot decode byte 0xFF at offset 9876543210.

I will not reveal what type of data this file contains, or what encoding or contents I expect.

In the past, I have tried to explain in comments how this is not sufficient to solve the OP's problem, and (sometimes repeatedly) asked for details and clarifications; but copy/pasting comments is frustrating and often ineffective.

Instead, I am hoping to collect in this meta post some guidance which we can link to from a single comment to help people who have this type of problem; specifically, to help them edit their post into a well-defined question that we can actually help them resolve, or which perhaps they can even resolve without our help.

So, my question is simple: What information should we ask the OP to provide in order to be able to help them solve their issue in cases like this?

6
  • How should I tag this? Suggestions welcome. I graciously abstained from using help-vampire here.
    – tripleee
    Jan 25, 2019 at 8:43
  • 3
    Added some tags. Don't worry about the dv, people will often vote without explanation and that's ok. :) Nice post btw
    – user3956566
    Jan 25, 2019 at 13:14
  • 1
    Thanks. I'm not worrying about downvotes on meta, just being vaguely amused.
    – tripleee
    Jan 25, 2019 at 13:21
  • 3
    Why is simply asking for a minimal, complete, and verifiable example not already covering these questions about decoding errors?
    – Trilarion
    Jan 27, 2019 at 14:53
  • @Trilarion If I understand your question correctly, my experience is that the OP often doesn't understand what exactly would constitute an acceptable and useful MCVE; hence, this post.
    – tripleee
    Jan 28, 2019 at 5:20
  • I see. This answer also details what an MCVE for this type of questions would be. Initially it looked to me more like a troubleshooting guide (which is not bad by itself).
    – Trilarion
    Jan 28, 2019 at 8:29

2 Answers 2

37

Just like your program, readers of your question have no way to guess what this byte is supposed to represent without additional context.

Unlike your program, however, human readers can help you resolve the problem in many cases, but they will need a fair bit of more information.

If all of the following is bewildering to you, today is the day when you should finally take the time to read Joel Spolsky's classic exposition The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)

The following collects some hints about details which are useful to provide if you can.

  • What is the encoding you hope to end up working with?

    • In other words, how is your system set up to display and manipulate text in a way which is suitable to your culture and locale?
      • On Windows, what does chcp display?
      • On Unix-like systems, what's the output of locale (in particular LANG and LC_ALL)?
    • Or in more sophisticated cases, does your program (or your OS or programming language) need text to be represented in a particular encoding?
  • What is the encoding of the data you are trying to read?

    • If it is text, there is a large number of different ways to represent text, most of which are not strictly ASCII, but rather a superset or variant -- be as specific as you can.

      In more detail, ASCII proper can only be used if there are no characters whose byte value is larger than 127 (hex 0x7f). It is only suitable for (a subset of) English, without "typographer's quotes" or many other characters which today we take for granted. If you think you have "ASCII" but it doesn't obey this constraint, it's not really ASCII. Perhaps you can find out what exactly it is, or at the very least help us help you figure it out by providing some samples?

      "ANSI" in this context is not well-defined, though it will reveal that you are using Windows, and perhaps do not know what your locale settings are. In many Western locales, this means you are using Windows code page 1252, but in other locales, the default will be different, or perhaps you are not using the default code page. In Windows 10, you can finally configure the system to use code page 65001, which is convenient if you want to process text in UTF-8 encoding.

    • Many higher-level formats like XML, JSON, etc have an encoding (typically UTF-8) but also additional markup which has specific semantics. If you are processing something which is supposed to be JSON (or CSV, or FASTA, or binary formats like pcap or MP3, or whatever), make sure you mention this.

  • Can you provide a small unambiguous sample with enough context?

    • In many cases, just a few bytes around the problematic spot are enough to at least guess what's supposed to be there.
    • Copy/pasting text in an unknown encoding is unlikely to be helpful. Use a well-defined, unambiguous representation. A hex dump of the individual bytes in the file or input stream is understandable to anyone with basic familiarity with computers and programming.

    Here are a few examples to hopefully help you see this in concrete terms.

    (Make sure your display is wide enough. There is text to the right of the hex.)

    24CB016E0: 7472 6164 75ff f76f 2065 6d20 706f 7274  tradu..o em port
    24CB016F0: 7567 7589 73                             ugu.s
    

    We still don't know what encoding this is, but somebody who speaks the language (or is good at googling) can guess that this is text in the Portuguese language, and that the correct rendering should be tradução em português. With enough broken samples, we can reconstruct enough of the encoding to fix all your sample data (assume 0xFF represents ç everywhere in the file, etc), even if we might never find out which actual encoding this was in.

    24CB016E0: 6e75 6c6c 2c20 2276 616c 7565 223a 2031  null, "value": 1
    24CB016F0: 3837 2c20 2261 73FF FFFF FFFF FFFF FFFF  87, "as.........
    

    This looks like perfectly fine JSON data up until the corruption, and then clearly complete garbage after that. Maybe you can truncate the file to discard the garbage (taking care to reconstruct the required close brackets to make it valid JSON again), or see if you can remove it from the middle if there is good data again somewhere later in the file; or even better yet, obtain an uncorrupted copy of this file from wherever you got it in the first place.

    24CB016E0: c383 c2a5 652c 67                        ....e,g.
    

    This is an example of mojibake: text was converted from one encoding to another on the mistaken assumption that it was in a third encoding, or perhaps mistakenly "converted" a second time when it had in fact already been converted. In this particular case, it happened multiple times, and it's probably almost impossible to reconstruct even if you know that the input was the Japanese text 日本

    There's also an honorable mention for "we wanted to convert between encodings, but we made our own tool based on incorrect assumptions, and basically simply corrupted the data." The SEC Edgar data infamously has this problem, though the corruption is by and large recoverable if you know how.

    (These hex dumps come from xxd but there are many ways to obtain a hex dump. In many programming languages, it's straightforward to write your own hex dump program if you can't find a tool which works for you. Here's a random one in Perl. If you use Python, probably simply use the standard repr() function to show (a subset of) your data in unambiguous form.)

If you can't provide some of these details, perhaps mention that you don't know; and perhaps even offer a qualified guess even if you can't know for sure. (But make sure you say when you are guessing. Don't state as fact something which might not be.)

The Stack Overflow character-encoding tag info page has an overview of terminology and some guidance similar to what is summarized here.

Eventually, questions which fail to provide enough details should probably be closed as unclear or unreproducible.

10
  • 10
    I'm not sure how useful this guidance is going to be in practice. Someone who makes the mistake of asking a question like the one you exhibit in the question probably can't answer "What is the encoding of the data you are trying to read?" in the first place - heck, they probably don't know what an encoding is - so they're going to be immediately stuck.
    – Mark Amery
    Jan 25, 2019 at 15:15
  • @Mark Thanks for the feedback. The answer should probably be amended with links to resources for more information. Obviously, feel free to edit if you have some.
    – tripleee
    Jan 25, 2019 at 15:19
  • 4
    I saw this article a long, long time ago, I think it might be still relevant as a link for very new developers who are unaware of how these things work: joelonsoftware.com/2003/10/08/…
    – Luan Nico
    Jan 25, 2019 at 15:37
  • 2
    @Luan Thanks. Added to the post. Guilty of assuming everyone has already read that but obviously that is just not true.
    – tripleee
    Jan 25, 2019 at 20:05
  • 1
    IMO, the "What is the encoding of the data you are trying to read?" could stand to distinguish between the actual encoding of the data and the encoding that the reader is assuming/using. These are separate things, and both are often needed. I'd also put Joel's article nearer the top, as it's context that greatly clarifies the issue for the unfamiliar (who are most likely to ask these questions).
    – jpmc26
    Jan 27, 2019 at 15:26
  • @jpmc26 Thanks for the suggestions, I hope I understood your suggestion correctly. If you want to submit an edit suggestion, please feel more than welcome; this is Community Wiki, after all.
    – tripleee
    Jan 28, 2019 at 5:17
  • Why not a question about how to debug "decode byte X at offset M"
    – Braiam
    May 9, 2019 at 14:42
  • @Braiam Do you mean designate that as a canonical for these questions? Or as an answer instead of this answer?
    – tripleee
    May 9, 2019 at 16:23
  • No. And no. As a question you can link people to, when you vote as unclear/lacking mcve for questions that require it.
    – Braiam
    May 9, 2019 at 17:07
  • That's what I mean by a canonical for these questions - a good proposed duplicate target. So any good existing questions on SO main to use for that?
    – tripleee
    May 9, 2019 at 17:12
4

A posted question that is about a problem with character encoding, but does not explicitly declare the character encodings used or expected, indicates that the poster has no understanding of what character encoding is, why it is important, and why it is relevant to their question.

The poster "lacks minimal understanding", as we used to say. The objective of SO is not well served by providing the poster with individual help for their problem. The poster, and all like them, need to be educated in the basics of character encoding and why it is important.

I therefore instead vote-to-close the question as a duplicate of What is character encoding and why should I bother with it, which I lightly edited several years ago to act as a canonical question. I added a link in a comment to Joel's classic essay on the subject

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .