121

A current problem

I recently came across this question on Stack Overflow where the OP was convinced that Java's String class was broken. They were comparing two string literals which looked 100% identical to the human eye, however, one of the strings had a zero-width space character at the beginning. This meant that the lengths of the strings varied by 1 in an extremely confusing manner.

Even when going into the Edit Question section, the character is completely invisible (which makes sense, since it's a control character in a raw textarea). The only way for me to see this character, is by copying the code snippet and pasting it into my own text editor, where it is nicely highlighted in red.

Screenshot from my text editor:

Image of zero-width-space character highlighted in red

Potential pranks

Since this character is currently completely invisible, people could write some nasty pranks that convince people a language is completely broken.

For example, consider this JavaScript snippet:

// Strings in javascript are broken!!!
const s = "​​​​hi";
console.log('The length of "%s" is: %d', s, s.length);
// The length is 6?! Javascript must suck!

It looks pretty convincing, right?

This kind of thing could have people scratching their heads for hours. Whereas, if the characters were highlighted, it would be instantly noticeable.

Summary

I think that highlighting zero-width space characters in red (or any other colour, for that matter) would be a useful addition to Stack Overflow.

It's not a huge problem, but it could save a lot of time and confusion.

What do you think?

9
  • 7
    I could see the red dots in the snippet editor is that's anything. Jul 7, 2017 at 1:17
  • 22
    @ChrisHappy I can't. Perhaps it's browser-specific.
    – byxor
    Jul 7, 2017 at 1:55
  • At least it could be shown for defined strings : it would show the dot for everything "That's in there" Jul 7, 2017 at 12:57
  • 7
    You can also use Unicode confusables for the same purpose.
    – user6560716
    Jul 7, 2017 at 13:36
  • 15
    There have been many SO questions of this sort, more often due to byte order mark characters (\ufeff). The ability to show these would be useful, but I would make it a toggle, as it’s possible some monospaced blocks (like program output) might actually want the true nonprinting behavior.
    – VGR
    Jul 7, 2017 at 13:39
  • 7
    Reminds me of the time, many years ago, when a classmate had the empty file ".<Ctrl-G>" added to his (UNIX) home directory after leaving his terminal unattended for a few minutes. Whenever he listed his home directory the keyboard would BEEP, until finally after a few days he realized that he had been pranked. The leading period, of course, made the file completely invisible to a vanilla ls command. Jul 7, 2017 at 17:27
  • 3
    A potential problem; people unaware of these characters may remove it from their stackoverflow post, thinking that the editor was broken and inserting weird characters into their text. Then it would be completely impossible for answerers to diagnose the issue. Unless the characters were hidden in the edit box, and only visible after posting.
    – Rob Mod
    Jul 8, 2017 at 9:17
  • 1
    @Rob - or maybe they would see the problem and decide not to ask a question that is inarguably pretty useless because it is so extremely localized.
    – user177800
    Jul 8, 2017 at 22:58
  • Here is an attempt (second to last line) to use HTML &zwnj; (zero-width non-joiner - effectively Unicode U+200C) to get only part of a word formatted in italics (unnecessary in that case). Revision 1. I far as I know, such HTML tags are ignored. Nov 14, 2021 at 15:57

4 Answers 4

25

This is just a rite of passage.

With BOMs, zero-width spaces, combining characters, confusables, etc... the best thing you can do is teach people to inspect their strings so they're equipped for this type of issue in the future:

This string has some zero-width spaces:

<script>                                                                                     </script><script src="https://gh-canon.github.io/stack-snippet-console/console.min.js?201805111946"></script><style>.as-console-wrapper { display: block !important; }</style><script>console.config({timeStamps:false,maximize:true});</script><script>
  console.table([..."​​​​hi"].map(char => ({code: char.charCodeAt(0), char})));
</script>

This string has a combining tilde:

<script>                                                                                     </script><script src="https://gh-canon.github.io/stack-snippet-console/console.min.js?201805111946"></script><style>.as-console-wrapper { display: block !important; }</style><script>console.config({timeStamps:false,maximize:true});</script><script>
  console.table([..."mañana"].map(char => ({code: char.charCodeAt(0), char})));
</script>

This one doesn't:

<script>                                                                                     </script><script src="https://gh-canon.github.io/stack-snippet-console/console.min.js?201805111946"></script><style>.as-console-wrapper { display: block !important; }</style><script>console.config({timeStamps:false,maximize:true});</script><script>
  console.table([..."mañana"].map(char => ({code: char.charCodeAt(0), char})));
</script>

Some characters just look really similar (or identical)...

<script>                                                                                     </script><script src="https://gh-canon.github.io/stack-snippet-console/console.min.js?201805111946"></script><style>.as-console-wrapper { display: block !important; }</style><script>console.config({timeStamps:false,maximize:true});</script><script>
  console.table([..."oοо"].map(char => ({code: char.charCodeAt(0), char})));
</script>

If you can come up with an elegant, non-intrusive way to visualize all of those cases, that I can optionally toggle on, then I'd support it.


As for your red dots...

CodeMirror, utilized by the Stack-Snippets™ editor, uses (char code 8226) to visualize the zero-width space.

red dot zero-width character replacement in snippet editor

Those characters are tagged with the cm-invalidchar CodeMirror CSS class. If we're stuck with materializing these spaces, we could at least change the syntax highlighting to more readily differentiate them from the other literal characters.

alternate syntax highlighting for cm-invalidchar

Personally, I'm not a fan of an editor injecting characters into my literals which literally aren't there. Strings have enough ways to deceive you. This method also fails to address any of the other issues which present in the same fashion.

11
  • 7
    I suppose I did learn quite a lot from that question. It has taught me a bit more about the importance of inspecting things more programmaticly when they appear to be okay. Still, time could be saved.
    – byxor
    Jul 7, 2017 at 13:45
  • 1
    Which browser is it that you're seeing the red dot on?
    – Cody Gray Mod
    Jul 7, 2017 at 15:49
  • @canon is this in the console, snippets console, or snippet result? Or all?
    – CalvT
    Jul 7, 2017 at 16:06
  • 1
    Yup I'm seeing them now, thanks! Chrome 59.0.3071.115, Microsoft Edge 15.16232 and FireFox 54.0
    – CalvT
    Jul 7, 2017 at 16:29
  • @canon: A lot of things relating to fonts and text encoding is browser specific. Jul 7, 2017 at 16:47
  • @canon: Alright. Jul 7, 2017 at 17:13
  • 5
    Oh. Yes, I was confused about where this red dot was being seen. I see it in Safari in the snippet editor, too. My confusion probably stemmed from the fact that I never use the snippet editor because it is irrelevant to all languages that I program in. It took me a hot moment to even find how to invoke it here!
    – Cody Gray Mod
    Jul 8, 2017 at 1:17
  • For people like me and Cody Gray: Either Copy snippet to answer using the button below the snippet or edit an answer with the snippet, then use edit the above snippet link below the preview of the snippet.
    – muru
    Jul 10, 2017 at 3:51
  • 1
    +1 for for perfectly imperfect hand drown circles.
    – mx0
    Jul 10, 2017 at 20:36
  • 2
    Re "inspect their strings": The most common ones (coming from copy-pasting through chat (e.g. Skype Chat - experienced first hand...), web pages (also experienced first hand...)), PDF documents, etc.) can be searched for directly (and optionally replaced, even ZERO WIDTH SPACE) using a reasonably modern text editor (in regular expression mode): \x{00A0}|\x{200B}|\x{200C}|\x{2013}|\x{2014}|\x{201C}|\x{201D}|\x{2212}|\x{00E4}|\x{FFFD}|\x{2217}. Nov 6, 2021 at 14:12
  • 2
    That is for NO-BREAK SPACE, ZERO WIDTH SPACE, ZERO WIDTH NON-JOINER, EN DASH, EM DASH, LEFT DOUBLE QUOTATION MARK, RIGHT DOUBLE QUOTATION MARK, MINUS SIGN, LATIN SMALL LETTER A WITH DIAERESIS, REPLACEMENT CHARACTER, and ASTERISK OPERATOR, respectively. More exotic ones are in this Stack Overflow answer. Nov 6, 2021 at 14:13
8

A zero-width space is, well, zero-width. It's not supposed to be visible, so let's not make it so.

If someone has a question like this then it's a good opportunity for them to learn about this class of programming problem, and the invisible way in which it often manifests.

8
  • 4
    Agreed. If this really comes up as often as this Meta question seems to imply, perhaps it would be worth creating a canonical Q&A that explains the issue. I certainly can't see the benefit of repeating this basic lesson over and over across multiple questions and languages.
    – Cody Gray Mod
    Jul 8, 2017 at 1:18
  • @CodyGray: Yeah perhaps Jul 8, 2017 at 1:32
  • 2
    A monospace-font environment is supposed to assign every character the, well, same width. So we have two conflicting requirements there. Or do we?, for IMO the character width, including that of a zero-width space, simply ought not apply at all in a code environment. Jul 8, 2017 at 21:44
  • 1
    @leftaroundabout: That is not quite accurate. There are a great many "characters" that do not take up a full pitch width even with a monospace font. That would completely annihilate the rendering of several widely-used writing systems, if it were true. Jul 9, 2017 at 0:59
  • 1
    @LightnessRacesinOrbit in a monospace font, yes, but to my understanding that's not the same thing as a monospace environment. The purpose of a monospace environment is that there's a clear correspondence from codepoint-index to displayed-character-position; the visual appearance, which is what the monospace font seeks out, is merely a side-effect. Jul 9, 2017 at 10:20
  • 2
    @leftaroundabout: The purpose of a monospace environment is that there's a clear correspondence between printable character and displayed character position. A printable character may be composed of multiple codepoints (e.g. accents, other modifiers; consider Chinese, Arabic!) which introduces the notion that codepoints themselves don't have a fixed width. A zero-width space has zero width, always. Hence the name. Jul 9, 2017 at 12:41
  • 4
    It's not supposed to be visible when rendered to final output. Code and mark up exist explicitly to give a visible representation to output that is not necessarily visible. (That's part of the reason we have escapes for things like tabs, newlines, and Unicode in general.) So that isn't an argument against giving them a visible representation in code and mark up.
    – jpmc26
    Jul 9, 2017 at 20:20
  • @jpmc26: Now that may be the basis of a decent argument :) Jul 10, 2017 at 0:29
5

TL;DR: The idea makes sense, but whether doing it on SE is a good idea is opinion based. If it gets implemented for SE, don't make it togglable.

Is displaying placeholders for invisible text a good idea?

Displaying a placeholder for an invisible character in a formatted code block is a sensible idea. This wouldn't be the first time that a programming tool displays a placeholder for something that isn't supposed to be directly visible.

Many editors have a "Show whitespace" option. This replaces the normal blank space with a symbol, usually separate ones for a space, a tab, and a newline. (Newline is often controlled separately from other whitespace; nonetheless, it's not uncommon to have the option of seeing it explicitly.) The reason for doing so is simple and logical: you can better predict and understand the behavior of tools that consume your file if you can tell exactly what kind of character is there.

Displaying something for normally invisible characters would achieve the same goal: it allows someone to better predict and understand the behavior of a particular piece of text. This is reasonable and useful, and those arguing against it are doing it mostly on the basis of aesthetics, which are secondary to functionality when you're dealing with code. One might argue that invisible characters exist primarily for aesthetics, but that only becomes more important than knowing what characters are there when the code is running or the mark up is being parsed.

What about when you're writing a post?

It would also seem that if code blocks are going to display a placeholder, the post editor probably should, too. We're writing mark up in the editor; it's not rendered output. Additionally, the editor has a previewer, so the rendered output would still be visible.

Whether it's feasible to change the behavior of the browser to do this is a separate question.

Should Stack Exchange do it?

All that said, just because it's a good thing to have doesn't mean Stack Exchange should do it.

Benefits:

  • Makes spotting characters that might cause unexpected behavior simple
  • Makes it easier to tell if you entered what you meant to

Drawbacks:

  • Time to implement (assuming it can't just be grabbed or turned on)
  • Uninformed users might assume Stack Exchange is inserting characters

In the end, I think this is opinion based. There are upsides and downsides, which is why most editors that have a similar feature allow users to turn it on or off.

Personally, I like seeing whitespace in my editor, if for no other reason than it helps me spot tabs vs. spaces. I'd personally like a feature in this vein as well.

What about making it optional?

I don't think making it something you can turn on and off is a good idea. It increases the complexity dramatically, and having it off would drastically reduce the main benefit: helping users spot the invisible character in the first place.

1

tl;dr This is should definitely be fixed. I'd recommend substituting in a character that shows the data in a little box, but alternatively any sort of character representation that's not degenerate works.

Example of recommended display for an instance of the invisible character &#8203;:

+----+
| 82 |
| 03 |
+----+

, scaled down to the size of every other character glyph. The 8203 might not be too visible due to its small size, but it'd still serve to display both the existence and identity of a character. Failing this, any sort of stand-in display would still be an improvement.


From an intentional perspective, the entire point of a source code view is to enable visualization of the source code. It's absurd to have any character invisible by default, even if the rendering of that character is meant to be invisible; this isn't supposed to be WYSIWYG.

From a specifications perspective, coding fonts are supposed to have a constant width per character, e.g. ! and W have the same width. So, characters should be displayed with the same, non-zero width.

Options:

  1. Display invisible characters as spaces, i.e. .

    • Probably not a good idea since it's degenerate and can also lead to confusion.
  2. Display invisible characters with a stand-in character, e.g. .

    • Also degenerate, but less confusing since the stand-in character doesn't resemble any commonly used character.
  3. Display the Unicode for each character in a small box that's the typical size for a character glyph.

    • Best solution, though may require creating a new font or other rendering approach.
3
  • 1
    What is 8826? Unicode code point U+227A (PRECEDES - "≺") (hexadecimal)? Unicode code point U+8826 ("蠦")? Or something else? Nov 6, 2021 at 14:27
  • 1
    @PeterMortensen: Near the end of this answer, the character 8226 (&#8226) is referenced as being a stand-in for an invisible-character. It's weird to think that I'd have made two mistakes (both mutating the number and misrepresenting its meaning), but that may've been where I got it from.
    – Nat
    Nov 8, 2021 at 20:17
  • 1
    (8226 (decimal) = U+2022, "•", BULLET. 8203 (decimal) = U+200B, "", ZERO WIDTH SPACE.) Nov 14, 2021 at 15:44

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .