2

I've just tried to replace a screenshot with actual code in this question.

The OP's first string includes a byte order mark, i.e. it starts with U+FEFF. Inserting this character in the editor works and the preview contains the character, too. But once submitted, the BOM is stripped from the string.

"<- there should be a U+FEFF"
0

1 Answer 1

3

The question is more: Why should it remain? I'd say that such characters serve no purpose in questions and should be removed instead, which already happens. This would make your bug report .

U+FEFF is a zero-width non-breaking space character when not doubling up as a byte-order mark when used as the first character of a document. A question body is not a whole document, and neither is a code block within a question.

Zero-width non-breaking spaces are not exactly visible, but can muck up the layout of a post, so removing them from posts has a purpose. It also removes the possibility to use the character to try and circumvent the minimum character requirements for a post.

Stack Exchange doesn't and cannot represent all data in literal form; tabs are expanded to spaces, newlines are normalised and HTML is escaped unless marked up as code, for example. As such, you'll always have to find ways to represent unprintable characters in other ways. Programming languages have various ways to represent non-printable characters, which would make such problems actually visible; in Python I'd use the repr() function to produce a Python string literal for example:

>>> print repr(u"\ufeff")
u'\ufeff'

It is no coincidence that the output looks almost the same as my input to the repr() function there.

You should find a similar representation for the Ruby question; how would you add a U+FEFF charater to a Ruby string literal, for example?

8
  • 1
    The OPs problem is caused by this character. Stripping characters from code doesn't allow to show / post such code so others can't reproduce it. IMO, code should not be altered.
    – Stefan
    Nov 10, 2014 at 13:24
  • 1
    @Stefan: there are loads of problems that can be caused by unprintable characters, tabs being the foremost one. But that doesn't mean that the characters themselves should be preserved as is. The site escapes HTML (for very good reason) unless you mark it up as code. If a U+FEFF is the root cause, then chose a representation that isn't literal. I use the Unicode codepoint notation to talk about that character for example.
    – Martijn Pieters Mod
    Nov 10, 2014 at 13:27
  • 1
    Yes there are. But not being able to post these characters makes it even harder to discuss such problems. This is why the OP posted a meaningless screenshot - there was no way to enter (or copy/paste) the actual code.
    – Stefan
    Nov 10, 2014 at 13:34
  • @Stefan: Note that the U+FEFF was part of the data there, not the code. So how would you propose Stack Exchange handles line feeds, form feeds and field separators? Or \x00 null characters? It is not as if these are visible when included in literal form.
    – Martijn Pieters Mod
    Nov 10, 2014 at 13:36
  • 1
    @Stefan: the screenshot failed to communicate the presence of the character just as much. Instead, the OP should be taught some basic data debugging skills. Check the length of the string, create a hex representation, etc.
    – Martijn Pieters Mod
    Nov 10, 2014 at 13:37
  • 1
    "So how would you propose Stack Exchange handles [...] characters?" - I'm not sure if there are technical limitations, but from a user's perspective, there should not be a difference between the preview and the final question. Entered characters should be stripped when generating the preview or not at all.
    – Stefan
    Nov 10, 2014 at 14:14
  • @Stefan: Meh, the character doesn't print anything; I'm more than fine with the stripping happening server-side here. And it is not as if the OP realised that there was an extra character there in the first place.
    – Martijn Pieters Mod
    Nov 10, 2014 at 14:20
  • I see. Somehow I was under the impression that the preview is generated on the server-side, too. Although having non-printable characters within string literals is clearly an edge case, it would be nice if text processing / stripping / sanitizing would be consistent on both sides.
    – Stefan
    Nov 10, 2014 at 14:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .