13

Bug

I answered a question recently on CodeGolf.SE and my answer included the character U+0007 in my code; however, Stack Exchange (including Stack Overflow) removed certain unicode characters from code blocks which can cause the code to stop working. Here is a link to my answer: https://codegolf.stackexchange.com/a/108573/16513

It is a very common practice for sites to remove zero-width unicode characters. Unicode alone has like 30 different characters for "space" I think and many sites like Facebook just strip them all out but the common space. I'm all for this, but I don't think it should be stripped from code blocks.

It is needed in these.

Removing characters from code blocks can produce buggy code when users copy paste them. In my referenced answer above I had to include a link to pastebin where the U+0007 character was not removed. I'd imagine most characters from U+0000 - U+0020 are auto-removed from posts (including inside codeblocks).

8
  • 7
    Somewhat relevant: Why does code formatting not work in this question? (formatting issue caused by a zero-width character).
    – duplode
    Commented Jan 30, 2017 at 4:18
  • @duplode good point, it's an interesting problem when sometimes the code should be corrected and sometimes it should not. Commented Jan 30, 2017 at 4:39
  • Surely you can escape these characters and don't have to include them literally? Commented Jan 31, 2017 at 8:44
  • 4
    @MartinSchröder Not in CodeGolf Commented Jan 31, 2017 at 8:47
  • 1
    Note that in code golf if the code contains unprintable characters a hex dump/code dump/whatever you want to call it, is usually provided Commented Jan 31, 2017 at 9:09
  • Instead of using an external source you could simply provide something like the base64 (or hexadecimal) encoded version of the program. This is already customary for answers in, say, machine code.
    – Bakuriu
    Commented Feb 1, 2017 at 18:48
  • I’ve also seen people use a control picture instead of a control character, then provide a hexdump with the correct code.
    – Jed Fox
    Commented Feb 1, 2017 at 19:06
  • 1
    This can also make it very difficult for users to post an MCVE, for example stackoverflow.com/q/36431437 Commented Feb 1, 2017 at 19:26

3 Answers 3

9

Yes, we filter some unprintable control characters, and I doubt this is something that we're likely to relax without a lot more discussion. In the case of code golf, I would recommend using some form of token to represent the unprintable character (so \7 for BEL, for example - or U+0007 as you used in the question). I don't think we're going to support answers in native "whitespace"

1
  • Have there been any further discussions on this? Commented Nov 1, 2018 at 19:48
4

Certain blocks of Unicode are removed at most SE sites – some of these, per decree, because lots of spam uses Chinese characters. (I darenot say whether this is because it is generally believed by spammers to circumvent spam filters, or whether this has been identified as a main source of spam itself.)

There are reasonable exceptions, though: imagine such a ban on Chinese Language (beta).

One could propose a similar exception for Programming Puzzles & Code , as there are a few exotic programming languages that deliberately make use of Unicode characters.

However.

The specific range of characters you ask about is the Control Character Block C0:

In addition to the special characters defined in the Unicode Standard for a number of purposes, the standard incorporates the legacy control codes for compatibility with the ISO/ IEC 2022 framework, ASCII, and the various protocols that make use of control codes. Rather than simply being defined as byte values, however, the legacy control codes are assigned to Unicode code points: U+0000..U+001F, U+007F..U+009F.
(The Unicode® Standard, Version 9.0 – Core Specification, p. 68)

and

The Unicode Standard provides for the intact interchange of these code points, neither adding to nor subtracting from their semantics. The semantics of the control codes are generally determined by the application with which they are used.
(ibid., p.822)

That means that, for a web interface, the exact behavior of inserting these characters is determined by the local user interface: your browser. Browsers in general can handle only a very few control characters (the two return types and tab are the only ones I can think of), and, even more, they usually do so through the intermediary form of a HTML page, which has even more restrictions.

Simply put: there is no valid way to insert these essentially non-HTML compatible characters in a compatible way. It is not a limitation of code blocks either; code blocks are formatted by SE's local version of Markdown to regular HTML.

3
  • 9
    "That means that, for a web interface, the exact behavior of inserting these characters is determined by the local user interface: your browser." ——— If this part were entirely true then why does the pastebin I posted work for including this character, but the SE answer does not. I'm using the same browser in both instances. Commented Jan 30, 2017 at 10:05
  • @AlbertRenshaw: fair point (and it works on my system as well, although I wonder how cross-browser compatible it actually is). Still: on SE you type everything in Markdown, so it always gets postprocessed.
    – Jongware
    Commented Jan 30, 2017 at 10:24
  • 2
    @AlbertRenshaw, presumably because those characters are removed by the SE site, but not by pastebin. They are removed because they may not display correctly on some browsers. They display correctly for you, great, but they might not for someone else using a different browser. Consistency is key here, I suppose.
    – SiHa
    Commented Jan 31, 2017 at 8:31
1

Filtering UNICODE Letters from Code destroys colors in Windows Batchfiles, because the \U+001b Letter is removed. Because of removing this Letter, the command sequence is printed to the user, instead of being processed.

Example with \U+001b Example with correct Code

Example without \U+001b Example with filtered Code

Sourcecode of Example with \U+001b letter: https://gist.github.com/mlocati/fdabcaeb8071d5c75a2d51712db24011

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .