-17

While asking a question, I just noticed that when using non-ASCII characters it a post title each character is counted as multiple characters, rather than a single one, towards the post title limit.

For example, when this post used the Unicode title of "โ„ญ๐”ฌ๐”ฒ๐”ซ๐”ฑ ๐“พ๐“ท๐“ฒ๐“ฌ๐“ธ๐“ญ๐“ฎ ๐•”๐•™๐•’๐•ฃ๐•’๐•”๐•ฅ๐•–๐•ฃ๐•ค ๐’ถ๐“ˆ ๏ฝ“๏ฝ‰๏ฝŽ๏ฝ‡๏ฝŒ๏ฝ… ๐Ÿ…ฒ๐Ÿ…ท๐Ÿ…ฐ๐Ÿ†๐Ÿ…ฐ๐Ÿ…ฒ๐Ÿ†ƒ๐Ÿ…ด๐Ÿ†๐Ÿ†‚", the title was counted as 79 characters (which I determined by seeing how many letter As can fit after it), despite being made up of only 46 glyphs (including whitespace and .).

I am suggesting that a single glyph or whitespace should contribute only +1 towards the post title limit. This is especially useful when using monospace in post titles when referring to code, like in the aforementioned question.

11
  • 11
    I'd expect the character limit is due to the space required to store it. As such characters that comprise of codepoints that doesn't fit into 16 bits should count as two, no?
    – Scratte
    Commented Aug 14, 2021 at 16:04
  • 2
    I wouldn't expect the title character limit to be caused by space requirements, as question bodies surely take much more space on the servers. On the other hand, this could be a limitation of the implementation, if it stores question titles in fixed-size buffers, which could require them to have a byte-length limit.
    – janekb04
    Commented Aug 14, 2021 at 16:11
  • 11
    Why use such characters in the title anyway? Your given question (and this current one too) really don't need such weird titles (it simply looks bad and also is bad for people searching)... Commented Aug 14, 2021 at 16:13
  • 7
    I'd say make those characters "cost" even more; to discourage people from using them!!
    – 41686d6564
    Commented Aug 14, 2021 at 16:15
  • 1
    @AbdulAzizBarkat For this question, I used them just to show that they can be used in question titles, as I didn't know if everyone knew that was possible. In reality, the only use of them, I would imagine, would be to use monospace for function names or other symbols, like mentioned in the question.
    – janekb04
    Commented Aug 14, 2021 at 16:17
  • 4
    On an off note, it's considered a pretty rookie mistake to count surrogate pairs as multiple chars in my field of work. That said, pragmatically speaking, this is actually a good thing as there are very few legitimate cases for multi-byte sequences in posts.
    – 0Valt
    Commented Aug 14, 2021 at 16:18
  • 1
    but on that note, please do something to the title, it is horrifying to look at in the inbox :)
    – 0Valt
    Commented Aug 14, 2021 at 16:20
  • 9
    "This is especially useful when using monospace in post titles when referring to code" Why do you think you should use monospace for "code" in the title? Even if you think this is useful, I believe the downsides outweigh the benefit. For example, the link of your question looks like this.
    – 41686d6564
    Commented Aug 14, 2021 at 16:21
  • 2
    Personally, Iโ€™m not opposed to this, but it also seems like an extreme corner case, and probably not worth prioritization relative to other features. That said, I assume @scratte is correct regarding character limits, and this wouldnโ€™t make technical sense. Commented Aug 14, 2021 at 16:21
  • 1
    This title is really nice.
    – Ghost
    Commented Aug 14, 2021 at 16:45
  • 8

1 Answer 1

10

Introduction to Unicode

What you see is a consequence of Unicode UTF-16 encoding of characters - a de facto standard nowadays for frontend scripting (as it is used by JavaScript which is among the most popular languages out there). "16" in UTF means the number of bits comprising one code point (which, of course, determines how many characters can be represented by a single code point).

One of the first encoding schemes has been US-ASCII, which used 7-bit code points, subsequently being able to represent 2^7 (or 128) characters. This was later extended to use 8-bit code points, increasing the range to 2^8 (or 256) characters. Obviously, that wasn't enough, but this is as far as a single octet (simply put, 8 bits) can go.

This is how Unicode came to be. Despite not being limited to 16 bits (there are UTF-8 and UTF-32), the UTF-16 version that uses 2 octets, is the most popular one so far, so let's assume it is the only one existing for simplicity.

Unicode consists of 17 character planes, each having 2^16 code points to represent a vast number of characters (65 536 * 17 = 1 114 112). The first one is called the Basic Multilingual Plane and is mostly unremarkable. It is the other 16 that are interesting because code points there are comprised of surrogate pairs.

While the nature of surrogate pairs is out of the scope of the answer, it is the number of bits used to encode them is that matters. As you might've guessed, it is 32 bits (or 4 octets). Now, given the UTF-16 encoding, how many code units will that be for one code point? Exactly, 2 units.

What does all this have to do with the character counter?

A lot of the programming languages, including JavaScript which is the frontend language for the network, use UTF-16 encoding for strings. What's more, JavaScript is historically bad at handling surrogate pairs. Note what the length property tells us (do not repeat the following at home):

const emoji = "๐Ÿ˜’";

console.log(emoji.length); //2??

for (const char of emoji) {
  console.log(char); //๐Ÿ˜’
}

emoji.split("").forEach((chr) => {
  console.log(chr); //๏ฟฝ x2 - broken surrogate pair
});

And here you have it, the length is 2 chars. It is usually referred to as a naive implementation, and developers are advised to avoid it, but, apparently, not on SE (unless this is an elaborate way of discouraging emojis, in which case - carry on!).

Take a look at this line from the source file which contains the logic for the character counter, and see if you can notice the problem (minified version I prettified a little):

c = t.ignoreWhitespace ? 
l.replace(/\s+/g, " ").replace(/^\s+/, "").replace(/\s+$/, "").length : 
l.replace(/\r\n/g, "\n").replace(/\n/g, "\r\n").length

Unminified version, courtesy of Makyen (link to file):

if (options.ignoreWhitespace) {
  cur = val.replace(/\s+/g, " ").replace(/^\s+/, "").replace(/\s+$/, "").length;
} else {
  cur = val.replace(/\r\n/g, "\n").replace(/\n/g, "\r\n").length;
}

Below is a (not so) simplistic implementation of how surrogate pair-aware input might be implemented (that is also able to count complex emojis correctly). Also available as a userscript actually replacing the native character counter on Stack Apps.

((w, d) => {

  const safeStringLength = (text) => {
    let count = 0;

    const ZWJ = 8205;
    const variationSelectorMatch = /[\ufe00-\ufe0F]/;

    let skipNextChar = false;

    for (const char of text) {
      if (skipNextChar) {
        skipNextChar = false;
        continue;
      }

      if (variationSelectorMatch.test(char)) continue;

      const code = char.codePointAt(0);

      if (code === ZWJ) {
        skipNextChar = true;
        continue;
      };

      count += 1;
    }
    return count;
  };

  /** thresolhold at which to start showing the warning */
  const showAt = 5;

  const input = d.getElementById("title");
  const counter = d.getElementById("counter");
  if (!input || !counter) return console.debug("missing elements");

  const max = +input.maxLength;

  input.addEventListener("input", () => {
    const chars = safeStringLength(input.value);
    if (chars < showAt) return counter.textContent = "";
    const left = max - chars;
    counter.textContent = `${left} character${left > 1 ? "s" : ""} left`;
  });

})(window, document);
label {
  display: block;
  margin-bottom: 1vh;
}

input {
  padding: 1vh 1vw;
  min-width: 40vw;
}
<label for="title">Title</label>
<input id="title" name="title" type="text" maxlength="150" tabindex="80" value="" placeholder="What is a surrogate pair?">

<p id="counter"></p>

14
  • 2
    While this is accurate information regarding how languages, particularly JavaScript, handle strings and Unicode/UTF-16, what SE is doing in their JavaScript code for counting space available in the title may be fully correct, including that it counts some "characters" as more expensive. The actual space available depends on the underlying definition for that field in SE's database. Thus, the way SE is counting the remaining available "characters" may be fully correct (assuming each remaining character fits in 16 bits).
    – Makyen Mod
    Commented Aug 14, 2021 at 17:35
  • 2
    Showing users a remaining character count, e.g. "50 characters left", is better than saying "You have 800 bits remaining; use them wisely." :)
    – Makyen Mod
    Commented Aug 14, 2021 at 17:35
  • 3
    Personally, I think "You have 800 bits remaining; use them wisely." would be awesome :)
    – Scratte
    Commented Aug 14, 2021 at 17:40
  • 2
    @Scratte I agree. However, it wouldn't really communicate effectively to a substantial percentage of users. Given that the amount of space actually required for a character is variable, there's not a lot that can really be done about informing users how much space is remaining while keeping what's said both short and easily understood, particularly when considering that a significant percentage of users don't really have a clue about character encodings and aren't expected to.
    – Makyen Mod
    Commented Aug 14, 2021 at 17:45
  • 1
    @Makyen I agree that their counting may be correct, but I'd prefer SE not expecting all users of the network (as the component is definitely network-wide) to be familiar with the intricacies of character encoding :) A lower max count could be set for strings containing surrogate pairs, or, better yet, the max char count could be determined dynamically instead of being hardcoded depending on the number of characters from supplementary plains.
    – 0Valt
    Commented Aug 14, 2021 at 17:45
  • 2
    @OlegValter I'm fairly "meh" on that. Basically, what you're asking the code to do is guess as to what the user's future input will be, which will result in the "remaining" count fluctuating both up and down (i.e. confusing), depending on the additional input. I don't see a solution here which isn't confusing under some conditions, without giving users a crash course in character encodings. The current solution, assuming it's accurately reporting the actual remaining space (with "character" = 16 bits), is simple and provides reasonable information under the vast majority of conditions.
    – Makyen Mod
    Commented Aug 14, 2021 at 17:54
  • 2
    @Makyen yeah, the dynamic part might be problematic to achieve, I agree that fluctuating limits would be confusing. However, with the current value set to 150 (if I recall correctly), and that the Title column is of type nvarchar(250) with SQL Server with UTF-16 treating surrogate pairs as 2 code units as well, we can safely assume the mapping is 1 to 1 (with 100 chars of buffer), so I don't see any problem in making the character count more accurate for everyone instead of seemingly jumping by one for chars from supplementary plains.
    – 0Valt
    Commented Aug 14, 2021 at 18:02
  • 1
    on nvarchar in t-SQL - sorry, the character limit was too low for the link, rene's query on the posts table structure
    – 0Valt
    Commented Aug 14, 2021 at 18:03
  • 1
    I'd far rather see a perfectly accurate "remaining" value when trivially converted by me to bits, allowing me to gauge how much space will be consumed by the characters I intend to add (possibly of different lengths), than to provide a "remaining" value which is variable based on what it assumes my future input will be. SE just needs to decide what actual limits they are putting on that input. If the actual limit is title.length, then it should be reported as it is. If the limit they are imposing is something else, then something more complicated could be used.
    – Makyen Mod
    Commented Aug 14, 2021 at 18:23
  • 2
    BTW: unminified versions of SE's code are available. For example: full.en.js. As might be expected, there are a substantial number of different files (at least 45+).
    – Makyen Mod
    Commented Aug 14, 2021 at 18:26
  • 1
    @Makyen as far as my investigation went, they just use length for validation purposes (and which is the culprit for the incorrect number of characters shown). What I meant by my last message is that how the value is stored in their database is identical to how it is treated in the front-end. The title field should accept from 250 single code unit chars to 125 surrogate pairs. Given that the limit on title length is already set to 150 normal chars (75 surrogate pairs), they do not risk anything by making at least the static count report valid values.
    – 0Valt
    Commented Aug 14, 2021 at 18:33
  • 1
    @Makyen on the note of the dev link - thank you, always slips my mind :)
    – 0Valt
    Commented Aug 14, 2021 at 18:37
  • 2
    Re "the UTF-16 version that uses 2 octets, is the most popular one so far: I thought UTF-8 was the most popular. For instance, strange characters usually turns up as UTF-8 byte sequences (some are also CE/CP-1250). Creating a new file in Visual Studio Code defaults to UTF-8. Commented Aug 15, 2021 at 9:45
  • @PeterMortensen ah, that's a carryover from my original draft - the intention was to focus on programming, and as far as I recall, JavaScript (still one of the most popular high-level languages), Java, .NET, and the Windows API all do use UTF-16. That said, I think Python switched from UTF-16, plus UTF-8 is the Linux choice. Probably should revise a little to make the context clearer, thanks!
    – 0Valt
    Commented Aug 15, 2021 at 9:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .