Introduction to Unicode
What you see is a consequence of Unicode UTF-16 encoding of characters - a de facto standard nowadays for frontend scripting (as it is used by JavaScript which is among the most popular languages out there). "16" in UTF means the number of bits comprising one code point (which, of course, determines how many characters can be represented by a single code point).
One of the first encoding schemes has been US-ASCII, which used 7-bit code points, subsequently being able to represent 2^7 (or 128) characters. This was later extended to use 8-bit code points, increasing the range to 2^8 (or 256) characters. Obviously, that wasn't enough, but this is as far as a single octet (simply put, 8 bits) can go.
This is how Unicode came to be. Despite not being limited to 16 bits (there are UTF-8 and UTF-32), the UTF-16 version that uses 2 octets, is the most popular one so far, so let's assume it is the only one existing for simplicity.
Unicode consists of 17 character planes, each having 2^16 code points to represent a vast number of characters (65 536 * 17 = 1 114 112). The first one is called the Basic Multilingual Plane and is mostly unremarkable. It is the other 16 that are interesting because code points there are comprised of surrogate pairs.
While the nature of surrogate pairs is out of the scope of the answer, it is the number of bits used to encode them is that matters. As you might've guessed, it is 32 bits (or 4 octets). Now, given the UTF-16 encoding, how many code units will that be for one code point? Exactly, 2 units.
What does all this have to do with the character counter?
A lot of the programming languages, including JavaScript which is the frontend language for the network, use UTF-16 encoding for strings. What's more, JavaScript is historically bad at handling surrogate pairs. Note what the length
property tells us (do not repeat the following at home):
const emoji = "๐";
console.log(emoji.length); //2??
for (const char of emoji) {
console.log(char); //๐
}
emoji.split("").forEach((chr) => {
console.log(chr); //๏ฟฝ x2 - broken surrogate pair
});
And here you have it, the length is 2 chars. It is usually referred to as a naive implementation, and developers are advised to avoid it, but, apparently, not on SE (unless this is an elaborate way of discouraging emojis, in which case - carry on!).
Take a look at this line from the source file which contains the logic for the character counter, and see if you can notice the problem (minified version I prettified a little):
c = t.ignoreWhitespace ?
l.replace(/\s+/g, " ").replace(/^\s+/, "").replace(/\s+$/, "").length :
l.replace(/\r\n/g, "\n").replace(/\n/g, "\r\n").length
Unminified version, courtesy of Makyen (link to file):
if (options.ignoreWhitespace) {
cur = val.replace(/\s+/g, " ").replace(/^\s+/, "").replace(/\s+$/, "").length;
} else {
cur = val.replace(/\r\n/g, "\n").replace(/\n/g, "\r\n").length;
}
Below is a (not so) simplistic implementation of how surrogate pair-aware input might be implemented (that is also able to count complex emojis correctly). Also available as a userscript actually replacing the native character counter on Stack Apps.
((w, d) => {
const safeStringLength = (text) => {
let count = 0;
const ZWJ = 8205;
const variationSelectorMatch = /[\ufe00-\ufe0F]/;
let skipNextChar = false;
for (const char of text) {
if (skipNextChar) {
skipNextChar = false;
continue;
}
if (variationSelectorMatch.test(char)) continue;
const code = char.codePointAt(0);
if (code === ZWJ) {
skipNextChar = true;
continue;
};
count += 1;
}
return count;
};
/** thresolhold at which to start showing the warning */
const showAt = 5;
const input = d.getElementById("title");
const counter = d.getElementById("counter");
if (!input || !counter) return console.debug("missing elements");
const max = +input.maxLength;
input.addEventListener("input", () => {
const chars = safeStringLength(input.value);
if (chars < showAt) return counter.textContent = "";
const left = max - chars;
counter.textContent = `${left} character${left > 1 ? "s" : ""} left`;
});
})(window, document);
label {
display: block;
margin-bottom: 1vh;
}
input {
padding: 1vh 1vw;
min-width: 40vw;
}
<label for="title">Title</label>
<input id="title" name="title" type="text" maxlength="150" tabindex="80" value="" placeholder="What is a surrogate pair?">
<p id="counter"></p>