41

Earlier today I answered this question about a mojibake issue where a single UTF-8 character was incorrectly being decoded into the Unicode string "’". Before I answered it, I thought there might be a canonical Q&A for this specific issue already. I did a couple searches, to see if I should close the new question as a duplicate. I searched by putting "’" in the Stack Overflow search box (both with and without the quotation marks and with and without [python]). The results did not include anything remotely relevant (it was more or less the top-voted posts on the whole site, and the top-voted Python posts). Not thinking too much about this, I assumed the irrelevant results mean this was a novel or obscure bit of mojibake and so I wrote an answer to the question.

A little while later (when I went to close some browser tabs), I looked more closely at one of the external searches I'd done when figuring out which encoding was being used incorrectly. I noticed that there are dozens of Google search results from Stack Overflow with the exact mojibake string I searched for in their titles (including a few regarding Python code, though none I've read yet that seem quite similar enough for the new question to be a duplicate, though feel free to disagree and vote to close as appropriate). I'm surprised I'd not seen any of them in my checks using our own search tools.

It seems that Stack Overflow's search won't ever use the or characters at all. The â character appears to be normalized so that it gets treated like an a, which means it matches every question and answer that includes the indefinite article. The bad matching happens even with quotation marks around the string, which should request an "exact" search (according to the "advanced search tips"). Searching for "’" seems to be the same as searching for "a".

7
  • 1
    Note that searching for â is not equal to searching a (different amount of results). Instead, it's equal to searching "a^". That is, if you use u00E2 Latin Small Letter A with Circumflex and not u0061 Latin Small Letter A and u0302 Combining Circumflex. Probably, unicode normalization is going wrong somewhere, if you implement it properly those should return the same result (if it's implemented at all, ES does support it).
    – Erik A
    Apr 18, 2019 at 7:52
  • 1
    You can use the Data Explorer for this, e.g. (excuse my SQL crappiness) this. Not ideal though.
    – Wai Ha Lee
    Apr 18, 2019 at 8:16
  • 19
    psst, stackse
    – ren
    Apr 18, 2019 at 11:57
  • 4
    I always use another search engine to search for duplicate. // There have been some suggestions (I can't find them anymore, but I remember that there are) that SO should use/encourage users to use custom Google search because SO search is bad.
    – user202729
    Apr 18, 2019 at 14:48
  • 1
  • 5
    Did you try code:"’" in SO search? That can help for short / weird strings if they appear inside code blocks in questions/answers. But IDK if that stops collation rules like â matching a. It can be useful to find things like %% for NASM macro local labels. Apr 19, 2019 at 16:25
  • 3
    @PeterCordes Yes, code:"’" avoids â matching a
    – wjandrea
    Apr 19, 2019 at 16:48

0

You must log in to answer this question.

Browse other questions tagged .