15

If I understand the search help correctly, searching for appreciat* should always also include the results for a search of appreciate.

But it seems it doesn't? Searching for appreciat* returns just 1,502 results while searching for appreciate returns 547,612 results.

Is this a bug or am I missing something obvious?

5
  • 1
    and searching apprecia*e returns 10 results
    – Tanner
    Commented Jan 14, 2015 at 9:39
  • 1
  • 1
    as a side note, searching for quoted strings: "appreciate" doesn't highlight search words in the results like it does without quotes.
    – Tanner
    Commented Jan 14, 2015 at 9:47
  • as a side note to a side note, searching for appreciate, highlights similar words like: appreciable and appreciations. How is this black magic performed?
    – Tanner
    Commented Jan 14, 2015 at 9:59
  • 1
    Taking a look. Re: the highlighting, it's because of stemming in both cases.
    – Haney Staff
    Commented Jan 14, 2015 at 15:18

2 Answers 2

12

Thanks for bringing this up. This isn't a bug, but instead a super unintuitive feature of the current search algorithm. In search, a good practice is to use stemming to help with the relevance of your results. Stemming in a nut shell reduces words to root or base forms (think of it as a one-way function). When we index, we stem, and when we search, we again stem. If the stemmed search terms match the stemmed index fields, you'll see them in your search results.

Stemming is done so that search is not rigid. Searching for "appreciate" and "appreciates" pretty much means the same thing, so the results should be identical. Otherwise, you'd have to type the exact phrasing of a question or answer for it to appear in the results, which would suck.

What's happening here is that Wildcard queries are not stemmed at all. So, when you search for "appreciat*" you're finding all results that have a term that starts with "appreciat" in them. When you search for "appreciate", we stem that term into its root: appreci (the root is not always English and that's OK):

Stemmer Fun

Search for appreci or appreci* will confirm this fact, by the way.

The problem here is that our current stemmer appears to be aggressive. Very aggressive. The more aggressive a stemmer is, the broader the root term will be. One might argue that the current stemmer stems too much. Searching for "appreci" opens you up to many more results than "appreciat" or even "appreciate". This means that our search currently supports good fuzziness (tolerance of varying forms of words and like words), but bad relevance (too much noise included in the results).

If you're not aware, we're currently working on improving search. To this end we've been running many tests in the background to confirm or deny that our incremental changes are better. After investigating this issue, I've decided that we will try a less aggressive stemmer in future tests. This stemmer will stem as follows:

input: appreciate appreciated appreciating appreciates appreciate's

result: appreciate appreciate appreciate appreciate appreciate

We don't know for sure that this will perform better on the whole (vs better just for your exact case), thus the testing of it. But know that it's forth-coming, and assuming that the lighter, less aggressive stemmer performs better, we'll roll it out to everyone.

A few paragraphs on the desired aggressiveness of a stemmer

6
  • Thank you very much for this detailed answer! Turnes out I have really missed something - it just wasn't very obvious. ;-)
    – Balder
    Commented Jan 14, 2015 at 16:28
  • @Balder it's not even a little bit obvious. We're working on making it much more intuitive. :)
    – Haney Staff
    Commented Jan 14, 2015 at 16:55
  • 1
    Does the fact that your are working on search mean that we will finally be able to search for ||= and the like? That's a bug that has existed since the days of the private beta. Commented Jan 15, 2015 at 0:22
  • @JörgWMittag eventually, yes. :)
    – Haney Staff
    Commented Jan 15, 2015 at 3:36
  • 1
    That's very good news. ||= is actually a pet peeve of mine, since there are dozens of duplicate questions about it in the ruby tag, and you cannot even search for them in order to close them. They also don't show up under the related questions whenever someone asks the n+1th duplicate. I remember Jeff Atwood's argument being "if Google can't find it, so can't we", which totally ignores the fact that Google is a search engine for English, whereas SO's engine is for Programmerese, in which &&, ||, ++, << are proper "words" and OR is not a stopword. Commented Jan 15, 2015 at 4:01
  • Similar issue: Just tried to search for "><>" to determine the origin of a language, search fell on its face. The ability to perform such searches would be quite nice :) Commented Jul 29, 2015 at 17:08
3

Just a heads up, you should find that this now works a lot more intuitively. The test was considered successful / practical. Cheers!

2
  • 1
    Congrats - a first look at the search results is very promising! Very much apprechiat* ;)
    – Balder
    Commented Jan 16, 2015 at 17:21
  • 2
    Thanks @Balder - but we're just getting started. Lots and lots more coming in the next few weeks. :)
    – Haney Staff
    Commented Jan 16, 2015 at 17:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .