Skip to main content
14 events
when toggle format what by license comment
Aug 5, 2016 at 20:40 comment added TylerH Any time I try to find a duplicate in the close vote duplicate search modal, I end up with few to no results. Even when I know I'm typing in the exact or nearly exact title of a popular question.
Feb 20, 2015 at 14:52 comment added Chris Pratt @Haney: Reading your description of all this unfortunately hits way too close to home. I recently had to deploy Elasticsearch and set up all the types and indexing. It's indeed very much an art, and I have total respect for you and whoever else works on the search for StackOverflow. It's enough to drive you insane tweaking everything to be just right.
Jan 9, 2015 at 16:03 comment added Haney Staff It's possible to do a lot of things. Please make it a meta request if you're interested!
Jan 9, 2015 at 8:39 comment added Travis J @Haney - Would it be possible to add a date range to the search?
Nov 21, 2014 at 23:44 comment added Haney Staff A lot of these tests will be announced to meta shortly after, not during. This is because bias will screw with our normalized outcomes/results. So we can't tell you about what we did until after we did it... I'm more than happy to share the results as we go however. :)
Nov 21, 2014 at 23:43 comment added Travis J Well it all sounds very magical :) Can't wait for part 2 in the future or perhaps a sandbox to play in.
Nov 21, 2014 at 23:41 comment added Haney Staff Yeah it's a science and an art. There will be synonyms, isolation of code blocks in questions, even titles stored in reverse to allow edge-gramming for things like type-ahead. ;)
Nov 21, 2014 at 23:28 comment added Travis J @Haney - Okay, that is much clearer thanks :) Sounds complicated to determine the value of the fuzz, as in whether the fuzz was the strongest part of the result versus the weakest. Perhaps some of the field generation could come from using the connections in tag synonyms.
Nov 21, 2014 at 23:17 comment added Haney Staff So for example: given a question title, we'd store the title verbatim, stemmed title, and title ngrams/shingles to allow for tolerance of misspellings. The query would hit all 3 with a boost on verbatim field. Exact match most relevant followed by fuzz.
Nov 21, 2014 at 23:14 comment added Haney Staff The query would have to enumerate over the result set and perform a calculation on each item until it reached the N threshold results. The enumeration and calculation are expensive and scale poorly. It's better to pre-calculate and store your desired data format (stems, ngrams, etc.) in the index along with the original unaltered field. This takes up much more index space, but hard drives are cheap relative to CPU.
Nov 21, 2014 at 23:09 comment added Travis J So if someone is searching for Math.random, when they get to Math., Math.random will be one of the guessed ngram fields (at least ideally)? At that point, the query would then be issued with both Math.random and simply Math (the stem) and results with Math.random would come first, followed by results with Math? As for your note about query versus index, can you clarify? Is that calculating what to search in the query, versus having some sort of composite key to index? Or are you talking about lookup tables for the field terms?
Nov 21, 2014 at 23:00 comment added Haney Staff Regarding fuzziness: at the query is expensive relative to pre-calculated in the index. The current thought is to try a combination of literal field, stemmed field, and ngram'd field, giving literal matches a boost so that they're on top. That and maybe a Google style quotation system to give you rigid literal matches.
Nov 21, 2014 at 19:45 history edited Travis J CC BY-SA 3.0
added 163 characters in body
Nov 21, 2014 at 19:35 history answered Travis J CC BY-SA 3.0