8

I was searching for "thanks in advanced". One minute I'm seeing 359 results, the next minute I see 5,031 results. Don't believe me? See below.

The question, of course, is what the heck is going on?

enter image description here


BTW, I have confirmed that the 5,000 search is returning at least some valid results. It returned Is there any JavaScript native way of creating a raw TCP Socket for the client side?, for instance.

1
  • 3
    Maybe the fuzzing algorithm happened to get applied, and you got really unlucky in your timing? That behavior seems awfully odd. Commented Jan 16, 2015 at 17:23

1 Answer 1

11

Thanks for bringing this up. Your timing is excellent (or terrible, depending on your view). I had just turned off an A/B test (resulting in the 359 results case), and then rolled out the new successful A/B code (resulting in the 5031 results). It seems that you searched right before I rolled out the code, and then right after, noticing the difference. The 5031 is the "standard" going forward and should no longer fluctuate. Sorry for the momentary annoyance and inconsistency.

EDIT

Also, for what it's worth, the new version is way more accurate. ;)

13
  • Can you explain the difference in the search algorithms? Commented Jan 16, 2015 at 17:28
  • @JohnSaunders primarily the difference lies between stopwords, stemming, and tokenizing. Less aggressive stemming in new, and no stopwords considered. Also tokens are split with special characters removed in the new case; wasn't so in old. Old had a very aggressive stemmer and included stopwords.
    – Haney StaffMod
    Commented Jan 16, 2015 at 17:31
  • 2
    Someone should post about the difference, and should include examples. Commented Jan 16, 2015 at 17:34
  • @JohnSaunders there will be a search project wrap-up post, detailing a ton of what we did and are doing. Part of it now is Observer Bias - we can't let you in on the specifics yet because it will bias the forthcoming tests. We will announce it however - it's not a secret. :)
    – Haney StaffMod
    Commented Jan 16, 2015 at 17:35
  • Perhaps in the future you could disable the search page or put explanatory text at the top of it, during the execution of such a task. Seems like a basic tenet of rollouts wasn't followed here. Commented Jan 16, 2015 at 17:36
  • 1
    @LightnessRacesinOrbit it would become a race condition. How long do I disable search for? 5 minutes? 10? Even then, if someone searched 30 minutes ago, ate lunch, came back, and paged over to page 2 and got the new result count, same issue. I cannot fix this particular problem, sadly.
    – Haney StaffMod
    Commented Jan 16, 2015 at 17:37
  • 4
    Wow, I can't believe my guess was this close. Commented Jan 16, 2015 at 17:46
  • @Haney: It's better than silently serving up an incorrect result for an unspecified period of time. If I load up a page, go somewhere for a reasonably long duration, then come back to it, and things have changed, then that is something I know I have to live with. But if I am right here making "live" page requests and they change second-to-second and you had the option to hide your dev tests from me, the end user, IMO you should have taken it. Commented Jan 16, 2015 at 17:47
  • @LightnessRacesinOrbit fair enough. I'll see what I can do going forward. Maybe some kind of short-lived banner of "we're rolling out a search change. Results are gonna be weird for a few minutes. Hold onto your hats!" or something?
    – Haney StaffMod
    Commented Jan 16, 2015 at 17:50
  • 2
    @BradleyDotNET yeah when I read your comment I was all like...
    – Haney StaffMod
    Commented Jan 16, 2015 at 17:56
  • 1
    @Haney: Yeah just sort of a best-effort thing would be nice. Thanks :) [and thanks for better search results] Commented Jan 16, 2015 at 17:57
  • @LightnessRacesinOrbit oh no worries at all, and sorry for any negative tone that may have come across - wasn't intended.
    – Haney StaffMod
    Commented Jan 16, 2015 at 17:57
  • 1
    @Haney: I perceived none Commented Jan 16, 2015 at 18:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .