46

In October 2019 the company conducted an experiment on Stack Overflow with A/B testing in which they showed score 0 if the score was in truth negative. The experiment lasted two weeks and after voting a message

Thanks for your vote! Your vote has been recorded and it affects this post's ranking. Since this post has reached the minimum score of 0 your vote will not be displayed.

was shown. The experiment was not announced and many people erroneously reported the behavior on meta as a potential bug. People critically commented on the ethical implications of deliberately lying.

Shog9 was heavily involved, but probably came not around to report on the results and isn't with the company anymore. It may have been simply forgotten with all that was going on.

Being A/B testing it probably had a clear result. Deliberately not showing the true score is a kind of serious action in my eyes and should at least elicit a proper post mortem with maybe an apology for the misinformation. And I'm curious, since I took part in it, about the results.

What were the results of the experiment? Why was it conducted and did the assumptions got confirmed or rejected?

2
  • 19
    Most likely outcome: nothing to see here. Commented Feb 12, 2020 at 22:52
  • 5
    Users generally stop voting when a post reaches -3. How many SO users got a permanent ban because of that test is likely to be the interesting statistic. But hard to dig up and surely will never be made public. Commented Feb 13, 2020 at 17:32

1 Answer 1

72

I don't have access to the data for that anymore, but going from memory:

  • there were a LOT of problems running that test. Most of them are documented in the discussions had at the time, so I won't rehash them other than to note: we thought it was gonna be a hard test, and... It was harder than we thought it would be.
  • we got a lot of data back, but in the end not enough to draw conclusive results. Given the severe disruption caused by the changes being tested, the decision was made to not continue it for a longer period.
  • there did not appear to be a clear win for hiding post scores in any scenario tested.
  • there may have actually been negative consequences - for authors, for moderators, even for readers.

There's probably more to learn here, but... I strongly suspect this approach was a dead-end; more useful gains would probably arise from changes to other aspects of the voting, scoring, or ranking systems. Donna has done a fair bit of research in this area; we had at one point planned to write about this on the blog, and I very much hope that she will still find time to do so eventually!

4
  • 23
    This aligns with what I recall but I also don't have access to the data. ;) Paired with the other research / data about voting gathered it would indeed be a cool blog post at the very least.
    – Meg Risdal
    Commented Feb 13, 2020 at 4:37
  • 50
    Seeing Shog and Meg without diamonds feels so damn different.
    – Bhargav Rao Mod
    Commented Feb 13, 2020 at 5:21
  • Did Donna's (I don't know who that is, actually) research ever go anywhere? Commented Aug 20, 2023 at 10:50
  • 1
    I believe Donna quit shortly after I wrote this, so gonna say "no" @karl
    – Shog9
    Commented Aug 21, 2023 at 3:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .