169

Update (2021-09-08): The experiment ran 2021-09-02 through 2021-09-08. We permanently unpinned the accepted answer 2021-09-08.


As we mentioned two weeks ago, tomorrow we are shipping a test on Stack Overflow where we change the default sort so that the accepted answer isn't pinned to the top and answers are sorted strictly by score. We are taking this action as part of our Outdated Answers project.

Background

The accepted answer is the answer with the green checkmark. Question askers can click the checkmark next to an answer to accept it if they feel that it solves their problem. Stack Overflow and Stack Exchange sites have always pinned the accepted answer (as long as it is not self-answered by the question asker) to the top of the list of answers, no matter how old it is and no matter if another answer actually scores higher. 

As many of you have pointed out, this can result in an obsolete answer enjoying favored status forever. It doesn't leave room for better answers to make it to the top, even if the community already deems another answer to be superior. This is the case for about 12% (or ~2.6 million) of our questions, where another answer outscores the accepted answer.

How we're conducting the test

For the subset of questions where the highest scored answer is not the accepted answer, we are splitting traffic. Fifty percent of users visiting these questions will see the current default sort: the accepted answer is pinned to the top. The other 50% will see answers sorted by the highest score with the accepted answer unpinned.1 

For both sorts, we are monitoring the rate at which users either upvote or copy from the top answer. We chose these two metrics because we are looking for a positive signal that the answer is useful. 

Looking at year-to-date data, upvotes only happen about 1% of the time (# of upvotes / # of question views). So we decided to also repurpose the copy event from our April Fool's prank since we saw that users copied roughly 13% of the time (# of copies / # of question views). Considering both events allows us to run the test faster.

We are not — I repeat, not — trying to hit some magic number in order to graduate the test. Think of the monitoring as an exercise in confirming that we are doing no harm. Our hypothesis is that there will be no statistically significant difference between the success rate of the two sorts. 

Unless we see a dramatic, negative, statistically significant difference, we plan to permanently unpin the accepted answer and sort by highest score at the conclusion of this test. We estimate that the test will run for a few days; the exact timeline depends on when we reach our sample size target. 

How unpinning the accepted answer fits into the Outdated Answers initiative

Unpinning the accepted answer is not going to be a cure-all for mitigating outdated content on Stack Overflow, but it is a relatively easy and important first step. It's one that the community has long requested, and we're excited to finally be able to act (assuming there's no unexpected negative impact). 

At the very least, unpinning ensures that an answer from 2012 isn't forever at the top of the list. In theory, at least, a newer, better answer can take its place.

Reality, of course, is a bit more complicated. We suspect that a large number of highest voted answers are also outdated. After all, an answer that has had several years to accumulate upvotes is going to have a higher score than a new answer that has only a few days' worth of votes. Unpinning the accepted answer won't solve this.

To start chipping away at this larger issue, we are in the early stages of developing a Trending sort that will give newer answers a better chance of gaining traction. We are looking for an effective way to give more weight to recent upvotes over older ones. We've just begun analyzing historical data. After we've had time to quantify voting patterns and map out a few potential approaches, we'll come back to Meta with an update.

We are also starting to explore what it would take to add version labels to specific answers — for example, [python-v2] vs. [python-v3].  We just solicited user input on a few high-level design concepts by adding a few questions to our monthly Site Satisfaction Survey. We will share the results and design concepts once we're further along in our analysis and discovery work.

To sum it all up, we're making progress on a gnarly, complicated issue. Step by step, we'll continue to make a dent in this beast, and we'll continue to keep you informed along the way.

Feedback

We are no longer monitoring this post for bugs. If you spot any bugs related to unpinning the accepted answer, please post them here as answers.


1 There is a minority of users who sort by Active (descending order by answer's created or edited timestamp) or Oldest (ascending order by answer's created timestamp) instead of Votes (descending order by highest score). These users will not be part of the test.

37
  • 41
    As someone who worked on this many years ago as a CM, I'm so happy we're doing this.
    – Taryn StaffMod
    Commented Sep 1, 2021 at 15:46
  • 11
    @Tom Sorted by score is the default and used by vast majority of users. For minority who sort by Active or Oldest, we will also unpin the accepted answer
    – Anita Taylor StaffMod
    Commented Sep 1, 2021 at 15:52
  • 7
    @tom Slight correction -- people who sort by Active or Oldest will not be part of the test itself, but when we permanently roll out unpinning the accepted answer, it will be unpinned for those sorts, too.
    – Anita Taylor StaffMod
    Commented Sep 1, 2021 at 17:43
  • 3
    This is very, very good news - many thanks for seeing the project through! Currently, it seems like the best initiate there is. On a more relevant note - can you clarify what do you mean by "not included"? Specifically, I am interested in whether switching preferences is going make the user eligible to get into the "unpinned" bucket (not necessarily for the same view) - will this be possible or should all who want to participate change the preference to "votes" for the duration of the test?
    – 0Valt
    Commented Sep 1, 2021 at 18:09
  • 7
    @OlegValter You should change your preference to Votes for duration of test if you want to be eligible. Experiment will only show if user is sorting by Votes (which is the default used by majority of users)
    – Anita Taylor StaffMod
    Commented Sep 1, 2021 at 18:23
  • 2
    @AnitaTaylor thanks - just to remove any doubt: is eligibility determined the moment a question is loaded or at the start of the experiment?
    – 0Valt
    Commented Sep 1, 2021 at 18:31
  • 5
    @OlegValter eligibility for the experiment is randomly determined on a user-by-user basis rather than a question-by-question basis. You will randomly be in either the baseline group (accepted answer still pinned) or the new group (accepted answer unpinned) for the duration of the experiment. Performing actions on the site will not change the group you're in.
    – Kyle Pollard StaffMod
    Commented Sep 1, 2021 at 20:05
  • 2
    @OlegValter End of your day today would be safest best. We don't have a drop-dead flip the switch timeline, but it will be BEFORE business hours U.S. Eastern time
    – Anita Taylor StaffMod
    Commented Sep 1, 2021 at 20:38
  • 7
    Hooray, this is a good change, and it sounds like you're going about it in a really responsible and community-minded way, so thank you! Commented Sep 2, 2021 at 0:03
  • 3
    That are really good news and I think this is going in the right direction now. Maybe a minor point here: It's not clear what a "negative, statistically significant difference" actually is. The metric is the "rate at which users either upvote or copy from the top answer" and it's applied to "the subset of questions where the highest scored answer is not the accepted answer". I think the selection of sample question already implies what the outcome of the metric will be. This experiment can hardly fail. How long will it run approximately? Looking forward to the analysis and conclusion. Commented Sep 2, 2021 at 9:16
  • 2
    @Jasper if you are in the experiment, then the only difference between accepted and others if the green check mark. Accepted status will not affect sort at all.
    – Yaakov Ellis StaffMod
    Commented Sep 2, 2021 at 13:26
  • 2
    Noisy comment avoidance be damned: thank you for tackling this! Commented Sep 2, 2021 at 13:54
  • 2
    Just to make sure, if this test is successful, and if the change is applied network-wide, are meta sites included?
    – Andrew T.
    Commented Sep 2, 2021 at 14:00
  • 6
    An answer that has had several years to accumulate upvotes is going to have a higher score than a new answer that has only a few days' worth of votes... We are ... developing a Trending sort that will give newer answers a better chance of gaining traction. Have you looked at the Reddit 'best' algorithm that is claimed to solve this very problem?
    – Ian Goldby
    Commented Sep 3, 2021 at 14:05
  • 7
    Could there be an indicator somewhere around the Question indicating the presence of an accepted Answer, so one doesn't need to go looking to see if there is one, please?
    – Scratte
    Commented Sep 4, 2021 at 3:35

2 Answers 2

24

This is awesome. Thank you!

We are looking for an effective way to give more weight to recent upvotes over older ones.

I'm curious how this will differ from or improve on the existing functionality to handle this. The "active" sort option should handle this pretty well... maybe just make the sort buttons a font-size or two bigger so they're more noticeable? Or give them a slight change in background color?

E.g. this:

Answers view showing current sorting buttons

becomes something like this?

enter image description here

Or something similar. Currently there's no on-screen info about sorting; you just kind of have to intuit that that's what those buttons do. Making them bigger and adding some explanatory text might change your metrics on how many users sort by non-default sorting options.

14
  • 11
    " how this will differ from or improve on the existing functionality to handle this" -> While active sorts by last edit/post date I think the new sorting algorithm will be much smarter, inline with some historical weighted sum proposals
    – bad_coder
    Commented Sep 1, 2021 at 16:34
  • 23
    The existing Active sort is just looking at edit/creation timestamps, not votes over time. And yes, as part of this effort, we will be looking at making Sort options clearer and more discoverable
    – Anita Taylor StaffMod
    Commented Sep 1, 2021 at 17:59
  • @AnitaTaylor Right, the point of Active is to sort by "newest", because you want to have more attention drawn to newer posts to even out that vote spread. Wouldn't that serve as the best payload for delivering votes to newer posts vs older posts, rather than trying to artificially "weight" newer votes with some extra importance vs older votes? Using some fancy, complicated algorithm like the one bad_coder linked to runs the risk of what Shog's comment there already covers.
    – TylerH
    Commented Sep 1, 2021 at 18:05
  • 7
    @TylerH The number of users who make the effort to sort by Active and then upvote is small compared to the number of users who upvote over time. Since this Trending sort is a separate topic unto itself, I'm going to hold off on more discussion until we do some internal data analysis and I write a separate post
    – Anita Taylor StaffMod
    Commented Sep 1, 2021 at 18:27
  • 6
    @AnitaTaylor I think if you make the sorting feature more visible/discoverable, it will change the number of users who use it, as I mentioned in the answer. However, I agree "Trending" would be an entirely separate, 4th sort method.
    – TylerH
    Commented Sep 1, 2021 at 19:40
  • 2
    I agree that discoverability is a bit lacking with the current UI. I'll admit that I used this site for several years before I even noticed those sort buttons, much less used them. Answers have visible breaks between them but the stuff in the header seamlessly flows into the topmost answer, which makes it easier for your brain to filter out without realizing it.
    – bta
    Commented Sep 3, 2021 at 20:32
  • @TylerH if you're old-fashioned and still using light mode (as opposed to dark), the type of sorting applied to the answers is visible with a different background shading that's pretty noticeable. I guess this got missed in dark mode.
    – MattDMo
    Commented Sep 4, 2021 at 20:02
  • 1
    @MattDMo I'd hesitate to call that "pretty noticeable". It's also not any more or less noticeable than the dark mode implementation, in terms of color contrast.
    – TylerH
    Commented Sep 7, 2021 at 15:53
  • @TylerH "fairly noticeable"? Just "noticeable"? "They're different colors"? I don't know. It's noticeable to me pretty easily on my screen, but maybe not to others...
    – MattDMo
    Commented Sep 7, 2021 at 15:55
  • Making something more noticeable means there are more users noticing it initially and using it for the first time, which may be good. But, after someone already knows about it, it would just be a distraction, pretty much by definition (if something stands out so you notice it, i.e. your attention is being put on it, and not on what you're trying to focus on, that is the definition of a distraction). Explicitly being noticeable is bad in the long term. It just needs to be noticeable enough to be found if someone specifically looks around or tries to find it, which I might say is already true. Commented Sep 8, 2021 at 21:13
  • 2/2 Although this isn't to say that the current display is perfect, but rather I just don't find the argument of noticeability to be compelling. The problem of initial noticeability (if it is a problem) is generally solved in 1 of 2 ways: (1) highlighting something so users click on it, and then removing the highlighting after they do or (2) having some sort of tutorial or guide that points users towards it and explains what it does. I might agree that adding "sort by" is probably a good idea, but this is less about noticeability and more about having buttons be self-explanatory. Commented Sep 8, 2021 at 21:22
  • 1
    @BernhardBarker The formal term for that is "discoverability", and part of the issue of "if someone specifically looks for it" is just that: they have to know it's there first. I agree something like a one-time modal of "hey try this out" would be ideal. There is a huge gulf, though, between "make this thing more noticeable so people are aware it's there" and "this is so in your face that it's distracting. I don't think anyone would seriously suggest that an incremental change in color or contrast would have such an effect.
    – TylerH
    Commented Sep 8, 2021 at 21:27
  • 1
    @TylerH I'd still be inclined to say an incremental noticeability change shouldn't be targeted at discoverability. Discovering something only takes a single moment of time (even if this could happen multiple times). Using a site for years is looking at a page containing that thing for dozens or hundreds or thousands of hours. The static view should be optimised for the latter, not the former. Different people have different thresholds for what they find distracting (I, for example, find the sort buttons to already be drawing quite a lot of my focus, although I wouldn't say it's "too much") 1/2 Commented Sep 8, 2021 at 22:21
  • 1
    2/2 Incremental changes lead to ... more incremental changes. If you're making the wrong type of change for the goal you want to achieve, then you're quickly going to start making more users angry than happy. Although I would be interested in reading some UX.SE answers on the subject of which degree you should optimise a static view for discoverability (but not quite interested enough to post a question something myself). Commented Sep 8, 2021 at 22:21
-15

+1 for adding a weighting system favoring newer votes.

May I suggest a couple of things?

Suggestion 1:

I suggest a bottom-up weighting system, rather than a top-down.

Top-down example: all votes on all answers are weighted evenly based on age.

Bottom-up example: Answer A has a very large number of votes. Nearly all of A's votes are between 5 and 10 years old. Answer B has relatively few votes (say, half what A has). But B's votes are spread out evenly over the past 10 years. A's votes that are older are given somewhat less weight per vote than B's votes of the same age. Age-based weight is informed by the trend for the answer itself, rather than a single weight applied across all answers.

That is, a weighting system that tries to determine the "current best" answer. Not that the above example is the right/only/best way to do this, but I wanted to promote the idea of looking at the nuances in an approach to surfacing value. Paraphrasing the OP, "Reality is complicated." +1 also for taking into account copying from answers in addition to votes. All value signals taken into account, with time factored in.

Suggestion 2:

Also of note, that sorting in itself is not the only way to communicate answer value to users. An answer, wherever it falls in the sort order, could be marked with an annotation, such as "best answer from 2012 - 2017". As an example, for a question about a JavaScript approach to a particular problem, knowing that an older answer really was the very best answer with ES5, taken together with the "2018-preset best answer" (based on ES2015+ widespread support) annotation on the answer at the top of the sort order, can greatly inform a developer. "Ah, this is the old way of doing it, so when I see that in the code base, I know I can use the more declarative/expressive new way."

I am not saying an annotation including language like "best" is actually appropriate (or even meaningful, as "best" needs context and is more complex than simple). Maybe "most valued" or similar. "Community resonance" may be nearer the mark, but less clear. "Scientific score" is more handwavy, but it may do the job, especially if a description of how it's applied is provided.

11
  • 12
    the current A/B test isn't a weighting system, it just... unpinned the accepted answer.
    – Kevin B
    Commented Sep 3, 2021 at 15:06
  • 7
    Read through your feedback. This relates to Trending sort, which will have its own dedicated Meta post after our data team does some preliminary analysis
    – Anita Taylor StaffMod
    Commented Sep 3, 2021 at 16:54
  • 9
    Implementing suggestion 2 would require a lot of effort for no real benefit. How does informing the reader that an answer was the "best answer from 2012 - 2017" add any meaningful "answer value"? Surely it just adds noise, and any implementation would provoke new controversy over whatever wordings were chosen. Adjusting the presentation order of answers is fine, but adding meta comments is a bridge too far for me.
    – skomisa
    Commented Sep 3, 2021 at 16:57
  • "than an answer with fewer overall votes, but which has had consistent votes over time weighs its older votes." This sentence fragment does not make grammatical sense. Can you clarify what it is you were trying to say here (in an edit to the answer, preferably).
    – TylerH
    Commented Sep 3, 2021 at 20:35
  • I agree with skomisa's critique of point #2; first, it doesn't seem like it would add much benefit (why do I need the best answer from the past? I want the best answer from now). Also, who gets to decide "the best answer from 2012-2017"? The system, based on votes?
    – TylerH
    Commented Sep 3, 2021 at 20:36
  • @TylerH it was a pretty winding sentence. I've re-worded the entire paragraph to clarify what I meant. Thanks for the feedback. Commented Sep 3, 2021 at 21:58
  • 1
    @TylerH re: point #2. That suggestion taken in context of a sorting as described in point #1 is what I meant. As you said: I want the best answer from now. point #1 would (hopefully!) put that at the top. A user seeing an answer further down the page which has far more votes than the answer at the top could find value in seeing why the answer is further down: "oh, right - this used to be the best answer". That was kind of my thinking. Commented Sep 3, 2021 at 22:02
  • A similar annotation could be given to answers that have a near net 0 score, but which have a large number of overall votes (up and down). "Polarized opinion" or "Controversial". A +23/-22 answer probably should be sorted higher than an ancient answer that only ever garnered +2/-0 votes, and probably should have its nature surfaced in such an annotation. These are only some outside the box ideas about other things that could help, in addition to unpinning the accepted answer. To @AnitaTaylor's point - my answer here probably belongs in a separate discussion. Commented Sep 3, 2021 at 22:07
  • 2
    I don't think annotating pages with historic vote info is a good idea, but could be something to add to the "timeline" page (already linked under the votes on each post now). Since it would be a separate page, a graph of answer scores over time for the whole question could be interesting, with one line per answer. (With the line greyed out for periods of time when an answer is deleted...) Commented Sep 5, 2021 at 4:24
  • Outdated answers have very limited use cases. If someone is using a language/API version where that's no longer the best solution, the overwhelming majority of people won't find it useful to know it used to be the best solution. I think the only compelling use for such a note is when the answer still applies, i.e. versions X-Y: use this, versions Z+: use this. But this is probably best left to answerers to note manually. People mostly know version numbers, not dates, which means it won't be all that useful to annotate by date (and such an annotation is not even necessarily always useful). Commented Sep 7, 2021 at 21:09
  • I think adding potential clutter to the screen should come with a rather strong and convincing justification, and not just be something some people might find somewhat useful occasionally. Commented Sep 7, 2021 at 21:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .