Update (2021-09-08): The experiment ran 2021-09-02 through 2021-09-08. We permanently unpinned the accepted answer 2021-09-08.
As we mentioned two weeks ago, tomorrow we are shipping a test on Stack Overflow where we change the default sort so that the accepted answer isn't pinned to the top and answers are sorted strictly by score. We are taking this action as part of our Outdated Answers project.
Background
The accepted answer is the answer with the green checkmark. Question askers can click the checkmark next to an answer to accept it if they feel that it solves their problem. Stack Overflow and Stack Exchange sites have always pinned the accepted answer (as long as it is not self-answered by the question asker) to the top of the list of answers, no matter how old it is and no matter if another answer actually scores higher.
As many of you have pointed out, this can result in an obsolete answer enjoying favored status forever. It doesn't leave room for better answers to make it to the top, even if the community already deems another answer to be superior. This is the case for about 12% (or ~2.6 million) of our questions, where another answer outscores the accepted answer.
How we're conducting the test
For the subset of questions where the highest scored answer is not the accepted answer, we are splitting traffic. Fifty percent of users visiting these questions will see the current default sort: the accepted answer is pinned to the top. The other 50% will see answers sorted by the highest score with the accepted answer unpinned.1
For both sorts, we are monitoring the rate at which users either upvote or copy from the top answer. We chose these two metrics because we are looking for a positive signal that the answer is useful.
Looking at year-to-date data, upvotes only happen about 1% of the time (# of upvotes / # of question views). So we decided to also repurpose the copy event from our April Fool's prank since we saw that users copied roughly 13% of the time (# of copies / # of question views). Considering both events allows us to run the test faster.
We are not — I repeat, not — trying to hit some magic number in order to graduate the test. Think of the monitoring as an exercise in confirming that we are doing no harm. Our hypothesis is that there will be no statistically significant difference between the success rate of the two sorts.
Unless we see a dramatic, negative, statistically significant difference, we plan to permanently unpin the accepted answer and sort by highest score at the conclusion of this test. We estimate that the test will run for a few days; the exact timeline depends on when we reach our sample size target.
How unpinning the accepted answer fits into the Outdated Answers initiative
Unpinning the accepted answer is not going to be a cure-all for mitigating outdated content on Stack Overflow, but it is a relatively easy and important first step. It's one that the community has long requested, and we're excited to finally be able to act (assuming there's no unexpected negative impact).
At the very least, unpinning ensures that an answer from 2012 isn't forever at the top of the list. In theory, at least, a newer, better answer can take its place.
Reality, of course, is a bit more complicated. We suspect that a large number of highest voted answers are also outdated. After all, an answer that has had several years to accumulate upvotes is going to have a higher score than a new answer that has only a few days' worth of votes. Unpinning the accepted answer won't solve this.
To start chipping away at this larger issue, we are in the early stages of developing a Trending sort that will give newer answers a better chance of gaining traction. We are looking for an effective way to give more weight to recent upvotes over older ones. We've just begun analyzing historical data. After we've had time to quantify voting patterns and map out a few potential approaches, we'll come back to Meta with an update.
We are also starting to explore what it would take to add version labels to specific answers — for example, [python-v2] vs. [python-v3]. We just solicited user input on a few high-level design concepts by adding a few questions to our monthly Site Satisfaction Survey. We will share the results and design concepts once we're further along in our analysis and discovery work.
To sum it all up, we're making progress on a gnarly, complicated issue. Step by step, we'll continue to make a dent in this beast, and we'll continue to keep you informed along the way.
Feedback
We are no longer monitoring this post for bugs. If you spot any bugs related to unpinning the accepted answer, please post them here as answers.
1 There is a minority of users who sort by Active (descending order by answer's created or edited timestamp) or Oldest (ascending order by answer's created timestamp) instead of Votes (descending order by highest score). These users will not be part of the test.
An answer that has had several years to accumulate upvotes is going to have a higher score than a new answer that has only a few days' worth of votes... We are ... developing a Trending sort that will give newer answers a better chance of gaining traction.
Have you looked at the Reddit 'best' algorithm that is claimed to solve this very problem?