As part of our Outdated Answers initiative, we shipped a temporary data-collection exercise in May. Unfortunately, the results were inconclusive and didn't lead to any big aha moments. After I briefly recap what we did (and didn't) learn, I'll update you on what we're doing next: sorting and labeling answers.
Flagging exercise results
From May to late June, we showed a prompt on Stack Overflow that allowed users to flag specific answers as being outdated, along with a reason why. We hoped we could use the data to quantify how prevalent the problem is, to inform decision making, and to provide an initial training sample for machine learning, but the data wasn't very compelling.
Users marked answers as outdated ~23,000 times. This was a low response rate: roughly 17 to 27 answers flagged out of every 100,000 answers.
There were no discernible differences between accepted answers (with the green checkmark), highest voted answers, and remaining answers. Accepted answers had the highest rate of outdatedness, but because it is always the top answer, we couldn't tease out the impact of position on page. In addition, we couldn't rule out that the differences were due to chance.
"Newer, more efficient answers exist" was chosen 45% of the time as the reason users marked an answer outdated. This was consistent with our March 2021 survey.
We compared answers that scored higher, lower, or equal to the accepted answer and didn't find any definitive differences.
We looked at recent upvotes/downvotes and found a weak association with outdatedness for votes cast within the last three months.
There was no strong relationship between question age, answer age, and outdatedness.
We found a low rate of accepted answers that were edited after they were accepted. Of the 3.7 million accepted answers in the data set, only about 4.9% were edited and only 9% of those edits were by the author.
Next steps: sorting and labelling
Despite the inconclusiveness of the study, we are making progress on a longstanding Meta request: to change the default sort so that the accepted answer isn't pinned to the top and answers are sorted strictly by score.
Our plan is to test this out on a subset of questions where the highest scored answer is different from the accepted answer. We'll compare the upvote rate and the rate of users copying all or part of an answer between the two sorts: accepted answer first vs. highest scored answer first. Assuming there is no negative impact, we then plan to roll out the change. Look for more details when we get closer to launch.
In addition to unpinning the accepted answer, we are in the early exploration phase of:
developing a Trending sort that prioritizes recent upvotes more than older votes.
placing a Trending indicator on answers that have recently received comparatively higher voting activity than other answers.
making it possible to label individual answers with technology versions — for example, [python-v2] vs. [python-v3].
We've been getting user feedback on these early concepts by adding a handful of questions to our monthly Site Satisfaction Survey every few weeks. We will post more details on Meta when we are further along in the discovery process and seeking feedback.
If my answer helped you, please give it the green tick, upvote it, and copy the text in my snippet.
?!? This should definitely spur people to add code snippets in addition to plain English text (unformatted text) into answers.