Update: The outcome of this exercise led two major site changes*:
- Accepted answers are no longer pinned.
- A new way to sort answers. See Trending: A new answer sorting option for more information.
As I mentioned last month, the next step in the Outdated Answers initiative is an exercise for flagging outdated answers. Starting May 13, you may see temporary prompts on Stack Overflow that ask if an answer is outdated:
Is this answer outdated? Yes | No
The prompts have a chance to appear when the question is at least 60 days old and there are at least two answers to the question. If you indicate that the answer is outdated, a modal appears that asks you to select various reasons why the answer is outdated.
Our goal is to collect a minimum of 1,000 answers that are flagged as outdated. We are showing the prompt 1% of the time, but may increase sampling if we aren't getting enough responses. The exercise will run for 1-4 weeks.
After the exercise concludes, we will analyze the data to determine:
whether outdatedness differs for the accepted answer (with the green checkmark) vs. other answers.
whether outdatedness differs by the answer's position on the page.
the distribution of outdated reasons and whether there are any correlations between them.
what keywords in the comments or the answer itself mention that it is outdated.
how sorting would be impacted if we were to unpin accepted answers.
correlations between recent upvotes/downvotes and outdatedness.
how we might weigh votes over time to maximize up-to-date answers appearing on top.
correlations between question age, answer age, last edited age and outdatedness.
correlations between specific tags and outdatedness.
The exercise will also provide an initial training sample for a potential machine-learning algorithm.
Additional research findings
Since last month's update, we conducted a survey and one-on-one interviews to get an initial read on how we might change answer sorting and other potential solutions.
When asked, "Should we change the way we sort answers?"
47% of respondents preferred no change, whereas 53% of respondents preferred unpinning the outdated accepted answer and changing how we sort.
32% wanted to sort by highest score.
21% wanted to sort by time published and recent votes.
We also asked users to evaluate the desirability of various potential solutions from "very undesirable" to "very desirable." After weighting the responses, the scores were as follows:
Potential solution | Score |
---|---|
Allow tags on answers to show versioning (i.e., python-3.9.0) | 0.47 |
Ability to flag outdated accepted answers manually | 0.43 |
Adding an "out-of-date" banner to outdated accepted answers | 0.43 |
Creating a new "legacy answer" status to preserve outdated accepted answers | 0.41 |
Color-coding the outdated accepted answers | 0.37 |
Create a new review process for updating outdated accepted answers | 0.33 |
Next steps: sorting experiment
After we finish analyzing results from the flagging exercise, we are planning an experiment where we will sort answers in different ways and ask users whether the top answer is the best answer. We will also track secondary metrics, such as copying, upvotes, downvotes and resorting to existing options (sorting by active, oldest or votes). Stay tuned for more details as we get closer to launch.
*: This info came from @RyanM's helpful comment.