In February, we announced an initiative to address outdated answers on Stack Overflow. Here's an update on what we've done since then and what we're working on now.
TL;DR: We are still conducting various forms of research, but we see unpinning the accepted answer as our first key area to explore. And we heard you loud and clear about not deleting answers.
Survey results: how you determine that accepted answers are outdated
In March 2021, we added a few questions to our Site Satisfaction survey, an ongoing monthly survey that asks randomly selected users how they feel about Stack Overflow. We focused on the accepted answer — the answer with the green checkmark. Based in part on your feedback, we asked:
- How often do you see this type of outdated answer on Stack Overflow?
Never | Rarely | Sometimes | Often | Always | |
---|---|---|---|---|---|
Answer no longer works due to updates to language or framework, etc. | |||||
Answers still work, but there is a newer, better way to achieve the same end | |||||
Answer code is generally broken | |||||
Answers that now pose security risks or are dangerous to use | |||||
Answers that are valuable to legacy technologies, but aren't valuable to newer technologies. |
- Please select how important each of the following factors are when you are determining if an answer is outdated.
header 1 | Not at all important | Slightly important | Moderately important | Very important | Extremely important |
---|---|---|---|---|---|
Timestamps: when it was answered originally | |||||
Timestamps: when it was last edited | |||||
Total downvote/upvote count | |||||
Newer answers with a high vote score | |||||
Comments say the answer is outdated | |||||
There is a notation on the answer that says it is outdated | |||||
Other answers refer to the outdated answers | |||||
High reputation users say the answer is outdated | |||||
Code is generally broken | |||||
Answer references, citations, and documentation (e.g., outdated syntax, outdated documentation links, etc.) |
Here are the results, from 963 respondents.
The first graph shows how often users saw each type of outdated answer, from "never" to "always."
Here is the most common response for each type of outdated answer.
- Answer no longer works due to updates to language or framework, etc.: sometimes, followed closely by rarely
- Answers still work, but there is a newer better way to achieve the same result: sometimes, followed by often
- Answer code is generally broken: rarely was most popular by far
- Answers that now pose security risks or are dangerous to use: rarely, followed by never
- Answers that are valuable to legacy technologies, but aren't valuable to newer technologies: sometimes, followed by rarely
The second graph weighs the frequency at which users saw each type of answer to produce a weighted average.
- Answers still work, but there is a newer better way to achieve the same result: 0.46
- Answer no longer works due to updates to language or framework, etc.: 0.34
- Answers that are valuable to legacy technologies, but aren't valuable to newer technologies: 0.33
- Answer code is generally broken: 0.21
- Answers that now pose security risks or are dangerous to use: 0.16
The third graph shows the frequency at which users rated various factors important in determining if an answer is outdated, from "not at all important" to "extremely important."
- Timestamps: when it was answered originally: very important, followed by moderately important and extremely important
- Timestamps: when it was last edited: very important, followed by moderately important
- Total downvote/upvote count: moderately important, followed by slightly important
- Newer answers with a high vote score: very important, followed by extremely important
- Comments say the answer is outdated: very important, followed by extremely important
- There is a notation on the answer that says it is outdated: very important, closely followed by extremely important
- Other answers refer to the outdated answers: moderately important, followed by very important
- High reputation users say the answer is outdated: very important, followed by moderately important
- Code is generally broken: very important, followed by extremely important
- Answer references, citations, and documentation (e.g., outdated syntax, outdated documentation links, etc.): very important, followed closely by moderately important
The final graph weighs the frequency at which users chose each level of importance to produce a weighted average.
- There is a notation on the answer that says it is outdated: 0.31
- Comments say the answer is outdated: 0.31
- Newer answers with a high vote score: 0.30
- Code is generally broken: 0.29
- Timestamps: when it was answered originally: 0.27
- Timestamps: when it was last edited: 0.26
- Answer references, citations, and documentation (e.g., outdated syntax, outdated documentation links, etc.): 0.26
- High reputation users say the answer is outdated: 0.24
- Other answers refer to the outdated answers: 0.21
- Total downvote/upvote count: 0.20
We analyzed the Feb 2021 Meta post
We categorized your answers and comments into common themes for what you want to see happen. These are some of your top concerns/suggestions:
Don't delete legacy answers
Accepted answers should not appear first
Introduce an “out-of-date” banner to be applied to outdated answers
Allow version tagging on answers (e.g., Java 4, Java 7, Java 56)
Add new sort/filter options
Happening now: new survey running
Starting March 31 and running through April 15, we added a few questions to the monthly survey to further validate some of the Meta suggestions with a larger audience. (Please note: we are not soliciting feedback on these questions.)
Should we change the way we sort answers?
No, I like the current sort (i.e., accepted answer always first).
Yes, sort all answers, including accepted answers, by highest score.
Yes, sort all answers, including accepted answers, by time published and recent votes.
Other (please specify) ____________
We want to make changes to the site to help find and deal with outdated answers. We have a range of solutions in the table below. How would you evaluate the desirability of each potential solution below?
header 1 | Very Undesirable | Undesirable | Neutral | Desirable | Very Desirable |
---|---|---|---|---|---|
Ability to flag outdated accepted answers manually. | |||||
Color-coding the outdated accepted answers. | |||||
Adding an “out-of-date” banner to outdated accepted answers. | |||||
Allow tags on answers to show versioning (ie python-3.9.0). | |||||
Creating a new ‘legacy answer’ status to preserve outdated accepted answers. | |||||
Create a new review process for updating outdated accepted answers. |
Coming soon: an exercise for flagging outdated accepted answers
In May, we plan to put temporary prompts on Stack Overflow that ask if the accepted answer is outdated and if the highest-scored, non-accepted answer is better than the accepted answer. The prompts will appear on questions where the accepted answer is 60 days or older and there are at least two answers on the post.
The data we gather will allow us to quantify how pervasive the issue of outdated answers is on Stack Overflow, get an initial read on whether sorting by score will be quantifiably better than pinning the accepted answer, and provide an initial training sample for a potential machine-learning algorithm.