In our blog post last month on our Community & Public Platform strategy & roadmap for Q1 2021, we announced an initiative to address outdated answers on Stack Overflow. Today, I'll give you more information on our approach and how we will solicit your feedback.
Why outdated answers? Why now?
Finding correct and complete answers is the reason users visit Stack Overflow. The vast majority of our users come from organic search by googling their issues and clicking to Stack Overflow to find answers. On top of that, there is a large and passionate volunteer community — I'm looking at you, Meta — that cares deeply about ensuring that the question-and-answer artifacts on the site are of the highest quality.
Yet, we have heard through a variety of feedback mechanisms — from Meta to internal staff — that outdated answers are an issue. According to our Site Satisfaction survey, 13.8% of users say that answer quality is one of the top things that they find most frustrating/unappealing about the site.
If Stack Overflow users can't quickly find working solutions to their problems, this is kind of an existential crisis. Good answers to questions is fundamentally the value that we provide to the developer community.
So the why is pretty obvious. But why now vs. why not yesterday vs. why not eight years ago?
The first thing we did to kick off the project was read through Meta. It quickly became apparent that this is not a new issue. There are many discussions, going back many years, with many thoughtful suggestions about potential solutions. (If you're curious, you can find a subset of relevant Meta posts at the bottom of this discussion.)
I can't talk about why we didn't prioritize it then because I wasn't on the team — but I can speculate. It's a hard problem, and hard problems are easy targets for procrastination. And the urgency of solving it for a two-year-old site or a six-year-old site is different from the urgency of solving it for a 12-year-old site. And, frankly, for a time, Stack Overflow and the rest of the Stack Exchange sites were under-resourced from a product-development perspective.
But the past is the past. This year, we decided that it's time. Stack Overflow's senior leadership team laid out our 2021 objective to measure and improve overall community and content health. And the Public Platform team decided that one of the best ways to meet that objective is to focus on keeping content as accurate and up-to-date as possible.
We're starting with product discovery — and listening to you
Rather than jumping straight into implementing some of the solutions you've proposed, we are focusing initially on product discovery. We want to make sure we:
understand the problem really well, so we can build the right solutions.
attempt to quantify the problem using a data-driven approach.
To do this, we will be reaching out on Meta during discovery to let you know what we're working on and to ask for your feedback. No one knows the problem better than you, so we are all ears.
When we get to the solution phase, we will conduct UX research, share early designs, keep you abreast of experiments and let you know what's launching when.
How do we define outdated content?
Our first step will be identifying, with your help, what constitutes an outdated answer and what type of outdated answer is most prevalent on our site. So far, Meta has told us there are different flavors of outdated answers. There are answers that:
become obsolete as new versions of frameworks become available.
still work, but there is now a newer, better way to achieve the same end.
maybe were never the best, but the question askers accepted them and forever bestowed the green check mark.
now pose security risks or provide code that no longer works.
still hold value to a subset of developers who use legacy technology, but aren't valuable to developers on newer technologies.
We will be gathering this information by adding a few questions to our site satisfaction survey. Check out this post where we ask for your input on the proposed questions.
How will we surface outdated content?
Identifying stale or outdated content is a hard problem to solve, and we know there isn't a silver-bullet solution here. It will likely require a mix of automation and user intervention. But we hope to identify some heuristics that make it easier to narrow down the universe of possibly outdated answers.
We're in early discovery, so we don't have a solution defined yet on how we'll surface outdated content. However, once we have a defined list of use cases, we want to explore ways in which we can get help from the community to label content as outdated so that we can begin training a machine-learning model as a complement to manually flagging content.
There is a lot of nuance to each answer and the amount of expertise required to evaluate answers for any given tag or technology. Leveraging the community's collective knowledge here will help us get the best data set we possibly can.
As with any training exercise, failure is always a possible outcome. Either way, we'll share our learnings along the way.
We are tackling Stack Overflow first
We are focusing our initial research efforts on Stack Overflow, and not on other sites on the Stack Exchange network. We understand that the issue with outdated answers is not exclusive to Stack Overflow and other technical sites may have similar issues.
However, none of the other sites operate at the scale of Stack Overflow: more than 31 million answers, compared to 11 million for the rest of the network. And Stack Overflow is our oldest site, so it is the site where the pain is most acute.
Rather than trying to shoot for a one-size-fits-all solution that may actually fit none, we are laser-focused on uncovering the Stack Overflow use cases and coming up with initial solutions that work really well for Stack Overflow.
Stay tuned for other activity TBD
We are currently mapping out other discovery tasks we will undertake in the near future, and we will reach out to Meta when we have new updates and requests. Our future posts will have the product-discovery, answers, answer-quality and discussion tags.
If you want to be considered for targeted surveys, usability tests and focus groups, please visit your Email Settings and opt-in to Research. This ensures that you are in the pool of people we contact when conducting UX research.
Meta literature review
There are many posts on the topic of outdated answers, but here are some of the ones we discovered while doing our initial research.
Please unpin the accepted answer from the top (Score: 555)
Introduce an “Obsolete Answer” vote (Score: 542)
How do we encourage edits to obsolete/out of date answers? (Score: 295)
How to deal with hugely upvoted, bad and outdated answers? (Score: 163)
Let's move some negatively scored answers from the top spot (Score: 107)
Can we exempt downvoted accepted answers from getting the top spot? (Score: 102)
Implement a vote-based warning system for potentially dangerous answers (Score: 90)
Keeping special status for Accepted Answers without sticking them to top forever? (Score: 78)
Will people use Stack Overflow less often if the information found there isn't 100% accurate? (Score: 74)
Deemphasise the accept mark if there's an answer the community strongly prefers? (Score: 65)
What's the etiquette on updating an accepted answer? (Score: 63)
Any initial feedback?
We're not locking in on any particular solutions yet, as we are more focused on understanding the what and why vs. jumping straight into the how. But we welcome any initial thoughts and reactions regarding the project overall, past experiences, or ideas you have about ways to attack this problem as answers below.