423

In our blog post last month on our Community & Public Platform strategy & roadmap for Q1 2021, we announced an initiative to address outdated answers on Stack Overflow. Today, I'll give you more information on our approach and how we will solicit your feedback.

Why outdated answers? Why now?

Finding correct and complete answers is the reason users visit Stack Overflow. The vast majority of our users come from organic search by googling their issues and clicking to Stack Overflow to find answers. On top of that, there is a large and passionate volunteer community — I'm looking at you, Meta — that cares deeply about ensuring that the question-and-answer artifacts on the site are of the highest quality.

Yet, we have heard through a variety of feedback mechanisms — from Meta to internal staff — that outdated answers are an issue. According to our Site Satisfaction survey, 13.8% of users say that answer quality is one of the top things that they find most frustrating/unappealing about the site.

If Stack Overflow users can't quickly find working solutions to their problems, this is kind of an existential crisis. Good answers to questions is fundamentally the value that we provide to the developer community.

So the why is pretty obvious. But why now vs. why not yesterday vs. why not eight years ago?

The first thing we did to kick off the project was read through Meta. It quickly became apparent that this is not a new issue. There are many discussions, going back many years, with many thoughtful suggestions about potential solutions. (If you're curious, you can find a subset of relevant Meta posts at the bottom of this discussion.)

I can't talk about why we didn't prioritize it then because I wasn't on the team — but I can speculate. It's a hard problem, and hard problems are easy targets for procrastination. And the urgency of solving it for a two-year-old site or a six-year-old site is different from the urgency of solving it for a 12-year-old site. And, frankly, for a time, Stack Overflow and the rest of the Stack Exchange sites were under-resourced from a product-development perspective.

But the past is the past. This year, we decided that it's time. Stack Overflow's senior leadership team laid out our 2021 objective to measure and improve overall community and content health. And the Public Platform team decided that one of the best ways to meet that objective is to focus on keeping content as accurate and up-to-date as possible.

We're starting with product discovery — and listening to you

Rather than jumping straight into implementing some of the solutions you've proposed, we are focusing initially on product discovery. We want to make sure we:

  • understand the problem really well, so we can build the right solutions.

  • attempt to quantify the problem using a data-driven approach.

To do this, we will be reaching out on Meta during discovery to let you know what we're working on and to ask for your feedback. No one knows the problem better than you, so we are all ears.

When we get to the solution phase, we will conduct UX research, share early designs, keep you abreast of experiments and let you know what's launching when.

How do we define outdated content?

Our first step will be identifying, with your help, what constitutes an outdated answer and what type of outdated answer is most prevalent on our site. So far, Meta has told us there are different flavors of outdated answers. There are answers that:

  • become obsolete as new versions of frameworks become available.

  • still work, but there is now a newer, better way to achieve the same end.

  • maybe were never the best, but the question askers accepted them and forever bestowed the green check mark.

  • now pose security risks or provide code that no longer works.

  • still hold value to a subset of developers who use legacy technology, but aren't valuable to developers on newer technologies.

We will be gathering this information by adding a few questions to our site satisfaction survey. Check out this post where we ask for your input on the proposed questions.

How will we surface outdated content?

Identifying stale or outdated content is a hard problem to solve, and we know there isn't a silver-bullet solution here. It will likely require a mix of automation and user intervention. But we hope to identify some heuristics that make it easier to narrow down the universe of possibly outdated answers.

We're in early discovery, so we don't have a solution defined yet on how we'll surface outdated content. However, once we have a defined list of use cases, we want to explore ways in which we can get help from the community to label content as outdated so that we can begin training a machine-learning model as a complement to manually flagging content.

There is a lot of nuance to each answer and the amount of expertise required to evaluate answers for any given tag or technology. Leveraging the community's collective knowledge here will help us get the best data set we possibly can.

As with any training exercise, failure is always a possible outcome. Either way, we'll share our learnings along the way.

We are tackling Stack Overflow first

We are focusing our initial research efforts on Stack Overflow, and not on other sites on the Stack Exchange network. We understand that the issue with outdated answers is not exclusive to Stack Overflow and other technical sites may have similar issues.

However, none of the other sites operate at the scale of Stack Overflow: more than 31 million answers, compared to 11 million for the rest of the network. And Stack Overflow is our oldest site, so it is the site where the pain is most acute.

Rather than trying to shoot for a one-size-fits-all solution that may actually fit none, we are laser-focused on uncovering the Stack Overflow use cases and coming up with initial solutions that work really well for Stack Overflow.

Stay tuned for other activity TBD

We are currently mapping out other discovery tasks we will undertake in the near future, and we will reach out to Meta when we have new updates and requests. Our future posts will have the , , and tags.

If you want to be considered for targeted surveys, usability tests and focus groups, please visit your Email Settings and opt-in to Research. This ensures that you are in the pool of people we contact when conducting UX research.

Research opt-in picture

Meta literature review

There are many posts on the topic of outdated answers, but here are some of the ones we discovered while doing our initial research.

Any initial feedback?

We're not locking in on any particular solutions yet, as we are more focused on understanding the what and why vs. jumping straight into the how. But we welcome any initial thoughts and reactions regarding the project overall, past experiences, or ideas you have about ways to attack this problem as answers below.

51
  • 220
    Nothing is ever obsolete, there's always someone out there working with tech that's old because they have to... I look forward to seeing what potential solutions you come up with that can get the right answers to the right people Commented Feb 18, 2021 at 16:42
  • 58
    Looks like some good research was done here, nice to see! Here's another discussion for the pile - arguably one of the first BIG discussions on the matter: meta.stackexchange.com/questions/103053/…
    – Shog9
    Commented Feb 18, 2021 at 16:57
  • 46
    Any initial feedback? Yes: Wow! This is gonna be a Herculean task, as you clearly understand, but it's absolutely fantastic that S.E. Inc. is embracing the problem. Can I create some sock-puppet accounts, so I can upvote this post some more? Commented Feb 18, 2021 at 17:06
  • 51
    Love the tone and style of this communication :-) Thank you.
    – QHarr
    Commented Feb 18, 2021 at 22:39
  • 37
    @Nick: I think you're conflating "obsolete tech" with "obsolete answers". Answers pointing out the lack of a solution to a problem can be rendered obsolete by the introduction of new tech that either purposefully or incidentally solves the problem. Of course, the original content would remain relevant to legacy environments that a reader might rely on. But they'd still need to be updated with the new information.
    – BoltClock Mod
    Commented Feb 19, 2021 at 6:08
  • 13
    My instinct is that the machine learning approach is unlikely to succeed, there's surely far too much variation in what defines an obsolete answer to get anything like a useful set of training data? Hopefully the machine learning bit will be a nice-to-have stretch goal, rather than something that serious time is invested in, before quicker wins are accomplished.
    – DPWork
    Commented Feb 19, 2021 at 8:10
  • 9
    I always liked the sorting by age-weighted votes a lot but it was never implemented. In the simplest variant one could for example just sort by the votes in the last X months. I really hope it may get a chance this time. Commented Feb 19, 2021 at 9:09
  • 24
    @BoltClock "Answers pointing out the lack of a solution to a problem can be rendered obsolete by the introduction of new tech" - That only makes the answer obsolete to people who have access to said tech, the answer is still relevant to those who do not Commented Feb 19, 2021 at 11:37
  • 8
    If quickly finding solutions to problems is the challenge, I wonder if the search functionality of SO should also be part of the discussion? Or is the assumption that people use google and arrive at the right questions and then only need to find the right answer within the right Q&A pair? Commented Feb 19, 2021 at 12:02
  • 12
    I am absolutely stoked about how you're tackling this task. That was such a great write-up, and it's really nice that you're coming to the community early in the process. Honestly it feels to me like you've thought of everything I could think of suggesting. Even your category break-downs are excellent. Thank you. Commented Feb 20, 2021 at 7:28
  • 21
    "...there is a large and passionate volunteer community — I'm looking at you, Meta — that cares deeply about ensuring that the question-and-answer artifacts on the site are of the highest quality." I just want to point out that the large and passionate volunteer community does not 100% intersect with the Meta community. I come to Meta when I see an interesting Hot Post or whatever on the side bar, but I rarely participate more than an occasional comment or vote. However, I'm very committed to the quality and purpose of main. Please make sure you reach out in other ways besides Meta posts.
    – MattDMo
    Commented Feb 21, 2021 at 20:30
  • 9
    Whatever happened to the "get rid of crappy questions project"?
    – mxmissile
    Commented Feb 23, 2021 at 15:38
  • 22
    Could we rename outdated to legacy. Many of the answers which reference old versions of software are very handy for maintenance of legacy systems. It is not uncommon that systems run on legacy software or have been written in it and have been designed 20+ years ago. These "outdate" answers are not outdated, they are extremely relevant for such a systems. Legacy might be a better name here.
    – kvantour
    Commented Feb 24, 2021 at 6:56
  • 9
    @kvantour Anita is already clearly differentiating legacy answers from other kinds of outdated answers, in the section How do we define outdated content? Yes, we want to keep those (useful) legacy answers, but we want them to be easier for people using legacy systems to find, but also we don't want those answers clogging up searches for the majority of people who are looking for newer answers. Of course, what counts as legacy (and how best to deal with it) varies, depending on the language / framework.
    – PM 2Ring
    Commented Feb 24, 2021 at 11:53
  • 9
    @gmoniava "...would be unfair to get downvotes to answers that were once correct." Why? Votes just means something is useful or not useful now. The past should play no role there. Voting is all about signalling to future visitors. Commented Feb 25, 2021 at 22:58

37 Answers 37

1
2
-1

I like Trilarion's answer. But it may be complicated to understand because people do not see the votes and their age.

I have originally thought of:

  • a button on the answer that the author can mark his answer as obsolete

  • such an answer will automatically appear at the bottom

The problem is that most authors do not care about their old answers. Stack Overflow is community driven so such functionality might work similarly to Close votes. If enough people having sufficient privilege mark the answer as obsolete, it would get some "tag" that it is deprecated and move to the bottom.

1
  • 2
    I care about my old answers, but I’m not going to spend time reviewing all of my almost 400 answers every now and then. I wish there was a feature with which you could mark a portion of an answer which is obsolete and write a comment explaining exactly how it’s obsolete; when updating the answer, you could then remove the comment (similar ideas have been proposed earlier). Often you find old answers with lots of comments; the comments criticize various aspects of the answer, the answer has been edited many times, but you’re not always sure if each edit has addressed each comment’s concerns. Commented Mar 12, 2021 at 3:37
-1

My solution for languages with version tags, like C++, would be to have an A.I. tag all answers with the version of C++ that is being used - and then allow users to sort by that tag: the latest C++ version first, then the next etc. And then allow another ordering within each version (either highest up votes, or newest answer - just like we can do now).

For example, I'd set the sorting/filtering to show C++20 at the top, then C++17, then C++11, each sorted by number of upvotes. While currently I am forced to simply sort by newest answer, ignoring the up votes thus, in order to find if there are any C++20 specific answers.

4
  • 3
    You can't throw an AI at a problem and expect it to work. What if an answer doesn't contain version-specific code, or the only difference between versions is functional? Also, the first half of this question is off-topic (rambling/ranting), so that's not adding value to your answer. I've edited that out of the answer.
    – Cerbrus
    Commented Apr 20 at 15:33
  • 2
    So, a simple plan: 1. Create versioned tags 2. ? 3. Profit.
    – VLAZ
    Commented Apr 20 at 15:46
  • LLM-based AI can perform some code analysis tasks well. But the problem is that if it doesn't detect a real pattern it's extremely likely that it will hallucinate a bogus pattern. You can mitigate that to an extent by training, but that requires input from human experts. That sort of training task can be rather gruelling, and few experts are likely to do it voluntarily.
    – PM 2Ring
    Commented Apr 20 at 16:43
  • @Cerbrus What I mean is that if code can't compile unless you use C++20, then it is labeled c++20. Current A.I. is stupid, but this can be automated. Maybe not by using an LLM, but at the least automated (for the existing answers). In terms of sorting the answer, we need a mix between sorting by newest answer, and highest up votes: the problem is that a recently added solution with a much lower number of up votes might be the by far the best solution. I'm all for sorting by number of up votes, but then I want old answers that use an old version to moved down.
    – Carlo Wood
    Commented May 4 at 12:44
-4

You ask for feedback. I came here because I was wondering what to do about a question (Getting fb.me URL) the accepted answer to which is now mostly incorrect. Once I got here, my first question/bit of feedback is around how we should collect feedback indicating that (due to the technological progress or the passage of time) some content (e.g. answers) are outdated. This is closely connected to defining and surface outdated content, but NOT the same thing...

There's no way for one to just provide this feedback*.

As far as I know, we don't have a defined best practice [clarification: that applies once we have the desired code functionality]. Perhaps a top priority for the community is to clarify best practice for what we do after we find the obsolete content that it's most important that we update. (Certainly how often its viewed should factor in!) The "Meta literature review" has good pointers, especially to What's the etiquette on updating an accepted answer? - but it's most valuable feedback is that the software needs tweaking to allow us to handle this situation better.

*Not good: Downvoting the obsolete answers to this question...

We COULD provide a better answer or edit the outdated one(s) (like by adding a header) to include the fact that it's obsolete. Sorry for any redundancy or repetitiveness.

2
  • 3
    Your answer isn’t that clear as to what exactly your suggestions are or what exactly your feedback is. To summarize, you’re saying that we need better tools and defined processes to “handle” outdated answers? But, as you said, that’s what we’ve already established in the “What’s the etiquette” question. Since then we’ve had a flagging exercise, the accepted answers have been unpinned, an age-weighted answer scoring is planned, etc. The tools are being discussed and made; we’re getting there. Do you have any specific feedback to those? Commented Nov 20, 2021 at 2:08
  • 1
    Not quite what I was saying. I'm saying saying that what we’ve already established in the “What’s the etiquette” question is that the practice is says is good for now if we don't have code changes, isn't very good. I was responding to the OP(it was last edited Mar 12 - so it's outdated a bit?), not up on the situation changes since . Let me read the flagging exercise stuff; again sorry for any redundancy or repetitiveness. Commented Nov 20, 2021 at 2:26
-6

The "Deprecate" Mechanism

Maybe we need a way for users to deprecate an answer because it is of lower quality than it used to be. Deprecating is similar to downvoting, but with a different meaning. It means the answer was once good but has become less valuable because the scenario posed by the question has been superseded by later releases of the related product.

In addition, maybe the deprecator could link to a newer, better answer.

Deprecation could eventually result in removal by cleanup people, or in transfer to some kind of attic or museum. Hopefully, the search engines could be given info to stop sending new searches to this answer.

Just an alternative.

2
  • 11
    Old does not equal lower quality! Many of us are stuck on legacy systems and still need the accepted answer to a question for an older version because it will solve our problem.
    – cb4
    Commented Feb 28, 2021 at 5:14
  • Agreed. That's why deprecated and downvoted are different. Maybe deprecated is the wrong word. Commented Feb 28, 2021 at 8:57
-8

According to data.stackexchange.com, there are 23M questions, 34M answers and 88M comments (some comments may contain answers) in round numbers about 70M artefacts that need surfacing for outdatedness. This is only going to be done automatically.

4
  • 5
    The auto tagging idea based on text mentions is a terrible one. For example: A front end web framework could be mentioned by name as being responsible for sending data to a server side back end but would be completely irrelevant if the issue is only back end related.
    – charlietfl
    Commented Mar 7, 2021 at 16:01
  • 2
    Yes, but there is also an extreme power law at play. Even without the extreme duplication, plagiarism, and now ChatGPT, 99.99% of the value is probably in 0.01% of the content. Commented Dec 21, 2022 at 1:36
  • Looked at as a whole this is all statistics, no one person knows the correct answer. A SME might know in their own field but a machine is only ever going to flag potentially out of date answers. I can not think of an example of something that is definitely out of date.
    – MT1
    Commented Dec 21, 2022 at 7:18
  • This - meta.stackoverflow.com/questions/411662/…
    – MT1
    Commented Dec 21, 2022 at 7:19
-10

This new tag on Meta.SO may be needed to tag this and similar discussions:
- discussions of outdated questions and answers.

Note that this single tag may be better than 2 separate tags for Meta purposes:
and

4
  • Either of these 3 tags exist... And this is about outdated answers anyway...
    – Tomerikoo
    Commented Feb 18, 2021 at 17:34
  • 3
    @Tomerikoo Did you mean: neither of these 3 tags exist? That's what I see when I click on these links... And that's the point of my answer: we need a tag for this, preferably a single one. Commented Feb 18, 2021 at 18:46
  • 1
    Meh, the answer to a problem on meta (or main) is rarely "create a new tag". Also, I'm not sure how this would help identify outdated answers on main, which is what this meta question is about... Commented Feb 19, 2021 at 14:45
  • 2
    Surely there will be new types of disputes on meta appearing. "Why was my answer marked as outdated?" Meta tags are part of the infrastructure for this. Commented Feb 20, 2021 at 9:09
-29

My personal experience is, that if I am on Stack Overflow as an answer hunter or question asker (typically coming from a google link), in about 10% of the cases am I here for some not-current material. If I could not find the information I need here, I would go elsewhere.

I also don't think that the quality of the content would be a big problem - not because it would be good, but because you are still the best on the Internet.

What I find the most annoying on the Stack Overflow, is that on many posts is it clearly visible that the OP is simply incapable/unwanting to formulate a round text even on his first language. No, it is not an English ("cultural diversity") problem; it is a functional illiteracy problem. I think this is your largest problem. The second is the inadequate reviewing, both in quality and quantity, on the Stack Overflow. The third is the open hostility of the meta sites and the reviewers. But these are another story. I am sure, this is what most of your 18% thought, and not on the obsolete answers.

That happens much more rarely, that I am coming here for some current technology, and I need to dig out what I want from a pile of obsolete answer(s). Yes, sometimes it happens (last time here), but I think the problem is not serious yet. Although it will obviously worsen with time, it could be useful to handle it in time.

I am sorry to say, but as I learnt you in the past decade, I think I have all the reason to foresee a mass deletion instead of some improvement. This would be unthinkably evil and harmful, not only for the Internet in general, but also for you (SE LLC)!

On this reason, I try to draw your attention to the other, destruction-free options. This is a brainstorming, and it is up to you where you go with it. Maybe you could use a combination of these for the different types of the obsolete content.

Noticing obsolete answers and pushing them to the end in the answer list

Answers could get a new flag option, "This answer is obsolete". This would put them to the VLQ review queue. If it gets enough votes (exact details are up to you), then @Community would put a "This answer is obsolete" notice to the answer, instead of deleting it. These answers would go automatically to the end in the default answer list ordering, independently from their score.

Migration to the Retrocomputing Stack Exchange site

If not only a single answer is obsolete, but a whole question, then the good news is that you already have a site, exactly for such questions. It is still far away from the average graduation, but not very far. While the exact criteria of a site graduation are not clear to me, a quick scan of your site list shows that the median question count of a site graduation is at approximately 10,000 questions. The Retrocomputing Stack Exchange site now has approximately 4,000.

Although questions older than two months are not migratable, as far I know, CMs are exempt from this restriction. And, in the case of such a major change in your system, maybe the change of the rules would be not impossible (the simplest option: allow community migrations to the Retrocomputing SE even if they are older than 60 days).

Just tag them

Another option to handle obsolete questions is just tagging them. It could be extended with a top search top bar improvement, which would enable us (visitors) to search only in the not obsolete (not [obsolete]-tagged) content.


In general, the capability to make such a big content deletion, is a big power and I don't believe that anybody could use it correctly. Thus, please

  • try to prefer the community decisions to the single-person decisions,
  • and try to prefer the non-destructive actions to the deletions,

in the process you are likely thinking on.

11
  • 34
    I can't call a legacy language from 2004 "Retrocomputing", and I don't think they'd be very appreciative of us dumping old content on their heads like that.
    – Makoto
    Commented Feb 18, 2021 at 22:21
  • 1
    @Makoto I can't understand. Please formulate more clearly.
    – peterh
    Commented Feb 18, 2021 at 23:04
  • 28
    It seems a little ironic for you to be criticising the communication skills of question askers. As I've mentioned elsewhere, I often find it hard to understand your posts because of the way you use English. I think I generally get the main points you're trying to make, but I'm rarely sure about the subtleties and the connotations.
    – PM 2Ring
    Commented Feb 19, 2021 at 4:59
  • 3
    @PM2Ring I think both of us know that not this is the problem.
    – peterh
    Commented Feb 19, 2021 at 9:16
  • 19
    Seriously, I don't know what that sentence is supposed to mean.
    – PM 2Ring
    Commented Feb 19, 2021 at 9:18
  • 1
    @PM2Ring Of course. ;-)
    – peterh
    Commented Feb 19, 2021 at 9:19
  • 8
    (Can you rephrase the paragraph containing "obsolete answer(s) fade what I want to know, happens much more rarely." (close to incomprehensible)?) Commented Feb 19, 2021 at 14:44
  • 7
    While I think you are certainly right about functional illiteracy, the egotistical minimum effort attitude (absolutely zero regard for current and future readers) is likely the dominant reason (where even using the Shift key is too much effort). Lack of skills is a convenient excuse. Commented Feb 19, 2021 at 14:45
  • @PeterMortensen Ok, I tried.
    – peterh
    Commented Feb 19, 2021 at 15:44
  • 4
    I have those same fears that this will result in a lot of deletions. Not sure about the "VLQ review queue" (Low Quality Posts queue) though. It seems everything is deleted there.
    – Scratte
    Commented Feb 19, 2021 at 15:51
  • 2
    @Scratte I watch closely the company & community behaviors since many years. I give about 45% that nothing will happen and the topic will be silently forgotten. I give another 45% that the company will invent some new rules enabling a mass deletion rampage. I give 9% that they will do some cosmetical changes what will be not insane. I give 1% that they will really improve anything.
    – peterh
    Commented Feb 19, 2021 at 18:40
1
2

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .