86

As part of our Outdated Answers initiative, we shipped a temporary data-collection exercise in May. Unfortunately, the results were inconclusive and didn't lead to any big aha moments. After I briefly recap what we did (and didn't) learn, I'll update you on what we're doing next: sorting and labeling answers.

Flagging exercise results

From May to late June, we showed a prompt on Stack Overflow that allowed users to flag specific answers as being outdated, along with a reason why. We hoped we could use the data to quantify how prevalent the problem is, to inform decision making, and to provide an initial training sample for machine learning, but the data wasn't very compelling.

  • Users marked answers as outdated ~23,000 times. This was a low response rate: roughly 17 to 27 answers flagged out of every 100,000 answers.

  • There were no discernible differences between accepted answers (with the green checkmark), highest voted answers, and remaining answers. Accepted answers had the highest rate of outdatedness, but because it is always the top answer, we couldn't tease out the impact of position on page. In addition, we couldn't rule out that the differences were due to chance.

  • "Newer, more efficient answers exist" was chosen 45% of the time as the reason users marked an answer outdated. This was consistent with our March 2021 survey.

  • We compared answers that scored higher, lower, or equal to the accepted answer and didn't find any definitive differences.

  • We looked at recent upvotes/downvotes and found a weak association with outdatedness for votes cast within the last three months.

  • There was no strong relationship between question age, answer age, and outdatedness.

  • We found a low rate of accepted answers that were edited after they were accepted. Of the 3.7 million accepted answers in the data set, only about 4.9% were edited and only 9% of those edits were by the author.

Next steps: sorting and labelling

Despite the inconclusiveness of the study, we are making progress on a longstanding Meta request: to change the default sort so that the accepted answer isn't pinned to the top and answers are sorted strictly by score. 

Our plan is to test this out on a subset of questions where the highest scored answer is different from the accepted answer. We'll compare the upvote rate and the rate of users copying all or part of an answer between the two sorts: accepted answer first vs. highest scored answer first. Assuming there is no negative impact, we then plan to roll out the change. Look for more details when we get closer to launch.

In addition to unpinning the accepted answer, we are in the early exploration phase of:

  • developing a Trending sort that prioritizes recent upvotes more than older votes. 

  • placing a Trending indicator on answers that have recently received comparatively higher voting activity than other answers. 

  • making it possible to label individual answers with technology versions — for example, [python-v2] vs. [python-v3]. 

We've been getting user feedback on these early concepts by adding a handful of questions to our monthly Site Satisfaction Survey every few weeks. We will post more details on Meta when we are further along in the discovery process and seeking feedback.

17
  • 36
    Yea, i mean, it's odd how so few people actually perform curation actions. ¯\_(ツ)_/¯ If it's only shown to a small subset of users, that small percentage of users who do perform them will be even smaller.
    – Kevin B
    Commented Aug 18, 2021 at 15:56
  • 28
    I’m glad that so many of the community’s ideas are being considered: trending indicator and sort, version tagging, and of course unpinning the accepted answer. Keep up the nice work! Commented Aug 18, 2021 at 15:59
  • 11
    Good to see a status update, even if the results were largely inconclusive this time around. It's really nice to see that new approaches are being considered and experimented with!
    – zcoop98
    Commented Aug 18, 2021 at 16:05
  • 36
    Since that data collection thing started, I haven't seen the prompt even once and it has always bugged me. I'm wondering whether it was only shown to a predefined group of users (even if randomly selected) or it was shown at random to anyone? Or maybe there were certain criteria for the system to decide whether or not to show the prompt?
    – 41686d6564
    Commented Aug 18, 2021 at 16:22
  • 26
    "17 to 27 answers flagged out of every 100,000 answers" is a fairly meaningless measurement. The impact of one bad/outdated answer when it's on a really popular question with a huge number of views can be a lot more important to address than cleaning up 1k outdated answers on boring low view questions.
    – wim
    Commented Aug 18, 2021 at 17:55
  • 4
    "the rate of users copying all or part of an answer between the two sorts" What does this mean? You will have client-side code running to detect when someone copies code from a specific answer using Copy/Ctrl+V/etc. (presumably similar to the code used in the April 1st joke from this past year)?
    – TylerH
    Commented Aug 18, 2021 at 18:59
  • 3
    @41686d6564 It had a 1% chance to show for anyone on any pageload while live. So it there was a really small chance to see it.
    – TylerH
    Commented Aug 18, 2021 at 18:59
  • 13
    @TylerH Yes, we are repurposing some of the code from the April Fool's joke to detect CTRL-C on specific answers. It is a more common action than upvoting, so we wanted to look at both.
    – Anita Taylor StaffMod
    Commented Aug 18, 2021 at 19:06
  • 8
    Very interesting. We will be tracking 1. Accepted Answer 2. Vote Tally and 3. Text Grabs. So can we expect to start seeing people commenting things like: If my answer helped you, please give it the green tick, upvote it, and copy the text in my snippet. ?!? This should definitely spur people to add code snippets in addition to plain English text (unformatted text) into answers. Commented Aug 19, 2021 at 1:38
  • 5
    Outdated can be in the eye of the beholder. I was at a place using "classic ASP" as late as October of 2020...... Commented Aug 19, 2021 at 12:38
  • 8
    I'm don't think measuring copy/paste of an answer is a good quality metric. It could be an atrociously bad answer, but the only one the provides code someone can copy/past without thinking, instead of a better answer that explains concepts, pitfalls, etc. Commented Aug 19, 2021 at 15:50
  • 1
    For one thing I am glad that you realized that data you collected was not good enough and that you are not making assumptions and conclusions based on bad data.
    – Dalija Prasnikar Mod
    Commented Aug 20, 2021 at 11:46
  • 5
    @JosephDoggie Right. Classic ASP is certainly not an example of outdated software, given that Microsoft are supporting it until 2025. Questions on SO about rarely used software that was released a long time ago are certainly not necessarily outdated, especially if that software is still actively supported.
    – skomisa
    Commented Aug 23, 2021 at 19:24
  • 2
    You asked them a yes/no question, no? So how many people said "yes" and how many said "no"? You're ignoring a huge amount of data that may allow you to draw a meaningful conclusion when only considering how many people said "yes". The way you presented the data makes it seem like only up to 27/100000 of answers are outdated, but if only say 0.1% of people responded, that number could be more like 27/100 (unless the 100000 is actually the number of people who said either "yes" or "no"). Although this is also complicated by how many were shown the same answer.
    – NotThatGuy
    Commented Aug 25, 2021 at 20:16
  • 2
    @KevinB Curation (and editing and moderation) aren't rewarded in the same way that answering or asking are, so naturally there will be a much smaller base of people who perform those actions. I wonder if the subjects chosen for the study were such "community-oriented" users rather than just "reputation farmers". Commented Aug 30, 2021 at 8:15

6 Answers 6

65

I think it's important to couch the Outdated Answers Flagging data/results in the context of the scale at which the test was deployed. It was something like a 1% chance on any page load to see the feature, no? How many of the site's top 100 flaggers (who all have 7000+ helpful flags), for example, ever saw the feature? How many times was the feature exposed to users in total?

For example, I tried feverishly to please the random number gods and land a page load on a page with an outdated accepted answer and was never able to see the option. So, in regards to:

Users marked answers as outdated ~23,000 times. This was a low response rate: roughly 17 to 27 answers flagged out of every 100,000 answers.

This low response rate is because you hooked a refrigerator's ice maker hose up to the answers pipeline. I would suggest, if you want better/more conclusive data, that you run a second, wider experiment, this time with a fire hose (or no filter hose at all; turn it on for everyone [or everyone over n reputation], all of the time, for a few weeks).

I suspect most of this kind of flag would, over its lifetime, come from a certain subset of users (the kind that happens to spend a lot of time on Meta, perform user moderation, and have high reputation and/or a lot of answers). However, a 1% chance for all users means a lot of your 'Outdated' flags during this experiment probably came from people who will only ever cast one or two such flags in their lifetime, which has a compounding effect on artificially shrinking your results count.

11
  • 14
    We could make do with .015%, if we make the selection non-random ;) Commented Aug 18, 2021 at 23:26
  • 11
    I completely agree. I never even saw this feature, which I think is too bad. Commented Aug 19, 2021 at 8:07
  • 12
    I didn't get it once, either logged in or out. It's still disappointing that they only did a tiny subset that ends up excluding the active flaggers, and consequently skewing the stats in favor of no action Commented Aug 19, 2021 at 8:28
  • 3
    I never saw this either and I was looking for it.
    – user692942
    Commented Aug 19, 2021 at 9:58
  • 2
    I think the only conclusion that can be drawn from the collection method and way the results are presented is that The Company doesn't want high reputation / highly active users to be flagging outdated answers. Commented Aug 19, 2021 at 13:53
  • 15
    @IanCampbell That seems like an unnecessarily negative viewpoint to have.
    – TylerH
    Commented Aug 19, 2021 at 14:40
  • I always wondered if I could get higher OC rates by hooking my refrigerator up as a heat sink, but I never thought about the Ice maker in it!. Everything aside, I saw it on 1 post and sadly that one was probably the only not out-dated post I found... Commented Aug 19, 2021 at 15:03
  • 2
    I was going to suggest this feature for flagging outdated content, but I come here to find they did an experiment and found the data wasn't helpful? I never saw this feature and I'm one of the top % users on the main site (though I never flag because I have most of the privs I need already). They should reconsider testing this feature with a wider audience and get a proper sample size before considering solutions like not putting the accepted answer at the top on a Q&A site.
    – codewario
    Commented Aug 19, 2021 at 16:43
  • I would recommend repeating this experiment like you said but only after the changes in the question are implemented (or at least the accepted unpinning)
    – Dharman Mod
    Commented Aug 22, 2021 at 10:24
  • 4
    Also keep in mind that you’d have to see it on an answer that needed flagging as well, which makes it even less likely to be used. I saw it 4 or 5 times total. I was grateful to see it pop up for me on a question about fixing TLS cert verification errors where the accepted and highest voted answer was “just disable verification.” 😊
    – miken32
    Commented Aug 22, 2021 at 13:05
  • 1
    I would have liked to be able to flag this answer - already deprecated for 16 years when the answer was posted in 2013. Commented Aug 29, 2021 at 14:53
25

To be clear on why this was inconclusive, what was the expected population of action takers in marking questions as outdated, and how does that compare with the typical trend of users performing low-lift curation (up/downvoting), medium-lift curation (some queue access and some tag edits), and some heavy-lift curation actions (post closure as duplicates, routine queue access and post editing)?

2
  • Also, how much curation happens on older Q&As (where the option to flag outdated was available)?
    – VLAZ
    Commented Aug 18, 2021 at 16:26
  • @Makoto, we didn't perform analysis comparing the outdated answers flagging exercise to other curation activities. It's not really apple-to-apples given that the flagging exercise was sampled at 1%.
    – Anita Taylor StaffMod
    Commented Aug 18, 2021 at 19:08
19

Users marked answers as outdated ~23,000 times

This can be very good or bad depending on the other side of this number: how many eligible users were actually shown this facility and interacted with it (it could also be an usability issue).

Maybe, to give more context to that number, we need to ask other questions too: how many views/impressions had a eligible question? How many of those impressions were registered users? How many were of users that could downvote? Descriptive statistics between those marked as obsolete, not obsolete (tags, age)?

15

I think that the trending indicator is going to solve for a lot of cases where 2009 rears up and smacks you in the face with a fossil. In almost all cases that (very anecdotally) come to mind when it relates to security issues, there's almost always a new and better answer that folks in the [language] chat room have been trying to promote through voting on it.

If we can just catch that bit of signal and amplify it as needed, by letting folks drill right into trending answers as they emerge, then the technical part of the problem has a solid workaround.

What remains is what to do with the fossil, and I still think allowing folks with a silver or better tag badge the option to vote to deprecate it (turn the checkmark gray, treat it as a non-accepted answer and add an admonition that it was once a prized artifact and has been archived for posterity) and mark the question unanswered again is probably the way to go.

I like the idea of labels, and that makes a ton of sense for Python, but there's still an awful lot of language-agnostic barnacles clinging to the SS Stack Overflow :)

11
  • 5
    Took me way too long to realize you meant "Steam Ship" by SS and not some network site similar to Stack Overflow X_X
    – TylerH
    Commented Aug 18, 2021 at 18:56
  • 2
    @TylerH Steam Ship?
    – Sabito
    Commented Aug 18, 2021 at 19:34
  • 5
    @Sabito錆兎 Also took me a while to get it, but it makes sense: “SS Stack Overflow, “barnacles clinging” Commented Aug 18, 2021 at 19:37
  • 2
    @SebastianSimon SS, when used in front of the name of a seafaring vessel (like "the SS Stack Overflow") is short for Steam Ship (or in Steam boats, it's short for Steam Screw), referring to the method of power used to propel the ship forward. Same rough concept as "HMS Victory" or "USS Maine".
    – TylerH
    Commented Aug 18, 2021 at 20:42
  • 3
    On an off-note, I have to say that those of us from Europe got quite confused when we saw the "SS Stack Overflow" :) Should we start calling Prashanth "my führer"? Joking aside, the silver badge idea is spot on as it would enable a broad-ish category of curators to help deal with outdated content without being a fossil themselves (it is quite hard to to a gold badge in any given tag, even if it is popular)
    – 0Valt
    Commented Aug 18, 2021 at 21:59
  • 8
    Regarding "mark the question unanswered again"; I think we should stop calling a question with an accepted answer "answered" altogether. Just say "has an asker-accepted answer" or something. I think lessening the importance of the little green checkmark works in everyone's favor... If the purpose of Stack Overflow is to help more than just the OP, of course. If we're a help desk, then that green checkmark is vital. /s Commented Aug 18, 2021 at 22:32
  • 5
    @HereticMonkey I disagree with that. First, you can still provide an answer to an "answered" question. Second, we already differentiate questions on the question page between "has answers" and "has an accepted answer" via the green outline around the number of answers vs the green fill (and accompanying tooltips). To change it to "has an asker-accepted answer" just seems too contrived. Unpinning the accepted answer from the top gets us to where we want to be, I think.
    – TylerH
    Commented Aug 19, 2021 at 13:23
  • 5
    @HereticMonkey I've always felt like the whole accepted answer functionality could be better as far as the signal it sends. When we were a MUCH smaller community, everyone just basically knew it only meant that the person with the question had at least. tested the solution and felt that it worked best with their circumstances. By the time it became apparent that most new users didn't really understand that, the scale was colossally different, and much harder to change. I certainly agree it's not optimal, but I don't really know what would be, all things considered.
    – user50049
    Commented Aug 19, 2021 at 16:10
  • 2
    @TylerH I've been on the site longer than you; I'm aware of how the accepted answer works and the changes it makes. The changes are kind of the point, in fact, since modifying the question's appearance, even marking the answer with a green checkmark, makes it appear as if the question is "solved". If the checkmark were less heavily promoted (it was colored blue or gray instead of green, less or no change to the question in the list, accepted answers gained fewer points), we'd have a better chance at getting better answers than the FGITW ones we get now. I'm glad for the unpinning; it will help Commented Aug 19, 2021 at 17:09
  • 4
    @HereticMonkey There's no need to be rude or throw irrelevant details like account age around as some demonstration of understanding or expertise. That aside, your comment seems to continue to miss numerous aspects of UX. But feel free to post a feature request asking for the check mark color to be changed. Maybe they will listen to you.
    – TylerH
    Commented Aug 19, 2021 at 17:54
  • The problem I have with accepted answers is that they have to be accepted by the OP. If a newb asks a question, gets a good quick answer and goes away happy, but never comes back to say "yeah, that worked", then the question will sit there "unanswered" forever. It could pick up thousands of upvotes from later readers and still not be "answered". It would be helpful if there was a mechanism that an answer over a certain age and with sufficient upvotes could be tagged as accepted by some level of respected member. Or even automatically.
    – Sinc
    Commented Aug 26, 2021 at 20:47
5

Why not just give us the option to unpin the accepted answer? Just like the Active, Oldest, Votes options, add a checkbox that says "pin the accepted answer". It might or might not be checked by default, you can experiment with that as much as you like. Why does it have to be a big deal? I imagine the wast majority of the users who want the option don't care if it "increases the user copy rate" or whatever, they just want to be able to unpin the accepted answer however it affects "an average user".

4

Users marked answers as outdated ~23,000 times. This was a low response rate: roughly 17 to 27 answers flagged out of every 100,000 answers.

23k outdated answers sounds like a lot, but <30 per 100,000 answers outdated sounds like this is no problem at all. If this is true, >99,970 out of 100,000 answers aren't outdated. No need to do anything about it.

So either there is no problem or there is a serious flaw in the experiment and the results and shown lack of correlations are not trustworthy. If we assume that more than 30 answers per 100,000 are outdated, you may have missed most of them and therefore also the important correlations. But why did you miss them? There is no discussion of that in this summary. Maybe recognizing if an answer is outdated is a hard problem and cannot easily and reliably be done by most users? If this would be the case even going for trending answers might not be enough (people might still simply upvote what is already on top). Is there maybe a way to find out, how people would vote if there was no score shown and the answers would be in a random order? (Think of it as some kind of reset to study outdatedness.)

Regarding:

We'll compare the upvote rate and the rate of users copying all or part of an answer between the two sorts: accepted answer first vs. highest scored answer first.

Why so complicated? Just treat acceptance as a kind of supervote with a slightly higher weight than a normal vote and be done with it. Rates will have large uncertainties at low scores. There are millions of questions and answers with score <10.

...developing a Trending sort that prioritizes recent upvotes more than older votes.

Sounds good. Hopefully it comes soon. Question is what will be the default sort order. I would even recommend the age-weighted score as a default sort order.

...making it possible to label individual answers with technology versions

Good idea but requires lots of discussion before to get it right because while beneficial also requires curation. Best would be to discuss that with the community as early as possible.

Regarding the initial problem:

If we have a problem with outdated answers and if most people aren't good in detecting if an answer is outdated, even the trending sort and the unpinning of the accepted answer (while both will be helpful) might not solve the problem with outdated content. The best hope seems to be the version tagging, but that requires careful planning to get it right.

Make more experiments to establish how reliable people can recognize outdated content. Get version tagging right. Think about other (better) solutions to the problem of outdated content (probably depends on how outdated content is recognized).

1
  • 1
    "you may have missed most of them and therefore also the important correlations. But why did you miss them?" Because the experiment only shown the box for 1% of the eligible posts (or users?).
    – Braiam
    Commented Aug 23, 2021 at 14:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .