14

That last meta post about the outdated answers project got me thinking. It mostly deals with identifying outdated content as well as what possibly can be done about it (including unpinning accepted answers, adding version (range) tags, adding warnings,...). The underlying concern is that the content of Stack Overflow ages and therefore quality and usefulness of the content and the order/way in which it is presented is reduced over time. People might not be able to quickly find working solutions to their problems anymore.

I wonder how we could best measure the success of any action that might be taken to do something about the problem of answers becoming outdated?

  • Visits to Stack Overflow might surely be an important indicator, but it's very general and may change only slowly. The signal that is contained might be drowned by other influences.

  • Time spent on pages might not be clear-cut enough. A longer time might indicate that more good content was found or that the search to find the good content took longer. A shorter time might indicate that good content was found quicker or that no good content was found at all.

  • Ideally, you would find only useful answers among the top sorted answers or enough guidance to find useful answers and in the end, the useful answer (s) get upvoted. An upvote for an answer that is not shown on top of the sort order might indicate a miss sorting or it might indicate a new answer rising up to the top or that you had a legacy problem being solved by a now outdated answer, again maybe not clear-cut enough.

  • There could be surveys like "have you noticed new feature X? Do you think it improves finding answers?" Or similar but then we would go away from hard numbers to interpretations. Might though be the most practical thing possible?

I'm running out of ideas here, so I ask the community how would you convince somebody with a suitable metric that a new feature X (unpinning of accepted answers, adding version (range) tags, adding warnings to answers, ...) really improves the ability of visitors to quickly find working solutions to their problems?

Searching for "measure success of" or "metric of" and "find the right answer" or "solve the problem" on meta didn't result in anything relevant regarding how to measure the effectiveness of actions.

9
  • 5
    I am afraid that it will be impossible to find any good metrics. None of currently available user actions or data will give you actionable data. Visits and even upvotes don't mean a thing. I also wouldn't count downvotes as good indicators, unless you have many of them - careful (avoiding reversal scripts) revenge downvoting of good and valid answers is a real thing.
    – Dalija Prasnikar Mod
    Commented Feb 22, 2021 at 12:11
  • 1
    @DalijaPrasnikar We could maybe ask people how satisfied they are with their search and then maybe do A/B testing. Like presenting two different default sorting orders and then asking people if they found what they were looking for after being X seconds on a page and then displaying the rate of people having found what they were looking for depending on A/B? Possible metrics for this question should not only include the currently available user data but also what could be gathered by the company within a reasonable amount of time and energy. Commented Feb 22, 2021 at 12:48
  • Use the same measure by which they decided to embark on the project.
    – philipxy
    Commented Feb 22, 2021 at 13:00
  • @philipxy I think they mentioned survey results and that some people complained about answer quality. I wonder if surveys are really a good way of measuring the success of an action? Getting to know the problems is one thing, judging the effectiveness of solutions may be another. But on the other hand if a value in a survey outcome is the best, we could simply vote on proposed solutions. So popularity of an action would best estimate future success of it? Commented Feb 22, 2021 at 13:10
  • 1
    @DalijaPrasnikar The alternative would be things get done because a single person (the CEO) or a couple of persons (the team or the meta crowd) liked it. This is just a try to make the process more data driven. I could even play the devils advocate and say that outdated answers are not a problem at all. Yes, content ages but we have voting and visitors may be smart enough to see what technology is used in an answer. So show me that there is a problem first that needs solving. Where specifically is the "kind of an existential crisis" SO speaks of? Commented Feb 23, 2021 at 9:45
  • @philipxy You're right. A metric that shows the problem probably also shows if a solution is working. However, I'm not convinced that survey results saying answer quality is low is directly related to outdated content. Answer quality can be low for lots of other reasons too. Maybe they had additionally other measures that they used internally but didn't present publicly yet. Commented Feb 23, 2021 at 9:47
  • 2
    In general I think outdated content issue is more of a general quality problem SO faces. Reducing inflow of poor questions that often yield poor answers is far greater priority. If that can happen, it would be automatically easier to find good content and answers and that would also provide more opportunity to properly mark outdated answers and provide updated and new answers.
    – Dalija Prasnikar Mod
    Commented Feb 23, 2021 at 10:25
  • 2
    A lot of talk about outdated answers, and I'm yet to see at least one decent example. From personal expertise: I know that [c#] evolved a lot, but many so called outdated answers can be fixed by adding language version (e.g. [c#-4.0]) to the question. same with many .net technologies. Instead I often see how people add complicated answers with modern syntax, but claim they are the best
    – ASh
    Commented Feb 23, 2021 at 18:01
  • 1
    "I'm not convinced that survey results saying answer quality is low is directly related to outdated content." But it's the survey (or whatever "measure by which they decided to embark") that matters, not "outdated" posts! XY. Dealing with "outdated" posts isn't being done for its own sake. It's a solution to a prior problem. (And I don't see that SO Inc has been clear about what they mean by "outdated" or what exactly motivated that term or that project.)
    – philipxy
    Commented Feb 24, 2021 at 1:10

3 Answers 3

6

We as consumers of this content have only one dimension to project when it comes to evaluating content of any age: if it helped, or if it didn't.

Maybe this could be extrapolated or built on to derive meaning; in the sense that votes are atomic and happen at measurable points in time, if a trend of positive votes around the time the answer was first provided means that it was valuable then, and a negative trend of votes in more recent times (within the last 6 or so months) means that the answer is not as useful, perhaps that could be the metric leveraged.

It simplifies the amount of requested input from a user - no sense in bombarding them with a survey for every action they take - and gives a predictable heuristic in establishing quality over time.

This is confounded by the fact that users will just...never downvote content. For that reason, excluding users who have a higher degree of extreme voting patterns (e.g. for a given tag, upvote 70% of the time or downvote 99% of the time) from the heuristic to filter out the noise.

Beyond that, I don't think there's much else to go off of. I'd hate to be presented with a survey when I vote on stuff; it'd mean I vote on less things to not be bothered.

9
  • I agree that we would have to measure usefulness somehow but often I feel that votes aren't adequate for that. If I really got help for a difficult problem, a single upvote seems not enough to express my gratitude. I'm not sure what to look at exactly. Votes per visits ratios or upvote to downvote ratios? Commented Feb 23, 2021 at 6:55
  • Helped/Helped not: and then there are those at the beginning of their journey into coding that would'nt know a good answer on sight because they want dee coode nowww - served and catered to their special needs. Commented Feb 23, 2021 at 7:07
  • Maybe surveys could be made less annoying. Their participation could be made optional, their duration could be short and it could be emphasized how important their results are. Commented Feb 23, 2021 at 7:09
  • 1
    @Trilarion: Surveys are annoying, because they are an unnatural interaction with normal content. Any interaction beyond the bare minimum needed to accomplish A Thing™ will always be orders of magnitude slower than just doing the thing, thus obviating people to ignore the survey and diminish its value. Heuristics are probably the best thing we can leverage.
    – Makoto
    Commented Feb 23, 2021 at 16:42
  • @Trilarion: Voting is, again, all we have to say if we think content is good or bad. I still think that votes at time of creation vs votes over time is a useful metric, since it's less ambiguous and there's always one anchor point that can be referenced as a baseline. Maybe bounties could factor in. Dunno.
    – Makoto
    Commented Feb 23, 2021 at 16:43
  • @PatrickArtner: I don't see what that has to do with any of this.
    – Makoto
    Commented Feb 23, 2021 at 16:44
  • 1
    Any metric based on up/downvotes is skewed because of drive-by-+1/-1. If somethings got 100 votes at some point it is more likely it will akkrue more upvotes due to placement on page 1 of the answers. The 20th answer on page 3 that might be objectively better, won't ever be seen or voted on. So "still gaining votes" is not a good measurement because the discoverability of mayhap better questions on page 2+ is worse then the 5y old accepted / highly voted answer that was good at the time, but is not optimal anymore. SO today caters to/favors young-ish coders that take "a" over "the" solution. Commented Feb 23, 2021 at 17:02
  • Sorry @PatrickArtner, you're still not making sense. If you look at an answer for each day, then there's a snapshot of the voting score. It makes some sense that if an answer continues to be valuable even long after that it's been asked, then that would be the best we could do without anything intrusive to suggest that this answer is still relevant in the modern era, whereas if an answer was hugely popular 5 years ago but is barely getting any votes now (and has some downvotes now), then that'd be a signal that it hasn't aged well or the info is out of date.
    – Makoto
    Commented Feb 23, 2021 at 17:18
  • Not sure I completely understand the proposed metric here. One part seems to be the number of upvotes given per time and in particular a possible slowing down of that voting speed. And we would be dealing well with aging content if we don't see such a slowing down because it means the content is still useful, I guess. This seems to be somewhat similar to Timur Shatlands idea about the votes per visit ratio. I fear that there might be other influences that drive the voting speed up or down, so the metric might suffer from the need for a careful calibration, but it's definitely a start. Commented Feb 24, 2021 at 19:21
-4

The number of upvotes divided by the number of page visits by the users with upvoting privileges could be a useful metric to measure such effects. This is applicable to many interventions, including methods to address the issue of outdated answers.


How does this metric work? In the ideal case, the very first post that the user sees would have the answer that the user is looking for. An ideal user then votes the answer up. This results in the ratio of 1 upvote per visit. In the worst case, the user never finds the relevant answer, which is hidden by the outdated or otherwise poor quality answers. This results in 1 or more page visits, but 0 upvotes, bringing the total ratio of upvotes to page visits to 0.

2
  • 1
    I don't see how that is even remotely accurate. Unless an answer is very good, I often won't upvote it if it already has tons of upvotes. I also rarely revisit old content to remove upvotes. So an old answer and an old outdated answer would appear to be rated similarly by me. Commented Feb 22, 2021 at 20:44
  • 1
    This reminds me on the definition of meaningful user engagement by the company, which also does emphasize actions like voting. I would agree that a low vote to visit ratio probably means that people found nothing upvote-worthy and may predict a future drop in visits. On the other hand most visitors probably aren't registered and may not been able to vote. We wouldn't get any signal from them. Commented Feb 23, 2021 at 7:04
-6

Of primary importance: Sometimes 'outdated' content isn't all that outdated. For example, I was at a job where our organization was still using "Classic ASP" through Nov. 2020. It would have been a pity if a clean-up effort caused posts involving "Classic ASP" to disappear.

As for metrics: # of page vistits, and time spent would be important metrics, as would be upvotes, even down votes, and of course comments.

4
  • 5
    fortunately (unfortunately?) this isn't a mass deletion initiative, it's more about curation. fixing/improving existing posts or finding ways to push newer better less voted answers to the top.
    – Kevin B
    Commented Feb 22, 2021 at 20:43
  • 1
    How would you look at comments? Commented Feb 23, 2021 at 6:56
  • @Trilarion -- I assume SO has some mechanism so people could find the count of comments per question as 'meta-data' of some sort. Of course, if not, it'd be difficult. Maybe that could be on someone's to-do list, if such a count doesn't already exist Commented Feb 23, 2021 at 17:44
  • 1
    Yes, they have such a thing. Even though many answers have lots of comments. Lots of comments could mean that the post is controversial or that it is inspiring, difficult to say if number of comments is related to usefulness. Commented Feb 23, 2021 at 21:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .