74

Area 51 recommends that for beta sites, a rate of 90% answered is a healthy beta, and 80% answered needs some work.

I'm not sure how to find the recent answer rate on Stack Overflow overall, but the top users tab on tag pages does show statistics per tag. This paints a troubling picture on the top tags:

Unanswered rate top tags
╔════════════╦══════════╦════════════╦════════════╗
║    tags    ║ All time ║ Last month ║ Last week  ║
╠════════════╬══════════╬════════════╬════════════╣
║ javascript ║ 25.5%    ║ 45.9%      ║ 50.2%      ║
║ java       ║ 25.1%    ║ 48.5%      ║ 50.8%      ║
║ c#         ║ 22.6%    ║ 45.9%      ║ 49.9%      ║
║ php        ║ 27%      ║ 49.8%      ║ 53.2%      ║
║ android    ║ 36.5%    ║ 62.5%      ║ 65.8%      ║
║ jquery     ║ 25.9%    ║ 48.3%      ║ 52.8%      ║
║ python     ║ 21.1%    ║ 42.2%      ║ 46.4%      ║
║ html       ║ 24.9%    ║ 42.9%      ║ 47.5%      ║
╚════════════╩══════════╩════════════╩════════════╝

Looking at some tags that are by their very nature new, this trend seems to be continuing too:

Unanswered rate new tags
╔══════════════╦══════════╦════════════╦═══════════╗
║     tags     ║ All time ║ Last month ║ Last week ║
╠══════════════╬══════════╬════════════╬═══════════╣
║ python-3.5   ║ 36.7%    ║ 45.1%      ║ 41.7%     ║
║ ecmascript-6 ║ 18.7%    ║ 30.6%      ║ 36.5%     ║
║ java-8       ║ 12.4%    ║ 22.2%      ║ 33.9%     ║
║ php-7        ║ 32.7%    ║ 58.8%      ║ 58.8%     ║
╚══════════════╩══════════╩════════════╩═══════════╝

The trend is true far all tags that I checked, although absolute numbers vary somewhat.

One should expect that unanswered rates for Last week are higher than for Last month, as the proportion of very new questions that aren't answered yet will be higher, but that doesn't explain the dramatic drop from All time to Last month.

Why are answer rates dropping, across all tags?

30
  • 74
    There are only very little good questions compared to all bad ones, so there aren't much questions to answer. If you see 1 good questions once a month you can be happy.
    – Rizier123
    Commented Mar 10, 2016 at 19:37
  • 12
    Overall is 73%, see stackexchange.com/sites#percentanswered
    – yannis
    Commented Mar 10, 2016 at 19:38
  • 25
    So basically you are asking why questions just asked have less answers than questions that have been available to answer for a long time?
    – Travis J
    Commented Mar 10, 2016 at 19:38
  • 18
    In order to show that these rates are dropping, shouldn't you show what the rates have been historically?
    – Travis J
    Commented Mar 10, 2016 at 19:39
  • 2
    @TravisJ Yes, one should, but I don't have data for questions asked between 11 and 12 months ago, but most questions that are answered, are answered within one week. I agree that it would be more accurate to collect the historical percent answered rate for each month since the inception of the site. I suppose someone with better SQL skills than me could craft a query to gather that information from the database.
    – gerrit
    Commented Mar 10, 2016 at 19:40
  • 7
    Note that the "90% answered is a healthy beta" number was basically just made up back when area51 was launched. This is really something of an arbitrary metric and several SE employees admitted as much. In fact, it's not really looked at in determining a site's graduation any more. Unfortunately, the Area51 site never gets updated for whatever reason (lack of manpower?) and it regularly confuses people who didn't see those few comparatively obscure meta hard to find meta posts... :-/ Commented Mar 10, 2016 at 23:07
  • 6
    Also, it's an arbitrary number for tiny sites. SO is two orders of magnitude larger than the next largest SE site, and three orders larger than most sites. Even if it works for small sites, it can't necessarily be expected to work for massive sites.
    – nobody
    Commented Mar 10, 2016 at 23:56
  • 1
    A more meaningful statistic would be to compare the "Last month" value across different months.
    – jadhachem
    Commented Mar 11, 2016 at 5:52
  • 51
    I dare not put this as as answer - I am finding a lot of the questions on StackOverflow more niche these days. Many of the more simple questions have been answered and people get hammered if they dare ask something similar. I look to answer but often cannot without making a reasonable sized investigation myself. This means I am often unable to answer. One time I put 45 mins into working out the problem for someone and they didn't even respond or mark it as the answer, so unless I'm feeling eager, I don't make the extra effort.
    – JJJCoder
    Commented Mar 11, 2016 at 16:30
  • 44
    @HockeyJ Lack of response after providing an answer is far too common and incredibly rude. Sometimes I miss the old "accept rate" metric being prominently displayed for every question author... Commented Mar 11, 2016 at 16:46
  • 3
    It's probably just that some questions haven't been roomba-ed yet. Commented Mar 11, 2016 at 17:29
  • 6
    Better questions... "Why are bad questions on the increase?" or "why does it appear that moderation is failing to shut down bad questions?" and the knock-on question "why is this drop in quality driving away more experienced users?"
    – spender
    Commented Mar 12, 2016 at 13:16
  • 9
    The C# tag has become really boring. Always the same beginner mistakes, or easy API questions. Scrolling through stackoverflow.com/questions/tagged/c%23 is a sad sight.
    – usr
    Commented Mar 12, 2016 at 14:22
  • 4
    I'm curious what counts as an answered question. If I close a dozen questions a day about "What is a null reference" as dupes, do those count as answered, unanswered, or are they not in the ratings? Because in case 2 or 3, we would expect the rate to drop as we have the easy questions answered leaving more difficult/specialized ones. A few years ago those would have been answered. Commented Mar 12, 2016 at 17:30
  • 4
    @Rizier123 That's what I expected. In which case falling answer rates don't surprise me as easy questions usually get closed as dupes these days. Commented Mar 12, 2016 at 17:50

6 Answers 6

38

Questions really are answered less often and getting fewer answers today than they were in the past:

Grim answering picture

If you want to look at the query, it's on SEDE. I've filtered out confounding factors:

  • I'm only considering answers within 30 days of the question being asked. This eliminates the effect of ancient unanswered questions getting their first answer years later. (2020 data is a bit skewed because we haven't experienced the entire 30-day window just yet.)

  • I'm not looking at questions with a negative score, or those that were closed or deleted within the 30-day window. We know there is an increase in bad questions, so I want to remove that factor from this analysis. (Not remove it entirely, as you will read. But just remove the obvious problem with bad questions not being answerable.)

  • I'm using our standard definition of "answered" which doesn't count answers that haven't been either upvoted or accepted. Somebody has to have vouched for at least one answer or we don't consider the question answered yet.

In case it isn't clear, unanswered_rate is the ratio of questions that didn't get positively-scored or accepted answer within the 30-day window. answer_rate counts the average number of answers with a positive score or that have been accepted. So not only are there more questions that don't get answered, questions are less likely to get multiple answers. One thing that has changed is questions have narrowed considerably in scope over the years. It wasn't unusual for questions to get dozens of answers in 2008, but that's pretty rare these days.

I don't think the reason behind the trend is very complicated:

The questions are many and the answerers are few.

The questions line includes all questions, including downvoted, closed and/or deleted. answerers counts how many people have contributed at least one upvoted answer in the year. Questions have far outpaced people willing to answer them since the beginning and we've seen fewer answerers since the peak (320,124) in 2016. Each question represents a measure of work to answer, close or edit into shape. As users (such as yours truly) tire of doing that work, we need to replace them with new volunteers. We've simply fallen behind in the last two years.

In my association with the site over the last ten years, I've proposed a number of reasons why people might leave:

This isn't a comprehensive list, but the upshot is there are many reasons people stop answering and not all of them are fixable with our current toolset. Even as question rates have dropped in the past two years (-10%), the number of users who have answered has fallen faster (-23%). As long as new users - retiring users < 0 we're going have a problem. (Unless you like the idea of more questions of varying quality with fewer people to handle them, of course.)

Naturally Meta has focused on retiring users: "If we could just fix a set of problems on the site, we'd stop losing valuable contributors." But that's only half the problem. Inevitably people will retire even if it's just that they are retiring in the real world and spending their time playing with the grandchildren instead of answers Stack Overflow questions. From 2008 to 2012, Stack Overflow saw meteoric growth then leveling off until 2015 or so. In the last few years we saw a drop off in the number of new answerers:

New answerers by year

Ignore the last point, obviously. We're a long way from knowing what will happen this year. That said, our plans for 2019 include initiatives such as the Ask a Question wizard to reduce bad questions (and hopefully increase good ones) and Custom Question Lists to help users find questions to answer. There are a few other projects we're considering that I hope will be announced soon. However, if these technical tools are to work, we'll also need to make cultural adjustments. As Clay Shirky has pointed out:

But you cannot completely program social issues either. So you can't separate the two things, and you also can't specify all social issues in technology. The group is going to assert its rights somehow, and you're going to get this mix of social and technological effects.

15
  • 6
    As long as the ask question wizard communicates to users that debugging questions are preferred, and how-to questions are off-topic, it isn't going to help the problem. In my opinion we need to remove the unwarranted stigma against how-to style questions and promote those as being preferred if we want to keep people interested.
    – user4639281
    Commented Jan 12, 2019 at 1:01
  • 3
    @TinyGiant: I'm extremely sympathetic, but this is very much a case of needing the social issues to come along with the technical issues. If we were to change the wizard to do what you suggest, it will be rejected by many existing users. This is what Shirky means by "the group is going to assert its rights somehow". It seems to me that we need enough users who value how-to questions before we can teach new users to ask them.
    – Jon Ericson Staff
    Commented Jan 12, 2019 at 1:08
  • 1
    See also 2018: a year in asking and answering for a network-wide analysis of total asking and answering. Commented Jan 12, 2019 at 1:10
  • Has there been any research to find out what questions people enjoy answering? Not just what they do answer, but what they really enjoy. Commented Jan 12, 2019 at 1:44
  • @PaulCrovella: Not that I know of, but that's a solid idea. In the early days of the site, I asked questions that I enjoyed answering, but those tended not to be appreciated by other users even then. That said, I'm likely to be an outlier in this respect.
    – Jon Ericson Staff
    Commented Jan 12, 2019 at 1:54
  • 3
    Thank you for the extensive answer to a nearly 3 year old question!
    – gerrit
    Commented Jan 12, 2019 at 2:03
  • @Jon That is what I meant by removing the unwarranted stigma. A wizard alone cannot remove the stigma. It would require a social awareness campaign lead by the company. It seems right now that the company doesn't think it is their responsibility. In fact it seems they think a lot of things aren't their responsibility.
    – user4639281
    Commented Jan 12, 2019 at 2:32
  • 2
    IIRC, some statistics (presented by Shog?) show that getting no reply at all is the strongest way to drive a new asker away. So a drop in answerers then indirectly reduces retention of new visitors.
    – Raedwald
    Commented Jan 12, 2019 at 10:18
  • 2
    @Raedwald: Yes. Shog showed that effect and I confirmed with another method. It could be a destructive feedback problem if the reduction in answers causes a drop in interesting questions which discourages people from hanging around waiting to answer. On the other hand, it could be that questions and answers will reach an equilibrium where fewer questions means more of them get answered. At the moment, it really seems like the trend for everything is down, which is concerning.
    – Jon Ericson Staff
    Commented Jan 13, 2019 at 0:04
  • @TinyGiant: You know, our experience with Documentation suggests we only have minimal influence when it comes to what people want to do on the site. Obviously Documentation had other problems, but many people were just not interested in that sort of writing.
    – Jon Ericson Staff
    Commented Jan 13, 2019 at 0:11
  • 5
    @JonEricson I was very frustrated that you didn't change the name to Examples or otherwise indicate what you wanted Documentation to be. I think you guys had a pretty good idea of what you wanted Docs to be, and built a system designed to support that vision, but then completely failed in communicating it to us, and so we misused it. You can't make us drink, but you should at least try leading us to water. (On the other hand, I think the company really does want SO to be mostly debugging questions, out of a misguided focus on "engagement", so...) Commented Jan 13, 2019 at 8:42
  • 2
    @Jon honestly that's just a cop out. This is most definitely a problem that could be solved by The Powers That Be if they chose to solve it. The unwillingness to consider trying is validating the belief that Stack Overflow is only for debugging garbage.
    – user4639281
    Commented Jan 13, 2019 at 17:24
  • 1
    I'm not sure what's the value of excluding answers with a score lower than 1. I understand that's the standard way to look at it, but the unanswered questions are often in niche areas where it's hard to get any upvote.
    – Nemo
    Commented Feb 5, 2020 at 9:27
  • @Nemo: We don't really know if the question has been answered until someone validates it. I don't have the query in front of me (SEDE is down for maintenance at the moment), but as the answer states, I counted the asker accepting an answer as "answered". Obviously that might mean only one of the answers gets counted on less populated tags. I don't remember for sure, but I believe the trend continues if you include all answers. Commented Feb 5, 2020 at 15:48
  • @JonEricson yes, but with the same reasoning if there's an answer we don't know for sure that it doesn't answer the question, unless it's very heavily downvoted.
    – Nemo
    Commented Feb 6, 2020 at 8:49
68

It unfortunately often takes some time for bad questions to actually get closed, in many cases longer than a week.

Unanswered questions that don't have a positive score are also eventually deleted, and since those stats are counting undeleted questions, that results in the unanswered rate of questions to be higher for times in the past, simply because the more recent bad questions haven't yet been deleted/closed.

And of course all of that is on top of the fact that old, unanswered questions do sometimes get answered over time, increasing the overall answered percentage of older questions.

6
  • I see, I forgot about the automatic deletion of unanswered 0-scored questions. That makes my measurement more biased than I thought it was. So to see whether the answer rate is really dropping would require more advanced querying of the database.
    – gerrit
    Commented Mar 10, 2016 at 19:42
  • 1
    But still even with this corrections there could be an underlying downwards trend. Probably nobody knows right now. But going back in time we could maybe compare March 2015 with March 2014 with ... Commented Mar 11, 2016 at 17:30
  • @Trilarion You're free to look at the data and see if you can find any insights there to try to confirm or deny the OP's conclusions. I merely explained the observations the OP provided.
    – Servy
    Commented Mar 11, 2016 at 17:38
  • 2
    I will check if I can come up with something. Commented Mar 11, 2016 at 17:42
  • I came up with something. Commented Mar 14, 2016 at 20:35
  • is there any i can find total answer rate of this community : magento.stackexchange.com
    – fresher
    Commented Apr 11, 2016 at 13:48
8

The question claims a drop in answer rates. Several answers here argue that either deletion/closing or late answers might distort the statistics presented in the question. I tested it a bit more.

This is my SEDE query.

I looked at all questions posed in a certain date range who have at least one positively scored answer. I calculated the time it takes until the first such answer versus how often that happens in a time range of 0 to 100 days and set all non positively scored answers as well as late answers later than 100 days to 100.

The result is for questions older than 100 days but younger than 400 days:

  • ~900k questions answered within 5 days
  • ~1130k questions not answered after at least 100 days

For questions older than 400 days and younger than 700 days

  • ~860k questions answered within 5 days
  • ~790k questions not answered after at least 100 days

And for questions older than 700 days and younger than 1000 days

  • ~1040k questions within 5 days
  • ~720k questions not answered after at least 100 days

Indeed the ratio of quickly answered questions to not answered at all questions has dropped over time.

Also it becomes clear from the graph that the overwhelming number of answers comes in the first 2-3 days. So I guess the contribution of late answers is not significant.

Example graph for (-700,-1000) days range:

enter image description here

What remains is either a real downward trend or a strong removal (roomba) effect also many days after a question has been created.

In case there really is a downward trend, here my speculation about the reason: A lowered ratio of answerers to questioners/questions or a higher percentage of questions of lower quality or both or something else.

8
  • Btw. the whole business is not simple. I could not convince myself to include zero voted answers although there are quite a lot of them. Also I now tried to exclude deleted or closed questions at all but the overall effect seems to be the same. Commented Mar 14, 2016 at 16:35
  • @Trilarion, interesting results. I came to this post after not receiving any answers to a number of my recent questions, despite searching for pre-existing answers to similar questions. Looking back over my previous questions, most were answered promptly and 98% received an answer. I suspect SO has lost of programmers willing to contribute answers in recent times, for whatever reasons. Personally, I don't visit so soften any more and think the site has gone badly wrong.
    – SmacL
    Commented Mar 14, 2016 at 17:16
  • @ShaneMacLaughlin The numbers do not seem to support that something has gone badly wrong yet. We are talking more about a possible drop in answer rates of 10%, meaning that you should still get an answer to your question with high probability if the question is clear and not too difficult to answer. People who worry however might extrapolate this. Maybe many of the good programming questions are gone and now it's more about live debugging (here is my code, it doesn't work). Probably you do not find enough volunteers for such a job. Maybe SO should turn down more debugging help questions. Commented Mar 14, 2016 at 20:34
  • @Trialarion, you could very well be right, and extrapolation from such a small sample group is never good. However, the last question that I asked I also posted on the MSDN forum, where I received an answer, where there was no activity of any kind on SO. As such, the dialogue continues on MSDN, and many of my searches for programming related content have led me to active and current content generated in other sites. Given the rate of change in delivered technology, I don't buy the 'all the good questions have already been asked' line at all.
    – SmacL
    Commented Mar 15, 2016 at 7:59
  • 1
    @ShaneMacLaughlin I think now too that "all the good questions are asked" is not true. With new technology you should get new good questions all the time. I could imagine that SO fails because experts in the fields just don't like it anymore. It could have the best system available, but it's worth exactly nothing if people aren't using it anymore. Well that doesn't mean I would advocate radical changes now, now, now. But I'm very worried that the live debug questions are usually of low quality. It would be much better to teach people to debug themselves. Commented Mar 15, 2016 at 8:46
  • 2
    @Trialarion, I think there are a few things fundamentally wrong with SO as it is now, most notably the draconian ban on being allowed ask a question of the form 'is there a library to do X?'. Given that many fundamental programming questions have already been answered, many of the more complex results are implemented as libraries, and a core part of modern programming is using such libraries rather than re-inventing the wheel. Suggesting people ask 'how do I do X?' where the answer is always going to be 'use library Y' is a nonsense IMHO.
    – SmacL
    Commented Mar 15, 2016 at 8:59
  • ... Many high ranking questions and answers on this site from its earlier years are actually specifically related to finding and selecting resources such as libraries and tools. The blanket ban on this type of activity significantly limits the usefulness of SO.
    – SmacL
    Commented Mar 15, 2016 at 9:05
  • 1
    @ShaneMacLaughlin In principle SoftwareRecommendations.StackExchange is specialized on this. Unfortunately their visibility is very low at the moment. And there are two problems with good software recommendations - you need a very detailed problem description to give a good recommendation, and they tend to get outdated after a while. Otherwise there is a good place and it's SoftwareRecommendations. Commented Mar 15, 2016 at 9:48
4

Having ended up answering three of the last five questions I've asked myself after no replies were forthcoming, and deleting a fourth on the basis that it was downvoted and closed for looking for a solution that might need a 3rd party resource, it certainly feels like the answer rate in SO has pretty much collapsed in the areas I'm posting in. Comparing answer rates to my own questions from a few years ago, out of 95 previous questions, only two remained unanswered and the bulk of the remainder had answers within the hour.

It is highly subjective of course, but I feel that many people such as myself are less likely to answer questions themselves if they're not receiving any response to their own questions, which in turn leads to a downward spiral.

I also notice that other sites such as MSDN have copied the SO model in terms of rep, badges, etc... and are probably taking a certain portion of the disaffected SO audience. For example, after failing to receive any feedback for one of my recent questions on SO, I posted the same question on MSDN where I did get a response. Next time I've have a similar question, where do you think it will get asked, and which site will I be hanging around on to answer other peoples questions?

I'm a big fan of SO, but I think as a community we really need to focus on keeping all of that growing community engaged. My opinion is that the seemingly prevalent attitude that the content of this site is more important than the community that generates it will be the downfall of SO. The site needs to look after the new users asking silly questions, as in ten years time these will be the experienced programmers providing needed content for questions relating to the current technology of that time. Driving away new users with draconian rules and intolerant attitudes is making this site stagnate. That in my highly subjective, biased, and under-researched opinion is why answer rates are falling and will continue to do so.

2
  • 2
    Good points all around. My one quibble is that it won't take 10 years for most new users to become helpful users.
    – Jon Ericson Staff
    Commented Jan 12, 2019 at 0:52
  • Ten months since posting this and I find that I use stackoverflow more rarely these days. Pattern seems to be that I ask a question, typically get a couple of upvotes on the question indicating it is on topic and of some interest, and either get just one answer, or more often, no answers. I find I'm getting better results posting on more specific sites, e.g. open cv forum, MSDN forum, etc... I like that SO has made a big effort to become more friendly, and hope it picks up momentum once again.
    – SmacL
    Commented Jan 14, 2019 at 9:08
3

... What @Servy said, and in my experience...

Numerous plugins are being used to enhance fundamental tags. This means questions are tagged with a fundamental tag plus one (or more) plugin tags.

It's less likely to find a critical mass of answerers for questions that have knowledge of a common main tag plus a less common plugin tag.

1
  • 4
    Yeah, if you use 42 different plugins/frameworks/libraries you're gonna have problems finding someone that knows all those exact ones. And that seems to be the way many newer devs (i.e. most the question askers) are heading. Commented Mar 11, 2016 at 18:38
3

Considering your Roomba's criterias and your delay of beyond one month, I would say Roomba is the cause of the gap between one month and forever.

For the small gap between last month and last week, I would guess that some stuff get cleaned manually and some others get answered.

2
  • 1
    "For the small gap between last month and last week" - Closed and roombable questions get deleted after 9 days, so there's that.
    – Artjom B.
    Commented Mar 11, 2016 at 20:13
  • I took various factors, including the Roomba, into account and there's still an obvious decrease in answer rate.
    – Jon Ericson Staff
    Commented Jan 12, 2019 at 0:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .