88

A simple plot of questions and answers activity on SO shows that the activity dropped a lot in 2014. What happened to SO in 2014?

select PostTypeId, year(CreationDate) as Y, month(CreationDate) as M, count(*) as n
from Posts
where PostTypeId in (1,2)
group by PostTypeId, year(CreationDate), month(CreationDate)

enter image description here

13
  • 6
    Peak oil effect. Commented Mar 31, 2016 at 22:02
  • 41
    Jon Skeets' KB broke. Commented Mar 31, 2016 at 22:02
  • 11
    April 2014 is when SO got its own meta. Enthusiastically used, a lot of dirty laundry got hung out to dry. Enough for lots of contributors to lose faith it seems. Commented Mar 31, 2016 at 22:03
  • 4
    It hit an iceberg. Commented Mar 31, 2016 at 22:05
  • 5
    That was when the low quality review queue would have really gotten going, deleting lots of posts that merited deletion that previously weren't getting enough attention to get deleted. Particularly non-answers (you'll notice the drop in answers is much more significant than questions). I imagine that if you were looking at the same stats but including deleted posts the treandline would stay pretty consistent around that time.
    – Servy
    Commented Mar 31, 2016 at 22:17
  • 3
    @Servy Querying PostsWithDeleted table shows the same pattern.
    – user3717023
    Commented Mar 31, 2016 at 22:43
  • blog.stackoverflow.com/2014/04/…
    – Quill
    Commented Mar 31, 2016 at 22:47
  • 2
    Valley of disillusionment? Commented Apr 1, 2016 at 0:22
  • 4
    FWIW in March 2014 they started burning down close queue. In summer 2014 they started rolling out features of se-quality-project
    – gnat
    Commented Apr 1, 2016 at 6:42
  • 1
    Fewer questions and answers are likely a good thing.
    – Pekka
    Commented Apr 1, 2016 at 8:31
  • 1
    There was another similar (although proportionally smaller, but remarkable) drop in the number of posts a few weeks after SO went live, around Aug 2008.
    – imrek
    Commented Apr 1, 2016 at 16:59
  • Now that you mention it, the downturn is also seen in total number of answerers: mpi-sws.org/~utkarshu/vis-score-trends Commented Apr 2, 2016 at 15:33
  • 3
    Interestingly enough, Why is Stack Overflow so negative of late? was asked in 2014. Correlation? Causation? I don't have the darnedest clue between the definition of the two so understanding my comment will be left up to you.
    – MonkeyZeus
    Commented Apr 4, 2016 at 21:47

2 Answers 2

117

Well, I guess you could say Stack Overflow peaked in 2014. At least, in terms of posts per day. March of 2014 saw 289,103 questions posted on Stack Overflow, the most in any single month in its entire history. And the strain was starting to show:

The cost of scaling to this size has been a constant battle against human nature. We are social creatures, and when asked - forced - to forego these personal connections, we get irritated. Scanning the answers to the most popular discussion here finds the same two stories repeated over and over again:

  • I'm here to learn but Stack Overflow doesn't want to guide me - my questions get downvoted and closed with nary a helpful comment.
  • I keep trying to educate folks asking bad questions, but no matter how much I write they keep coming - so I get more terse, more mean as I lose patience.

...The frustration level on Stack Overflow was peaking also.

Here's another graph, showing the entire history of Stack Overflow:

You'll notice that while the number of questions asked keeps increasing until March of 2014, it starts to, um, oscillate noticeably starting in 2012, becoming increasingly variable by the latter part of 2014. Every new record high is followed by a sharp decline, retreating from the maximum for several months before again creeping upwards.

Something else starts in 2012... Answered questions and questions that remain visible for more than a day start to fall noticeably below questions asked. By 2014, there are a lot of questions that just aren't getting answered, and a lot of questions that are just awful. I don't really think those things are independent.

...Nor do I think the number of questions being asked is independent. The best way to get someone to leave is to give them no feedback at all. And the best way to get someone to come back is to answer their question. If questions aren't getting answered, if they're actually getting ignored, then fewer people will come back.

As much as I'd like to take credit for some of this stabilization, as much as I'd love to point to the work done by Tim, Ben, Geoff and others aimed at identifying and blocking bad posts and slowing down problematic users... I don't think that had this big of an effect. Here's the same data in the graph above, with questions answered and asked by month represented as a ratio:

Note that while March 2014 was a peak in the previous graph, it's a sharp descent in this one: the month with the most questions asked and answered in Stack Overflow history also has the biggest imbalance between the two of them to date. And after that... It kinda settled down between 0.86 and 0.89 for the next two years.

In other words, we appear to have hit an equilibrium. Stack Overflow, as it is currently designed, does not seem to be able to handle more than about 8 thousand questions per day on average. When that's exceeded, folks can't find questions they want to answer, questions don't get answered, folks stop asking, and this continues until answerers can find what they're after again.

Now, we've done a ton of work in the past two years to try and fine-tune that; ideally, after all, it's the worst questions that'd go unanswered. Triage currently handles roughly 20% of questions asked on the site and tries to prioritize them with this exact goal. But I don't think we're going to exceed this 8K/day average any time soon; not without a massive change to how folks are able to use the site at any rate.

10
  • 19
    Economy cycles at their finest.
    – Braiam
    Commented Apr 1, 2016 at 0:30
  • 4
    On the last graph, I'm struggling to figure out how the questions answered to asked ratio can exceed 1.0. Is it really answers per questions asked?
    – Jon Ericson Staff
    Commented Apr 1, 2016 at 16:37
  • 11
    Hover your mouse cursor over it, @Jon...
    – Shog9
    Commented Apr 1, 2016 at 20:21
  • 3
    Hmya, I suppose it is your job to remind us that this happened because we suck by not giving enough answers. Instead of the fanciful idea that 4 peaks per year might be correlated to the rotation of planet Earth around the Sun and the way the money people plan projects. In return, any concrete plans to actually get it back down to 8K/day? That would be wonderful. Commented Apr 1, 2016 at 22:50
  • 8
    @HansPassant: At least at the moment, the rate is 8.1k a day. The equilibrium theory suggests it's not anybody's fault there aren't more answers, but that this is a natural consequence of the existing system. Certainly I don't read any call to answer more in this post. Truth is, people will only participate as long as they are enjoying what they are doing.
    – Jon Ericson Staff
    Commented Apr 1, 2016 at 23:10
  • 1
    Kind of funny that one of the longest periods without a post was in the middle of 2014.
    – Jason C
    Commented May 22, 2017 at 19:10
  • 8
    Yeah, that's what happens when you plug a UPS into itself, @JasonC
    – Shog9
    Commented May 22, 2017 at 19:31
  • 1
    If there is a fixed upper limit on the question answering rate, that suggests efforts should be made to prevent bad questions being asked in the first place, rather than letting them be opened then closing and downvoting them.
    – Raedwald
    Commented Dec 30, 2018 at 9:51
  • Can you rerun the graphs up to (about) today? I'd love to see whether we're still at the 8k/day average, or that it has changed due to e.g. the welcome wagon.
    – Adriaan
    Commented Oct 17, 2019 at 8:38
  • 1
    @Adriaan It rose a bit between the 2014 "drop" to 2016, but has been declining steadily ever since. It's currently sitting at around 7k questions a day. What's much more noticeable however is the decline in answer / question ratio. 2014-ish, there used to be around 15-20k more answers every week than questions. In 2016 that was more like 5-10k more answers. In 2019, there's barely any difference, which would tend to suggest fewer questions are being answered, and more are just attracting a single answer rather than multiple answers competing for quality. Commented Nov 4, 2019 at 12:04
30

Shog suggests there is a natural equilibrium between askers and answerers. My gut reactions were:

  • ~8k questions a day is pretty remarkable given the site's humble, even naive, beginnings. So much has been done to help the whole operation scale and yet sometimes the works get jammed up. Regular users don't often see that flags (3.3m so far) require constant attention from a relatively small squad of moderators. It would be even more remarkable if the whole operation were not implicitly rate limited somehow.

  • The explanation is not exactly satisfying.

    Weigh the last couple of years of questions

Mechanically, the number of answers on a site can be described as:

answers = questions * answers_per_question

Since we optimize for pearls, it's really the answers that are important. If you input a junky question, the best you can hope for is no answers at all. And, in fact, questions with lower scores are less likely to be answered:

Answered rate by score band and month asked

This chart shows the rate questions are answered by ask date. The downward trend is partially an artifact of it taking time to answer difficult questions: as time goes by, the odds a question with a score of 5+ getting answered approaches unity. There might also be an increased reluctance to answer questions, but it's hard to tease out of the data without analysing timing of votes and answers. For our purposes, however, the important point is to notice the answer gap between positive, zero and negative question scores.

So far, the equilibrium theory seems plausible. Questions that are unwelcome (as indicated by downvotes) are less likely to be answered. When users' questions are not answered, they are less likely to return to ask more. Roughly half of downvoted questions are never answered and eventually deleted.

The trouble, at least when it comes to 2014, is that unwelcome questions (i.e., those with a negative score) have been increasing steadily since at least 2011, if not beta:

Positive vs. negatively scored questions by month

In economics, there's a concept called price insensitivity in which products can be sold for increasing prices without decreasing demand. High end fashion, for instance, tends to be bought by people who don't pinch pennies. Drugs (both the ones that correct health problems and the addictive type) can also be sold for higher prices without hurting sales if their effects can not be duplicated by a cheaper product. Then there's See's Candy, which raises price per pound each year and I, along with thousands of other loyal customers, buy roughly the same amount. My guess is that Stack Overflow's reputation for quality answers encourages people to ask questions even in the face of downvotes and silence.

Instead the decreased activity seems correlated to a decrease in questions getting upvotes and a corresponding increase in questions not getting votes at all. It's difficult to infer causation and I see two possible theories:

  1. Starting around 2013 and peaking around March, 2014, people began asking fewer interesting questions. That lead to a decrease in voting on questions and fewer answers being given. Since the feedback on these uninteresting questions was discouraging, people began asking fewer questions on the whole. Meanwhile, truly poor questions continued being asked with little regard to negative feedback.

  2. Stack Overflow users began noticing increasing numbers of truly awful questions and decided, rightly, that downvoting and refusing to answer them is the best remedy. These questions fit broad categories of awful and users began withholding votes from questions that were not themselves awful, but bore some of the markers of awful. Fewer of these questions got answered and askers of mediocre questions did not see any point in trying to improve.

These are equally probable theories in my way of thinking. I could even see a blended theory being correct. But the evidence does not suggest that increases in awful questions is the proximate cause in decreased answering during 2014. Instead, the decrease seems to be a result of fewer welcome/upvoted questions.

7
  • 5
    could it be that flood of awful questions simply obscures answer-worthy ones, making them harder to discover. This could explain why even worthy questions are getting less votes and answers. If this is so, I'd say this sounds suspiciously as spam control problem (unwelcome questions would be similar to spam polluting one's inbox and burying useful stuff)
    – gnat
    Commented Apr 5, 2016 at 8:30
  • 1
    @gnat: I do think the proportion of upvoted to downvoted questions is a problem. But the culprit is not awful questions, which have gotten a lot of attention, but a noticeable decrease in upvoted questions. In other words, I doubt people would be much bothered by downvoted questions if there were plenty of interesting questions to look at instead. I know when I look at tags I have experience with, I have to spend a lot of time sorting through dreck to find something worth answering.
    – Jon Ericson Staff
    Commented Apr 5, 2016 at 17:15
  • 2
    I mean, maybe too much of unworthy garbage is the reason that better questions miss the upvotes they would get in cleaner environment. Think of it, regular user looks at (default) 15-questions tag page. If there are 3 bad ones, not a big deal . Assuming that reader votes up half of potentially worthy content, there would be (15-3)/2=6 upvotes. If there are, say, 7, this possibly can harm because there is less worthy content shown and because some worthy content gets to second page. Same assumption of voting half content would now give us (15-7)/2=4 upvotes, quite a noticeable decrease
    – gnat
    Commented Apr 5, 2016 at 17:24
  • 1
    If it is voting which suddenly changed, then that would be a cultural aspect. On Stack Overflow, almost all cultural changes are reflected from meta activity at the time of incident or over time. In my opinion, the removal of meta from Stack Overflow was detrimental in the near term. While there may not be direct discussion related to voting changes, there was also no longer guidance provided anywhere on MSO with relation to upvoting good content - while there was a very large amount of guidance about downvoting bad content.
    – Travis J
    Commented Apr 5, 2016 at 23:30
  • 1
    I believe that if the drag on answering is a result of less upvoting on questions in general, that would be related to a cultural aspect. One very large glaring issue related to this topic that really irks me is questions with no votes and several answers. If someone answers a question, and they do not upvote it, then why answer it?
    – Travis J
    Commented Apr 5, 2016 at 23:30
  • 2
    If an ever increasing tide of bad questions is forcing out the good content and discouraging the providers of good content, perhaps the solution is not to be welcoming to the kinds of visitor who posts bad questions?
    – Raedwald
    Commented Dec 30, 2018 at 9:43
  • 1
    @Raedwald: Here's the catch: people tend to be poor evaluators of future contributions by new users. When I was moderator on a small site, I occasionally spent time helping new users learn how to answer the Stack Exchange Way™. Sometimes really promising contributors turned out to not be that helpful and sometimes people I had little hope for turned out to be great contributors with just a little nudge in the right direction. See also: What can we do to encourage (or discourage) a second question?
    – Jon Ericson Staff
    Commented Dec 31, 2018 at 20:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .