30

In the last couple of weeks, many countries have started ramping up things like Working from Home users, and Remote Learning users. Others can't get to work due to the lockdown in their city/county/state and so are forced to stay at home. My question, therefore, is how has this impacted recent questions on Stack Overflow? Has it impacted it at all? Have the volume of questions risen, or has the quality of said questions gone up or down?

The goal of this question is not to identify of the volume or quality of questions has changed compared to last year as a whole; it is specifically looking at how things have changed in the last few weeks.

7
  • 9
    That is normal this time of the year. Much more notable is the sudden and unusual drop in the number of answers. Look here (10K rep required). Commented Mar 20, 2020 at 10:32
  • 17
    @HansPassant: (25k rep required)
    – BDL
    Commented Mar 20, 2020 at 10:37
  • That doesn't really show the same thing I'm looking at here , @HansPassant . Certainly that shows volumes of questions, and how many votes there have been, but it doesn't show how that effects quality, which was also my aim. I'm specifically looking at questions here, not answers. As for the drop in number of answer, that could be due to that the percentage of questions asked are either "neutral" or negatively scored; which do attract less answers.
    – Thom A
    Commented Mar 20, 2020 at 10:45
  • @HansPassant: I don't see a sudden and unusual drop in the number of answers. You don't mean the end-of-year dip, do you?
    – einpoklum
    Commented Mar 20, 2020 at 11:35
  • 3
    I'd say that Hans is right there, @einpoklum. Compare 2020 to 2019 and there's an obviously stendy trend downwards in the 2020 numbers.
    – Thom A
    Commented Mar 20, 2020 at 12:54
  • @Larnu: Ok, I see it. Thanks for that. If I had the time I'd create a window-function version of those queries to get a running average, and put the curves for both years on top of each other.
    – einpoklum
    Commented Mar 20, 2020 at 15:10
  • The latter part is probably comment to go under the answer, @einpoklum . I did consider that, but I felt the graphs would get a little messy if i put more on there. Hence why i used the percentage and volume ones further down, and followed up with a stacked column chart.
    – Thom A
    Commented Mar 20, 2020 at 15:17

2 Answers 2

29

I used this SEDE query to get some figures on this.

Note: I complete further analysis and provide another TL;DR later down the question, several weeks after the initial data, that has a slightly different conclusion.

Recent events started back in December, however, things didn't really start kicking off in the rest of the world till much more recently. I therefore ran the 2 date ranges for 2018-12-02 to 2019-03-16 and 2019-12-01 to 2020-03-15 (I'm British so my weeks start on a Sunday and you'll like it!🙃). This gave a big enough range to hopefully try and spot if anything has changed.

TL;DR: Currently there appears to be no significant change to the volume or quality of questions compared to last year. For the Week Commencing 10 March 2020 there was a more noticeable drop in quality compared to the same week in the previous year, however, more weeks of data will be needed to see if this trend continues.


So, let's start off with last year (I won't post the raw data, as you can obtain that from the above link):

2018-2019 Volumes

Unsurprisingly, for last year, there was a volume reduction around Christmas, but other that that, things are pretty much flat. This, however, was there for a base line to compare to this year. On average about 21.5% of questions are negatively scored per week, and 11.4% were closed. Neutral Score Questions (those with a score of 0) were about 52.7% on average per week.

Of the questions asked on average 33.7% were deleted.

Now, this year:

2019-2020 Volumes

Again we have a dip at Christmas, but that is also again expected. Like last year, we have a gradual upwards trend of question volume as we get towards March and then both have a sudden dip in the last week (second week in March).

Percentage wise, on average 20.5% of questions a week have received a negative score, 14.7% were closed, and about 60% are neutral. I have included the deleted figures in the graph, however, any average is skewed as Roomba will not have deleted more recent questions. I therefore won't be using this in the comparisons.

Firstly, let's see how those percentage look next to each other:

Percentage Comparisons

Let's start with the Positive Questions. For 2020, for every week, these have been lower than 2019. That gap, however, has been getting larger (although slowly) and the last week ends with completely opposite results; 2019 gets a bump up where as 2020 gets a bump down.

For the Neutral Questions, these two lines are actually pretty much following the same pattern. Both dip at first, and then rise up a little. Again, however for the final week they have opposite bumps, with the volume going up in 2020 and down in 2019.

Negative Scoring question very much go hand in hand here. in fact apart from a rise in % around week 4, there is little to no difference.

For Closures, the same trend really, but just that 2020 has more. This, however, I feel could easily be influenced by the change to the 3 votes to close, so isn't really a fair test.

From this, it does suggest that the percentage of positively received questions is down, and slowly trending downwards; but it has been for months. There is a sudden spike last week, however, there is not enough information to suggest that it is going to continue. If the volumes stay down, this might be an indicator.

Volumes wise, we have a similar story:

Volumes Comparison

All are following a very similar trend, with 2020 Positive being lower than last year, Negative being about the same, and Neutral and Closed being higher.

At this point, therefore, I would suggest there isn't enough information to suggest that recent events are effecting Stack Overflow in a positive or negative way, however, at present it looks like it is not. Certainly positive scored questions are down, but that has been consistent for the whole 15 weeks in the data view, so that's nothing new.

I will, however, see how thing looks again in 4 weeks or so, and see if there's any changes. For some countries, that aren't fully making use of home working, this could begin to show their impact, or it might simply not show anything has changed.

There's also, visually, very little difference if you stack the percentages of positive, neutral and negative between the years (so far):

2019 Stacked

2020 Stacks

Note: Data shown in graphs was correct at the time the SEDE query was run. Question votes, and closure/deletion volumes can change, and the graphs will not be reflective of this unless they are updated at a later date.


OK, so we're a few weeks later now, and I wanted to do another update. Almost every country (or state) appears to now be in some kind of lockdown now. Previously when I first wrote this question, and answer, I was still at the office but it was my last couple of days of doing so. Many other countries are following the same process, so I'll likely do this as a final update; if anyone else wants the Excel document I used, I'll upload it later for others to download and you can use the SEDE to update to fill curiosity.

OK, so first let's get to the volumes, as I found the last week really interesting:

Volume Comparison - Final Week

Notice the huge spike in questions last week. This isn't a small increase, it's over 5,000 more questions than the prior week, and (excluding that prior week) around 9,000 more questions that any other week in the entire dataset. In just a week the total volume of questions increased by almost 10% (about 8.5%). I don't expect the volume to keep increasing like that, but I certainly think that the impact of recent events surely has incurred this spike.

As for the volume well, poorly and neutrally received questions all areas went up, which is not really a surprise. More questions means more reactions, and thus more of everything. Especially with such a big increase.

SO, let's have a look at the percentages instead, which are more likely to show how well the community is receiving these new questions:

Percentage Compare - Final Week

Well, it seems that actually the questions are being quite neutrally received, but certainly not poorly. In the last few weeks the percentage share of questions that have been poorly received by the community has fallen; from over 21% in week 15 to just 18.3% in week 18. That isn't a huge fall, but I see that as a positive. Any fall in negatively received questions is a good thing for the community. Now, that could be due to new users (or users that previously lurked) gaining new privileges to vote to be being active, but even if they find the questions find the questions helpful, while other more active members find them unhelpful, or lack research, it's still a net gain for those users.

Positively received questions did go up in the last week and it's settled at around 15%, with just a 0.1% downward change since last week.

Neutrally received questions got a a big increase over the last couple week, going all the way up to 66.3% of the questions asked in a week. These questions aren't necessarily bad, but it (based on my personal experience) I wouldn't be surprised if it's simply that the question shows some attempt or research, but had they spent a little more time (not asking at the first hurdle) they would have got their answer. These aren't bad questions, and don't (in my belief) don't deserve a downvote, but they probably aren't going to be helpful in the future when another search, or taking a moment to read the error in it's entirety, would have got the answer.

The stacked columns just show the percentages in a different way, but help visualise:

Stacked Columns - Final Comparison

You can see, for recent weeks in 2020, that Neutral questions are making up the vast majority of the questions and are steadily making up more of the total. Positive questions, although down, aren't going down as steadily, and so too are negatively received questions.

Final TL;DR:

Like my initial TL;DR, it appears that quality isn't really being effected, and in some regards it appears that overall there is possibly a "small" improvement due to the lower percentage of questions that are received negatively.

On the other hand, volume has now certainly been effected, with the last 2 weeks having more questions that any other week (and the last week gaining a huge increase). Perhaps I'll see, when this is all over (in x months time) just how this all panned out in the end for the volumes; but I'll let things get back to "normal" first. :)

xlsx file download: Stack Overflow Quality Comparisons

8
  • I like the contrast of the plots. Very well readable. May I ask what plotting software/library was used? Commented Mar 23, 2020 at 10:31
  • 2
    I just used Excel, @Trilarion. It's gone through quite a few changes since I made all the graphs. Especially the colours to get them consistent for all KPIs across the different ones. Think i got there in the end!
    – Thom A
    Commented Mar 23, 2020 at 10:40
  • 1
    Interesting to see 2 downvotes here. I appreciate that votes are different here on meta, so it would be interesting to understand what you disagree about the results here, or what you feel makes it unuseful or incomplete. If I can add more information, I'm happy to do so.
    – Thom A
    Commented Mar 23, 2020 at 18:29
  • I can only speculate but maybe the lack of any visible strong positive effect makes some people think that the question and answer may not be useful enough. In that case the question should have motivated more why an effect is to be expected. In general, I observed that people seem to look a bit more critical on self-answered questions. Still, the scores makes this a top question and answer on meta. Commented Mar 23, 2020 at 18:43
  • 1
    When doing analysis, the lack of any strong change is just an important as noting a change though, @Trilarion . When I wrote the question, I had no idea if there would be a change or not. I was intentionally neutral in the question; are the more, are there less? Is the quality of those questions different (specifically I didn't state higher or lower).
    – Thom A
    Commented Mar 23, 2020 at 18:45
  • The lack of an (strong) effect is usually only interesting if an effect is to be expected. Otherwise one would just assume that the two things are unrelated. Question and answer have the same date and time, you should have known about the content of the answer when posting the question?? Commented Mar 23, 2020 at 18:48
  • I wrote the question, and then did the research, @Trilarion . Also, when asking a question your self answering, it should still be in a format that expects an answer. That's something that's covered quite well in a blog post from SO (I'll find it if needed). If the question was phrased in a way that already knew the answer, it wouldn't be much of a question, would it. :)
    – Thom A
    Commented Mar 23, 2020 at 18:51
  • I often thought about similar things in the late months. I have a few ideas for questions on meta, but I'm never sure how much of the data analysis I shall put in the question and how in the answer. The question in this case could have been "Why is there no significant impact of the recent influx...?" (Makoto tried to answer that question I think) if the research would have been included in the question. One can always ask why, because the data will never explain everything. Anyway, that is only speculation. The downvotes might be for other reasons. Commented Mar 23, 2020 at 19:01
6

I feel like the data by Larnu is fairly complete and accurate, but I wanted to offer an anecdotal perspective.

The vast majority of people who have the privilege of working from home and are already Stack Overflow users wouldn't have had their habits changed terribly drastically when it comes to using the site, unless they weren't working for any reason.

I would not be under any impression whatsoever that there is an "influx" of home users who are also Stack Overflow users who also ask questions, for a few simple reasons.

  1. If you're asking questions, you would be doing so irrespective of your work location.
  2. If you're looking for answers, you would be doing so irrespective of your work location.
  3. We're always going to have an issue with question quality since...well, question quality is a bit nebulous, and the best you can hope for to establish if things are improving or not is a hearty ¯\_(ツ)_/¯.
  4. If someone needs help with something they're working on as a part of a product working for a company, they already have a network of help and guidance. That shouldn't magically change because now they have to work remote, and they would be getting the same guidance from the same coworkers as ever. (Note: those without this network could easily find themselves at a crossroads with their employment at a given company, since this would have a direct and indelible impact on their ability to deliver.)
4
  • I can image that many questions that would normally be asked face to face, to a co-worker that is next to you in the office, or a fellow student or teacher in a lab session at university, or your whizzkid nephew visiting your house, are now more likely to be asked at SO because the face-to-face interaction is not available and it is harder to reach people by e-mail or some collaboration tool than to ask a question here.
    – Marijn
    Commented Mar 21, 2020 at 13:14
  • 1
    @Marijn: I adamantly disagree with this notion. It's a culture shift to be sure, but just because one is working from home, the others that one relied on in the past for help/guidance aren't completely removed from the equation. It is entirely within reason that one can send an email (at a bare minimum) to ask a question to someone that is relied on. This applied doubly so for employees at a company. I have never heard of a company working in software that does not have some kind of instant messaging tool or teleconferencing tool to facilitate these kinds of things.
    – Makoto
    Commented Mar 22, 2020 at 17:07
  • @Marijn: That would also imply that there's a shift in the numbers, which I never believed there would be, and is also supported by the data posted by Larnu. More working from home doesn't necessarily mean more people asking from questions for reasons I've outlined above, and I'm having a hard time believing otherwise.
    – Makoto
    Commented Mar 22, 2020 at 17:08
  • 1
    "More working from home doesn't necessarily mean more people asking from questions" In fact, for the Week Commencing 08/03/2020, the volume of questions was several thousand lower than last year. There has been gains for W/C 15 March, however they are still lower than last year.
    – Thom A
    Commented Mar 23, 2020 at 14:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .