8

A sampling of the questions at the stackoverflow.com homepage gives me the impression that:

  1. Questions now usually come with some sample code, and they do look answerable from the first glance
  2. The question list is no longer dominated by newbies with <100 reputation.

Can we thus conclude that the quality of the questions is actually improving at Stack Overflow?

Additionally, is there a way to quantify this improvement, and hence show that whatever recent measures that were put in place to boost question quality are actually working?

8
  • 32
    No, we can conclude that the algorithm used to select questions for the home page has improved. Dec 10, 2019 at 8:21
  • 11
    Question with code <> better quality. Many of those questions with code don't actually have anything that could pass as minimal reproducible example. Dec 10, 2019 at 8:22
  • 4
    @CodyGray, you sort of answer my first question, but you and I both know that that's not really what I was asking
    – Graviton
    Dec 10, 2019 at 8:23
  • 6
    Well, increase your sample size. Go to the new questions feed. Take a look at your favorite tag. Do you still like what you see?
    – Gimby
    Dec 10, 2019 at 10:28
  • 1
    How many questions did you sample? And how did you sample? Perhaps add the information to this question. Dec 10, 2019 at 11:28
  • 5
    This is the time of the year when students have to have their end-of-semester projects completed. Yay. Pretty sharp cut-off for some reason, the one in the spring lasts longer. Nobody can postpone Christmas, perhaps. Dec 10, 2019 at 11:36
  • 8
    Just based on the tags I "live" in: No, not really... Dec 10, 2019 at 11:37
  • 6
    @CodyGray FWIW here is an unfiltered search I tend to use to estimate question quality at the opposite side of spectrum so to speak. And per my observations it improved dramatically after 3CV change: poorly received questions are now like 4-5 times less frequent than before. Which means that selection algorithm, whatever it is, has less "sand" to throw away when picking pearls for the homepage
    – gnat
    Dec 10, 2019 at 12:15

2 Answers 2

3

Can we thus conclude that the quality of the questions is actually improving at Stack Overflow?

No, I don't think we can conclude based on "impression" (as you mentioned about your impression of the homepage), conclusions can only be drawn based on "facts".

I am not an expert on how SO is architectured and how well we can do analytics on it but I think, with my limited knowledge based on my experience with SO, that we can analyze "review queue" statistics to draw such a conclusion.

Additionally, is there a way to quantify this improvement, and hence show that whatever recent measures that were put in place to boost question quality are actually working?

I personally feel that this is too abstract because firstly there should be an exhaustive list of "actual measures" that were put in place and then running a check whether they achieved their purpose or not. I don't think there is such a list published by SO, and in absence of such a list IMHO the answer to this question is also 'no'.

1
  • What the company often used in the past was a simple quality criterion like bad (negative score or closed), neutral (score 0) and good (positive score, possibly with positively scored answer). Over the years however the bad questions made up about 20% constantly, and the percentage of good questions slowly declined from 60% to 50%. It may also just mean that people lost interest. Dec 10, 2019 at 22:52
3

Is there a way to quantify this improvement?

Oh yes. A blind experiment. I could select a number of questions randomly from a desired time range of StackOverflow and show them to you without any context like score or author or date. You then rate the quality of them and a few other users do that too and then we correlate the results with the hidden date property. The null hypothesis would be that the quality lately did not improve.

We would need to make sure that the selection of the questions is not biased, i.e. include deleted questions and always get them at a certain time point in their life.

If you give me 6-8 weeks and select the time point (I propose 1 day old questions, that's old enough to mature a bit but still young enough to not have lost all interest) I will try to setup the experiment. Maybe even the statistical power would be large enough for tag specific statements.

On the other hand I fear that question quality after 2014 may have been rather constant. If anything the 5 to 3 close vote reduction lately will have increased the average quality of open questions somewhat by closing more of the bad ones.

P.S.: Just realized. For legal reasons, I would have to display the author name and link to the original contribution, which would allow a person to see the date. The attribution requirement of the content license unfortunately does not allow to conduct completely blind experiments.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .