89

I've been on Stack Overflow since 2012 and have seen a fairly steady decline in the quality of questions asked in the tag over the years. However, I noticed a significant incline in the sheer volume of these questions in the last couple of years.

A lot of the very low-quality questions stem from every-day problems like syntax/parse errors, why isn't this code working PEBKACs, and other poorly-researched questions that could have been better answered by a quick Google search or just spending a little more time playing with the code, or even just searching Stack Overflow for similar questions that provided useful answers.

Sometimes even closing a question as duplicate led the OP to re-ask the same question with "this did not answer my question" in the title/body, when the duplicate clearly did. It seems that if the answer is not specifically tailored to their use-case/code, a general answer is unacceptable from their point-of-view.

I developed a theory that the majority of these questions come from students or hobbyists that have been misguided, to believe that Stack Overflow is a general help forum, where anyone can ask for and get some quick help with their code.

So I decided to head over to SEDE to see if there was any data to support my theory. Sure enough, I dug some interesting data, but I'm still not quite sure what to make of it yet.

The Hourly Trends

Looking at a histogram of when questions are asked, it seems that historically, the peak hours are usually between 8 AM and 4 PM UTC.

Enter image description here

So far it seems that, generally, day-light hours bring in the most questions, which if we go by the theory that the bulk of these questions come from students spending their days looking for help with their projects/homework, it might make sense.

The Daily Trends

Enter image description here

Looking at a histogram of questions by days of the week supports this notion even further. There're almost twice as many questions asked during week days as there are during weekends.

The Monthly Trends

Enter image description here

Looking at a histogram of questions by month further supports this theory as you can spot a significant ~20% drop, historically, during the start of the school year, and 5-10% upticks during holidays and summer months.

The Yearly Trends

Enter image description here

Looking at the data by year, you can clearly see that question scores are tending downwards, while the amount of questions are trending upwards very quickly.

In 2011, the amount of questions asked that were tagged , nearly doubled from the previous year in 2010. By 2012 we can see that aggregate question scores start to suffer and trend downwards as the amount posted questions rise and trend upwards.

Enter image description here

Even though is the 4th most popular tag on Stack Overflow, it's oddly the 7th highest scoring tag out of the top 10, lagging way behind less popular tags like , , and . Which leads me to believe that the quality of scores that the tag suffers from must be the result of a lack of precedence.

Most people tag their questions as just because PHP is involved in virtually every aspect of their stack in their web development process. So while the question might really be about JavaScript or Apache httpd, or even just HTML/CSS, the fact that PHP is somehow involved means the question likely gets tagged under php as well. Meaning that PHP tagged questions share in a lot of cumulative blame as well.

If it's not the result of masses being misled to believe that they can turn to Stack Overflow whenever they run into a problem with their PHP code, then what is it? What can be done to improve the overall quality of the PHP tag score on Stack Overflow?

Should I even care? Is this really a sign of a bad trend in the works that's detrimental to the PHP ecosystem on Stack Overflow or just typical behavior that's to be expected as the site grows?


Update

I'm including the monthly standard deviations scatter chart based on further discussion in the comments to see if there is any more meaningful data there that contradicts or strengthens my theory. It's come to my attention that the histogram of question distribution by month may not be as meaningful as it is presented to be.

So this scatter only includes data for complete years (2009 - 2015).

enter image description here

Showing my math

Here's a gist with the CSV dump of the data in the chart.

Here's a gist with the CSV dump of the aggregate monthly question data

A standard deviation is calculated as the square root of the variance. The variance is the average of the squared differences from the mean. The mean is sum of all members in the set divided by the number of members in the set.

So, for example, during the year of 2009, there are 12 months. The total number of questions asked are 20548. This is given from s = [645, 775, 909, 976, 1189, 1586, 2042, 2205, 2279, 2385, 2714, 2843]. Thus the mean of of the set s is 20548 / 12 which gives us 1712.333.

The variance is then calculated by the following function.

function variance(set, mean) {
    var sum = 0;
    for(var i in set) {
        value = set[i];
        sum += (value - mean) ** 2;
    }
    return sum / set.length;
}

So variance(set, 1712.333) gives us 575158.222

The standard deviation is then the square root of the variance giving us Math.sqrt(575158.222) == 758.391.

So on each month of 2009 if we take the number of questions asked, subtract the number of questions asked from the previous month, we can see how many standard deviations removed the current month is from its previous month.

Of course, for the first month we have no previous month so it's (645 - 0) / 758.391 for January of 2009. Then (775 - 645) / 758.391 for February of 2009... so on and so forth.

So far this shows there are some months that tend to be further from the standard deviation then others, but not consistently enough. There's probably some math error in my calculations here that I'm not aware of... So please do feel free to point out where I might have gone wrong.

I'm by no means a data scientist or have any advanced mathematical skills beyond the average Joe. So constructive criticism is both valued and welcomed.

59
  • 75
    Those patterns are not just limited to [php]. And those users are not being mislead either, it is the inevitable side-effect of the CEO's efforts to make SO more accessible. More about that in this post. Commented Sep 10, 2016 at 10:14
  • 5
    @HansPassant Thanks for sharing that post. It was very insightful. Glad to know I'm not crazy and that this is a fluidly evolving trend. Still not quite sure what to make of all of this. Is it a good thing? Is it a bad thing? But at least I can be sure it's a thing.
    – Sherif
    Commented Sep 10, 2016 at 10:47
  • 5
    @Kaiido It is not an issue when less good questions are asked. The issue is the questions being less good. The main cause are these too localized questions. Code dump of something complex including frameworks and unrelated code, why doesn't it work, and - with a bit luck - perhaps what's the expected behavior/output. People aren't investing much effort into reducing code to the smallest possible reproduce case. They don't even really attempt to show that they see where the problem comes from. I really wished I could close these more effectively … Perhaps give silver badges 3 CV weight.
    – bwoebi
    Commented Sep 10, 2016 at 15:01
  • 18
    @Sherif: "which if we go by the theory that the bulk of these questions come from students spending their days looking for help with their projects/homework, it might make sense." You could also go by the theory that these questions are from non-students looking for help with their work. The time of day, and day of week of the questions ultimately tells you little about who's asking. Commented Sep 10, 2016 at 16:53
  • 8
    @Sherif: "Looking at a histogram of php questions by month further supports this theory as you can spot a significant ~20% drop, historically, during the start of the school year, and 5-10% upticks during holidays and summer months." If your theory is that students are the majority asking such new questions, then we would see an increase during school periods and a decrease outside of them. But that's not what your graph shows. It's far more likely that we simply have had more questions recently, since the falloff happens when you started to count Sept '15 against Aug '16. Commented Sep 10, 2016 at 16:55
  • 5
    You should run these queries on the 40 mayor tags and then against the entire site... the trend is general.
    – Braiam
    Commented Sep 10, 2016 at 17:03
  • 8
    "day light hours" is rather judgmental. The sun never sets on . . . places where people use Stack Overflow, and there are large user bases in many time zones. The rest of your observations about quality are probably true, though. Commented Sep 10, 2016 at 17:35
  • 38
    I'm unsure how daylight hours strengthens the theory that students are causing the trends. I work during the day and typically ask questions during the day, whereas when I was in university I did my studying in the evenings...is this not typical?
    – ARich
    Commented Sep 10, 2016 at 17:39
  • 46
    I respectfully disagree with the conclusions drawn from your data--questions during the working day are unlikely to be from students in lecture. And if the number of questions on particular days of the week were driven by students, I'd expect to see an influx on Sunday (homework crunch), fewer questions in July/August and more questions in the fall semester months.
    – Luigi
    Commented Sep 10, 2016 at 18:23
  • 8
    @Braiam Yes, but PHP is just exceptionally bad. All-time, it has the lowest average question score of the top 25 tags, and ranks 4th in the top 50 only behind [vb.net], [excel] and (behold!) [wordpress]. In 2015 it looks similar, except that sql-related tags like [mysql], [sql], [sql-server] and [database] rank even lower, making PHP only the worst on the top 10.
    – Siguza
    Commented Sep 10, 2016 at 21:42
  • 12
    The stats here are pretty dubious, but so far no one has mentioned the troublesome choice of a non-zero base on many of the graphs. Commented Sep 10, 2016 at 22:05
  • 15
    People always forget the roomba when looking at trends over time. There's a big drop-off at september because recently-posted rubbish is still around, whereas crud from last year has been removed. You also haven't accounted for the fact that posts gather upvotes over time, so it's inevitable that older posts score higher. But, subjectively at least, there has indeed been a huge decline in the quality of content all over SO in recent years (and meta is full of people asking if anyone else has spotted it).
    – Dave
    Commented Sep 10, 2016 at 22:17
  • 14
    8 am to 4 pm UTC is not daylight hours for most of the world's population, nor for most of SE's users. That's the middle of the night and early morning in the Americas and afternoon/evening in central and Eastern Asia.
    – reirab
    Commented Sep 11, 2016 at 5:31
  • 4
    @Sherif: The raw data is not in question. The statistical inferences and graphical representations are (for reasons given by others), and throwing data into a charting tool is unfortunately likely to give misleading results much of the time. Commented Sep 11, 2016 at 13:06
  • 12
    @Sherif: That's not a personal attack. This post does not demonstrate a sound grasp of statistics and is making probably-invalid conclusions from the data and giving misleading impressions with its choice of presentation. I only know enough statistics myself to be dangerous, so I can't easily correct all the errors, only point out their general existence. Commented Sep 11, 2016 at 13:47

2 Answers 2

27

There comes a point where a resource like has reached the full breadth of what it can cover within the constraints of what Stack Overflow considers to be on topic. What remains is a lot of noise throwing itself at the wall, with a tiny bit of signal occasionally emerging and bringing more depth to the resource. This problem isn't at all surprising, and I'm not even certain that the cause itself is problematic - any encyclopedic resource should have completeness as a big part of its definition of success.

What remains is what to do with that extremely irritating cloud of noise from which a tiny bit more signal occasionally emerges. That's something that we're going to be spending a significant amount of time studying and building around as we kick off the second iteration of the Stack Exchange quality project (highlights from the last one on MSE and MSO, though there's more that unfortunately wasn't tagged as such). This project is perennial, it doesn't really end, we just take a look at the impact of major efforts every 1.5 or so years and then start planning new ones.

Since I'm about to get a little windy, here's a tl;dr; of the first things we hope to focus on:

  • Much more intelligent duplication detection: We're not showing people the right duplicate even though we probably have it.

  • Better stating our expectations of questions in the interface: We don't do a very good job of helping brand new users ask questions with all of the ingredients that we expect. We also don't do a very good job of providing incentive to bother with any of that at all. We're looking at testing a more explanatory and guided interface for brand new accounts (it would be optional, but strongly encouraged).

  • Better ways of filtering out noise: Let's come up with better ways of not showing total noise to our most engaged and knowledgeable users until we can better determine that there's some signal in there to mine.

All of this is still in planning and subject to change. But I wanted to put it out there now so folks know we're working on it. Now, here comes the wind:

Sometime later this week, I'm going to be announcing a special collaboration effort with the University Of Melbourne. A graduate project focused on fast analysis of questions in order to determine not only if they're duplicates, but quickly determine the best duplicate given the context of what's being asked. Interestingly, and unsurprisingly, the method also requires very quickly being able to determine if something isn't a duplicate.

They've gone as far as being able to validate their model through human analysis of many non-academic questions, but the expertise of our math, science and programming communities is needed in order to vet the rest of the data. They'll be going to 15 or so individual meta sites and providing a link to a tool where users with close privileges can say if any given questions are duplicates, similar but not duplicates, or in no way duplicate each other. This will help to validate the method, and we'll see what we can learn from it.

Even if we don't get the validation of the data that we expect, we're going to have learned a heck of a lot and we'll have many sets of data that have been human reviewed that can be used to test other models.

If things look promising, we'll then run Stack Overflow data through it, and open it up for validation. We do not want to do that initially because it essentially means analyzing every single question asked since the dawn of the site and putting folks to a pretty big task. While there's some value in doing it just to have the sets, we'd like a bit more confidence before investing that much time.

Later this week, or early next, I'll be posting something on Meta Stack Exchange to let folks know when the project is going to kick off, how they can help, what kinds of ideas we're looking to add, and what's in scope to get things started. There's some big stuff, but the majority of it is lots and lots of small things that could make a pretty big difference.

14
  • 1
    Wow, I had no idea this was a thing. I was actually working on figuring out a way to model the data that could demonstrate the most common duplicates not being marked as duplicate. Glad to know StackExchange is on top of this then. I'd love to see what the results are and any further reading on the modeling is of great interest to me.
    – Sherif
    Commented Sep 12, 2016 at 6:52
  • I take it you'll be writing a separate post about this duplicate testing
    – user3956566
    Commented Sep 12, 2016 at 6:56
  • Tim, do you plan to do anything this Fall with regards to increasing amount or impact of close votes? Do you intend to focus solely on duplicate questions and ignore the rest of close worthy ones this Fall?
    – gnat
    Commented Sep 12, 2016 at 7:22
  • 2
    @gnat That's something we're currently considering, yes. It's going to be on the list of things we want to talk about. While we want to avoid putting even more responsibility on the shoulders of few, we also want the few to be able to be more effective with the amount of time they're willing to give. If we can strike a balance there, I think we can do it.
    – user50049
    Commented Sep 12, 2016 at 7:50
  • @Yvette That'll be a blog post, actually.
    – user50049
    Commented Sep 12, 2016 at 7:51
  • sounds good, thanks. Wonder if this suggestion is on the radar? it's not so much about getting more questions closed but more about decreasing frustration of active users: "so long as somebody is (a) voting well and (b) bumping up against the limit, gradually raise the limit. If those votes aren't being used well, reset. (Possibly also start aging away the extras if they aren't being used...)"
    – gnat
    Commented Sep 12, 2016 at 9:58
  • 6
    "Better ways of filtering out noise: Let's come up with better ways of not showing total noise to our most engaged and knowledgeable users until we can better determine that there's some signal in there to mine." This is a bad idea. If you can determine that something truly is noise, it shouldn't be here. If you have an automatic filter that is that good, then it ought to be applied to users post when they post them, not to users reading what has been posted. Noise is noise, for both the experienced and the inexperienced alike. We do not want noise on the site! Commented Sep 12, 2016 at 19:01
  • 4
    @NicolBolas I think that battle's been lost. SO (the company) is just not going to risk scaring off the bulk of its user base. As Your Common Sense said, "Traffic is the king, and while it flows, nobody cares about such trifle matters like answer quality." Commented Sep 12, 2016 at 20:49
  • @NicolBolas The problem is, we can't truly determine what's not just noise. Going to be running some numbers soon on the efficacy of the helper queue from three months after it was launched until now in order get a better idea of how accurately humans can separate wheat from chaff, which gives us a more realistic goal of what automated things could be doing (a fraction of the accuracy humans achieve is likely). But not putting 'undecided' stuff in the faces of folks that have clearly expressed disinterest in seeing it is in the realm of possibility, so we're looking at it.
    – user50049
    Commented Sep 13, 2016 at 8:16
  • 1
    "Better ways of filtering out noise", hmm, sounds a bit like Allow users to optionally filter out low-quality questions and Feedback requested: New “recommended” homepage, phase 1. ;) I might suggest going back to those two discussions, they seemed to provide a decent amount of feedback.
    – hichris123
    Commented Sep 14, 2016 at 1:49
  • @hichris123 I think that is better to reject them outright. Why allow something if you aren't going to show it to anyone?
    – Braiam
    Commented Sep 16, 2016 at 14:49
  • @Braiam There's a state in which it's unknown if something should be rejected (some questions pass through the help & improvement queue two or three times, then go on to do very well). Essentially, we want to allow a filter for 'show me only known good, as of this moment'. It's in a different scope to do better about ensuring that the input is better overall, and the parts going into that are still being fleshed out.
    – user50049
    Commented Sep 16, 2016 at 16:06
  • @TimPost Am I missing something, or is this announcement not out yet? (As it's already being 2 weeks)
    – bwoebi
    Commented Sep 26, 2016 at 9:30
  • "Since I'm about to get a little windy, here's a tl;dr; of the first things we hope to focus on:" Almost 6 years later, I'm not really convinced that anything ever came of this. Commented May 23, 2022 at 1:05
1

I'm sorry, but my answer will definitely NOT be as high quality as the question is.

The trend you are noticing is not limited to PHP tag alone. Other tags suffer the same and the change in quality affects both the questions and the answers.

There are multiple reasons for that, here's a few:

  1. The truly-generic questions got asked in the mature tags. Such questions & their respective answers get the most hits from search engines as well and are as a result getting the most upvotes. If you repeat your statistical analysis in a while, you will see that numbers for "older" questions moved up more than the ones for the newer questions.

  2. Newer questions being asked are more specific, problem bound, simply because the generic questions have (all) been asked. There are still some being asked about a currently-popular library and are getting upvotes, but generally they don't get the upvotes simply because the audience for the question is much more limited than audience for a question dealing with a basic tag technique.

  3. If you adjust your statistical analysis to separate the grain questions from the chaff questions - and adjust that with users in a tag - you will see that chaff percentage in php tag does not significantly deviate from its percentage for the other tags.

  4. Continuing with grain vs chaff debate, the general upwards trend in chaff questions is a direct result of (mostly) two factors:

    4.1. - SO has become the primary source of help for pretty much everyone in programming nowadays (increasing user count significantly)

    4.2. - SO's policies allow for such questions and perhaps even encourage them. I'd venture to say that search engines take votes into account when ranking results and generally tend to link the higher-quality questions, resulting in "all being good".

  5. Sometimes, it's not the question, but the answer that spoils everything. My experience shows that it used to be "easy" to answer a question, today it's not: in our quest for reputation, we write shorter, less detailed, less fact-based answers (such as this one, perhaps?). I used to go check out the new questions and for the larger part, first page still had no answers. Today, even the newest questions already have answers even when all they should have received is duplicate & close votes.

In summary, I don't think php tag is alone suffering from your conclusions. Even if it compares poorly to other popular tags, that's maybe because it has many new students learning it? Or maybe because it has a steep(er) learning curve.

1
  • 3
    PHP definitely doesn't have a steeper curve than C++ or Python. If anything, I would expect a lower bar-to-entry would induce worse questions. Set the bar too high and you get no questions at all. Set the bar too lower and get too many questions. Set the bar just right and you tend to find a sweet spot of good questions and good answers.
    – Sherif
    Commented Sep 12, 2016 at 6:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .