39

Lately I've been seeing complaints that we've taken away valuable tools to fight the rising tide of low quality questions. The assumption often seems to be that it's harder to close questions, so the social pressure to ask good questions has vanished. A typical example:

This all changed when Jeff Atwood left the company and the "Summer of Love" campaign in the summer of 2012 outlawed some common practices. A not-so-pleasant side-effect of programmers liking the SO site model was their response to questions that they did not think belonged on the site. SO users were afraid to ask questions, worried that not getting their ducks in a row before asking would get them responses that were intended to chase them away.

. . .

The changes in the summer of 2013 were very impactful as well. They removed ways to get poor questions closed. Particularly the kind that were commonly used to curate the site, like "Not a real question", "Not constructive" and "Minimal understanding required". The site owners considered these close reasons to be abused and replaced them with friendlier sounding reasons, the kind that cannot be used anymore as a blanket way to remove bad content.

As someone who observed both changes from a distance and thought they seemed helpful, I wondered if there was any evidence that these particular changes have hurt the community's ability to police itself. Are there more broken windows now and do they give users licence to ask bad questions?

15
  • 20
    Of the 15 newest [android] questions just now, 3 looked alright, 3 looked iffy, and 9 were garbage. Regardless of the underlying cause, any limitation on the tools to deal with bad questions is going to be met with resistance when ~60% of new questions are crap.
    – blahdiblah
    Commented Jun 3, 2014 at 23:21
  • 2
    @blahdiblah: Feel free to downvote and VtoC the garbage. The tools are there as always. The goal of this question is to determine if those actions will discourage more garbage questions. (The evidence suggests to me that they will not.)
    – Jon Ericson Staff
    Commented Jun 3, 2014 at 23:36
  • 2
    @JonEricson The tools are there, however it takes a non-short amount of time to get them closed. Usually a few hours or more... it does make me wonder what would happen if we closed them faster.
    – hichris123
    Commented Jun 4, 2014 at 0:40
  • 2
    I'm always a bit more reluctant to close than I should be because I've yet to see a closed question be reopened, even when it's clearly edited enough for the original close reason to be invalid. I've seen other people express the same sentiment. Maybe the reopen system does work and what I've seen hasn't been representative, but I think this perception does deter people from voting to close Commented Jun 4, 2014 at 10:19
  • 6
    I suspect that a major problem is simply that the quality of the questioners is decreasing, being rapidly diluted by "wannabe" app programmers who have no real programming background and are not at all willing to invest in a "liberal" programming education before rushing to create their viral apps. I don't know how you deal with this.
    – Hot Licks
    Commented Jun 4, 2014 at 10:55
  • 5
    @JonEricson The tools are insufficient. People using them are way too outnumbered. So we have no chance to discourage garbage questions. Be careful, your "evidence" might mislead you.
    – kapa
    Commented Jun 4, 2014 at 13:06
  • 1
    @kapa: I agree that the tools are insufficient. I've been thinking a lot about what Joel proposed on the latest podcast. In particular, downvoting a question doesn't currently reduce the odds it will be seen and answered. All a downvote does functionally is start new users down a path of being blocked from asking. That's a very blunt tool and easily worked around. What is needed is clearer and more direct signals to new users that they are headed down a bad path.
    – Jon Ericson Staff
    Commented Jun 4, 2014 at 15:50
  • 3
    @JonEricson A huge number of these "new users" will not care about any signals. We already give enough and sufficient signals. The problem I see is that people misunderstand the site. No matter what kind of fancy newbie-helping facility we find out, that won't help. What we need now is strict and very unpopular rules, that will drive lots of people away (among them some we might not want to), but will save the site. Of course there is a business side of things, I don't know about that. They would probably never let anything like this happen.
    – kapa
    Commented Jun 4, 2014 at 20:38
  • @kapa: I can tell you that the people you have in mind do listen to at least one signal: when they are blocked from asking questions, they let us know. Even the automated warning shows promise. I suspect a good deal of frustration regular users have right now is the sorts of signals we traditionally use (closing, commenting, and voting) have far less impact on new users than we wish they had.
    – Jon Ericson Staff
    Commented Jun 4, 2014 at 21:13
  • 1
    Blocking a newbie from submitting new questions is a pretty severe punishment (amounting to banishment), yet in other threads folks are concerned about hurting their feelings due to the wording of closing reasons. This seems to be getting things backwards.
    – Hot Licks
    Commented Jun 4, 2014 at 22:01
  • @Hot Licks: Part of the problem with all of this discussion is that we are talking about a wide variety of people when we say "new users". The sort of people who get banned ask awful questions and tend to stick in memory. But there are other new users who are conscientious and would ask decent questions. Being mean to the former may or may not help. But it certainly discourages the later from even trying. We cut off our noses to spite our own faces.
    – Jon Ericson Staff
    Commented Jun 4, 2014 at 22:13
  • We get complaints from a few newbies that they got cut off after some understandable bungling (or, very possibly, someone taking a dislike to them). For every one that complains you know 100 others are gone and never come back. But, as is being said, the real offenders are not getting the message. I think a different approach is called for, perhaps a "newbie score" that newbies can examine (perhaps presented every time they start a new question) and which will tell them what they need to do to improve.
    – Hot Licks
    Commented Jun 4, 2014 at 22:17
  • @Hot That's what it looks like, but remember we can't see their deleted questions. Usually the people posting here, complaining about being question-banned and appearing all innocent are not nearly so innocent. Rather, after being question-banned, they tried to "clean up" all their old questions and dug themselves even deeper into a hole. These offenders get the message we don't want their questions quite clearly. So do the people who quickly rise to our standards without ever being banned. The problem users are the ones who persist asking poor questions, not bad enough to hit the ban.
    – Cody Gray Mod
    Commented Jun 5, 2014 at 6:13
  • It seems to me that it would be effective to treat new user's questions like low-rep edits: only the asker can see the question and those with high enough rep to review can see the question but only within its review queue, and no answers can be entered until the question has cleared review. The asker would see one of: pending review (along with current status and reasons for fail votes), unaccepted status and reasons, or a posted question. The asker could edit at any time, resetting its vote (but limited to prevent abuse).
    – mah
    Commented Jun 5, 2014 at 10:12
  • Note on my above comment... clearly this would not work for low traffic sites because there are not enough reviewers to keep things moving. Stack Overflow, however, should be able to support such a review burden -- particularly if I'm right in thinking this would reduce other review queues. Plus, good or bad, it would give badge hunters another queue to (mis?)use.
    – mah
    Commented Jun 5, 2014 at 10:15

7 Answers 7

30

The primary difficulty with the theory is that more questions are closed now than before 2012 or 2013:

Questions closed by week

Question closes per week are the yellow line. The blue line represents the average number of hours between asking and closing for questions closed that particular week. Questions closed via the queue are presumably older on average than other questions. It should also be noted that this graph includes deleted questions, unlike the public query. Finally the red line shows new users for each week. Other than the large spike around the time of the close queue burn down, we've closed between 5,000 and 10,000 quesitons a week since Fall, 2012.

Even accounting for increased question volume, question closing has increased dramatically since the "Summer of Love". Here is the rate new questions are negatively scored, closed, and deleted by their creation week:

Percent closed

Compared to previous years, a greater percentage of new questions are downvoted and/or closed. Due to our automated deletion process, a large number of quesitons asked last summer are about to be deleted. The site is using rejection signals (downvoting, closing, and/or deleting) more and more often. As far as I can tell, comments are not holding back from criticism either. Extant comments that include the word "vampire" have increased since the "Summer of Love", for instance. In any case, I struggle to find evidence in the data that people have gotten softer on bad questions.

If the community continues to send strong signals that questions are unwelcome as the data suggests, why are there still so many unwelcome questions coming into the site? The cynic inside says confirmation bias; if you are looking for examples of bad behavior, the increasing numbers of questions will provide plenty of evidence. But I think the more probable diagnosis is that we are sending the wrong messages more often than not.

Singapore might be the premier example of the broken windows strategy working; for a while chewing gum was banned since it was associated with minor vandalism. But the theory fails when cultural values are not aligned among all participants. The Michael P. Fay incident demonstrates how an outsider could fail to catch important signals when entering into a culture that punishes minor offenses. Given that many questions come from outsiders, Stack Overflow bewilders a lot of askers.

Here's the problem with our the broken window model on Stack Overflow: the people we punish aren't learning and the people we want to ask are learning all too well. If you talk to a programmer who isn't already using the site, the skuttlebutt is that it's too hard to ask a question that won't be immediately shut down. Meanwhile, we are seeing patterns of behavior that suggest that our current system of closing questions and banning users for low quality posts results in some people spamming the site with questions and opening new accounts when banned. The folks who likely would ask decent questions if they weren't afraid are repelled by the close, downvote, snark, and ban system. Meanwhile the people who just want someone else to solve their problem are temporarily inconvenienced. All of this reinforces the problem: more bad questions repeated until answered and fewer "real" questions.

We've been experimenting with new signals. Recently we implemented an automated warning when posters are nearing a quality block. It's too early to know if that signal results in better posts, but it's part of a larger plan to send better signals. We like to think of Stack Overflow as a community where a little bit of public shame goes a long way. But the truth is the site passed the boundary ages ago. Other sites and some tags might still be community sized, but Stack Overflow as a whole is a publishing platform and the greater part of the audience is people who come here to read answers. Let's not go out of our way to alienate them.

19
  • 24
    "Meanwhile, we are seeing patterns of behavior that suggest that our current system of closing questions and banning users for low quality posts results in some people spamming the site with questions and opening new accounts when banned." - To be fair, with or without the question ban, these people would still spam the site with bad questions. They come here because of the high Google rank and will do anything to get people to do their work for them. Nothing we do could get most of these folks to change any part of their behavior.
    – Brad Larson Mod
    Commented Jun 3, 2014 at 22:48
  • 1
    @Brad Larson: Absolutely. The work Tim is doing is to find more effective ways to slow these folks down. Our current approach is failing.
    – Jon Ericson Staff
    Commented Jun 3, 2014 at 22:53
  • From cursory stats, ~20% of questions are closed and/or negatively scored in a day. It may help to make a similar query for the past (including deleted questions, but not spam if possible).
    – jmac
    Commented Jun 3, 2014 at 23:43
  • 9
    The seasons have changed. It's now the winter of 'how can I con those morons on SO to do my homework/coding/debugging for me so I can go down the bar while those naive idiots work for me in parallel'. Commented Jun 4, 2014 at 0:19
  • Luckily for me, I've had a few beers and I tend to hold back when drunk, else I would surely be banned by now. Commented Jun 4, 2014 at 0:21
  • Your graph shows the average score for a new question is almost 2 downvotes. Really? If so, I find that very depressing indeed.
    – Raedwald
    Commented Jun 4, 2014 at 6:00
  • 1
    @Raedwald: That's a new question that has been closed. It also includes questions destined for deletion.
    – Jon Ericson Staff
    Commented Jun 4, 2014 at 6:04
  • 1
    @jmac: Here's a version of the query that doesn't account for spam and doesn't include any deleted questions. Many of the recent rejected questions are destined for deletion, which explains the spike. Next week I'll try the same query on the internal database that included deleted questions.
    – Jon Ericson Staff
    Commented Jun 4, 2014 at 7:06
  • 6
    Your data shows average score decreasing over time, but # of answers remaining almost consistent (with only a slight decrease). This is one example (and I think a good one) which shows your probable diagnosis is spot on; questions that get closed, and get downvoted still get answers; and all the "problem people" (the drive-byers and help vampires) care about one thing; getting their question answered... and that signal hasn't changed.
    – Matt
    Commented Jun 4, 2014 at 8:41
  • 17
    ... until we stop answering their questions (or otherwise prevent them asking it in the first place) they're going to keep coming back; regardless of the downvotes they get, and regardless of whether their question ultimately got closed/ deleted.
    – Matt
    Commented Jun 4, 2014 at 8:44
  • 4
    +1 for the idea that unwarranted harshness is more likely to deter the kind of askers the site wants than the kind it doesn't. Commented Jun 4, 2014 at 10:14
  • 4
    More close votes might just be due to more crap coming in. The idea that more crap comes in in absolute numbers seems to be clearly true to me. Stack Overflow is still growing, especially in problematic countries. Jeff recently gave a talk to Indians (on YouTube) teaching them how to ask good questions.; So your theory that more close votes in absolute numbers implies that the tools have not gotten worse does not follow.
    – usr
    Commented Jun 4, 2014 at 11:00
  • 9
    @Matt is right, help vampires will not stop flooding SO with bad questions unless they stop getting what they want. Banning them from asking questions for a while alone won't work; we need a way to disincentivize answering bad questions. The first step is to stop upvoting answers to bad questions ...
    – itsjeyd
    Commented Jun 4, 2014 at 11:23
  • 1
    @Matt The only way to stop people from answering bad questions is to stop rewarding them rep for doing so - but there is no way that the SO community will ever approve that idea because they all value their precious rep far too much Commented Jun 4, 2014 at 11:48
  • 2
    Something to keep in mind once you have access to the full data: an awful lot of questions are deleted without being closed, either by the automated scripts or by their own authors. That's not all low-quality stuff, but... A disproportionate number of them are not good.
    – Shog9
    Commented Jun 4, 2014 at 15:29
16

I compiled such a graph a while ago as well from the SEDE data when looking for something measurable that shows correlation for what I thought I was seeing happening to SO. I plotted the number of closed questions as a percentage of posted questions. This is what I got:

enter image description here

I refrained from publishing it before, the data is too corrupted by the close campaigns that have been organized. March 2014 being the most notable spike, the one that was instigated to try to burn the close queue down. It isn't otherwise fundamentally different from what you got.

Take it for what it is worth, not much, there is however a notable trend present. The close rate suddenly doubled in September of 2012, right after the Summer of Love. And stayed consistently elevated after that. This doesn't indicate a problem with SO-users voting to close, it indicates a problem with more bad content getting added that needs to be closed. This is not a positive trend in any way shape or form.

What is completely invisible from SEDE data is whether we are able to keep up, are questions that should be closed actually getting closed? The chronic problems with the close review queue gives a strong enough hint that this is something to deeply worry about.

So yes, I really do strongly correlate the Summer of Love campaign with a sharp increase in the amount of bad content getting added. The signal is there. What I find particularly ironic is that the complaints from SO users about getting treated badly did not disappear. Just in that respect alone, the campaign was an abysmal failure.

These kind of complaints are a disease of a social web site. They are a simple trick to attract sympathy and getting your way, it plays on the normal human instincts that are by nature inclusionary. By making SO a more "friendly" site, it actually made the problem worse, it just added a lot more users that have something to complain about. Having to deal with the junk they add is making it worse for everybody.

12
  • 3
    The graph looks pretty similar to the close volume graph I have in my answer. Obviously, we agree that there is a problem. Where we disagree is the diagnosis and prescription. I don't know what the solution is, but turning back the clock won't help unless you also reverse the incredible growth Stack Overflow has seen in the last 5 years.
    – Jon Ericson Staff
    Commented Jun 4, 2014 at 1:12
  • 15
    Well, it is not something you have to worry about anymore. There's nothing to reverse, SO stopped growing in the fall of last year. These unilateral decisions established a new normal, a web site with a lot more bad content and no growth. Well done. Commented Jun 4, 2014 at 1:22
  • 8
    The social issues you mention are real. I notice many users begging for help and trying to coax answering users. Especially the methodically incompetent users do not care about learning. They try to extract an answer at all costs. SO was designed to prevent such behavior but it somehow fails to do so now.
    – usr
    Commented Jun 4, 2014 at 11:04
  • 7
    @HansPassant 'it indicates a problem with more bad content getting added that needs to be closed.' That is the conclusion that you want, not necessarily what is actually going on. It could just as easily be interpreted as 'it indicates a problem with veteran users casting close votes because they assume that all questions are bad and that the summer of love was the wrong thing to do'. You're making an assumption and you know what that makes you... Commented Jun 4, 2014 at 11:40
  • 2
    The old data in this kind of graph is typically misleading as many questions are deleted and not counted anymore. This often makes the past look better than it actually was. Commented Jun 4, 2014 at 12:24
  • 12
    @Clara - I never "know what that makes me" when you post these veiled insults to my posts. You need to treat me like the typical male chauvinistic geek that's so common in IT, slow on the up-take on indirect speech. All I get is that you don't like me for some reason, I have no idea why. You'll need to spit it out if you want to teach me a lesson. And of course, if you have an opposing opinion then spit that out as well, post an answer. Commented Jun 4, 2014 at 12:51
  • 2
    "Correlation does not imply causation". You've shown a correlation, but you have not shown a causation. I believe @ClaraOnager's comment is saying that there are other possible explanations for the correlation which would need to be ruled out before reaching your conclusions. Commented Jun 4, 2014 at 14:21
  • 2
    I don't want to put words in anyone's mouth, but perhaps @Clara is referring to the saying, "When you assume you make an ass of u and me." Commented Jun 4, 2014 at 16:50
  • @ThisSuitIsBlackNot Bingo! Hans Passant is the epitome of what I consider a bad user of SO. Obsessed with rep, unwilling to see anything from another point of view, incapable of change. Commented Jun 6, 2014 at 6:22
  • 6
    @Clara Have you actually taken the time to read any of Hans's answers? He might seem grumpy sometimes, but he's rightfully earned every bit of the reputation he has. There are few users who can hold a candle to the thoroughness and helpfulness of his answers, much less offer his degree of experience. To make a statement like yours is not only utterly misguided, but downright rude. If there's any type of person that we don't want, it's people who make comments like yours. Remember, this is a site where we discuss what would be best for the site; we're entitled to our opinions.
    – Cody Gray Mod
    Commented Jun 6, 2014 at 6:46
  • @CodyGray I must be very unlucky then because each of the interactions I've had with him have been very rude from his direction. I also find it funny that you say, in one breath, that everyone is entitled to opinions except me Commented Jun 10, 2014 at 7:23
  • @Cody - I think I found the post that set her off. I undeleted it when I saw the other posts. Be very careful what you say, you'll never live it down. Commented Jun 10, 2014 at 8:55
11

Broken windows and the good old days

The broken window theory suggests that, by fixing decay or destruction promptly, we can prevent it from being seen as normal and then spreading...

We've put a lot of work into closing questions in order to discourage folks from asking questions on certain topics: polls and GTKY questions were frequently justified by pointing to past examples that had been allowed or overlooked, and getting rid of them did ease that pain a bit.

But poorly-asked questions are another matter. To apply the broken window solution here, you'd have to first believe that these questions are being written by folks setting out to write badly, to follow the examples set by the other questions they're reading. We'd also need to get rid of all of them - not just close them, but delete them. And we'd have to have started doing this years ago...

Every community - online or otherwise - reaches a point where nostalgia kicks in with a vengeance. Remember back when these United States of America was a God-fearing nation of law-abiding citizens who respected their elders and taught their children the value of hard work and honesty? Yeah, me neither - and I sure as hell don't remember a time when I could open up Stack Overflow and not see crap floating on top.

Flapping herrings

I have a lot of respect for Jon and Hans, but reading their answers here I can't help but think they're... Well, standing in front of a hurricane looking for butterflies.

The big problem with measuring question quality over time based on which questions get closed is that it relies on something that has always been applied inconsistently. Even blatantly off-topic questions can persist for years without a single close vote purely due to their obscurity, and changing standards over time have muddied the waters further by making broad swaths of once-acceptable questions less acceptable. If nothing else, close voting requires at least 5 trusted users to care strongly about a given question, and that's a lot to ask for mediocre questions that few want to even visit much less read.

Just about everyone who has tried it realizes that even trying to gauge question quality based on score is dodgy due to a lack of signal... And there are many more active voters than there are closers.

Summer of review

Back in 2012, while the blog was all a-flutter with the talky sort of love, Geoff Dalgas & crew were hard at work on their own labor of love: a bigger, badder review system, one designed from the ground up to mobilize the community and get them involved in day-to-day moderation. By October, we had some stinkin' badges and for the first time in a very long time were actually making a small dent in the backlog of questions with pending close votes.

You might not know it from the sorts of discussions that tend to crop up on meta, but the revamped review system had a rather large impact on how Stack Overflow was moderated, on how new questions (and answers) were received, and especially on how quickly questions were closed - something that had been a pain point in the closing system for years. Coupled with more aggressive automatic deletion, and we're doing a better job now of getting rid of broken windows than ever before...

So, tl;dr: no, the theory does not apply to low quality questions and closing.

16
  • 12
    -1 for "we're doing a better job now of getting rid of broken windows than ever before" - I don't know in what kind of world you live, but the revamped close reasons achieved the exact opposite for me personally (the exception being the "typo" reason which is actually useful)... Also, this still doesn't solve the problem that someone breaks a new window every 3 seconds.
    – l4mpi
    Commented Jun 4, 2014 at 11:04
  • 2
    @l4mpi -1 for not understanding the answer, we really need downvotes on comments for meta Commented Jun 4, 2014 at 11:44
  • @ClaraOnager I do believe I understand the answer. I even agree with parts of it. But I completely disagree with the sentence I cited - the revamped close reasons and the current state of closing questions are not "better than ever before" for people who care about the quality of questions on SO; and as proven by the past discussions about this, many people agree with me here. Oh, and what happenend to your stance of removing downvoting completely?
    – l4mpi
    Commented Jun 4, 2014 at 12:36
  • 3
    Hm, but we are still losing. We are not even close to being successful in making the site a better place. We have better tools now, but the amount of junk entering the site is increasing with a far bigger rate.
    – kapa
    Commented Jun 4, 2014 at 13:00
  • 7
    Of course it doesn't solve the problem, @l4mpi - that's my point: the broken window theory only works if folks aren't motivated to break windows without ample examples of existing broken windows and you can actually fix them promptly. But no one needs an excuse to ask poorly. I've spent the past 3 years of my life answering barely-legible complaints from the very folks posting the worst questions on Stack Overflow, and lemme just tell you: none of them think their questions are all that bad.
    – Shog9
    Commented Jun 4, 2014 at 14:27
  • 3
    Heh... Absolutely, @kapa - everything increased. Question volume increased, so we added more moderators. When it kept increasing, we built better tooling because moderators alone couldn't keep up. So we've gotten really good at cleaning up after folks who make a mess... But there are still thousands of people making messes. At some point, we have to admit that, as nice as we've made the tools, we're using the wrong one for the job at hand. No one wants to hear that, but... It's still true.
    – Shog9
    Commented Jun 4, 2014 at 14:36
  • 4
    "none of them [the bad posters] think their questions are all that bad". Well, no, they aren't that bad in comparison to the average. That is precisely the point of the broken window theory. The existing average sets the social norm. One more bad question, what's the big deal? Don't assume the broken window theory is about intent, it's about unconscious behavior.
    – MSalters
    Commented Jun 4, 2014 at 15:04
  • 2
    If we could get the people asking these questions to read any existing questions before doing so, @MSalters, we'd be a hell of a lot better off. Getting rid of broken windows has been reasonably effective in cases where folks were putting the work in to ask reasonable questions that were simply not a good fit for the site, the folks who were aping some of the top-voting questions here - but by and large, they're not the ones contributing to the flood.
    – Shog9
    Commented Jun 4, 2014 at 15:14
  • 1
    @Shog9 I absolutely agree. The sooner the leaders admit that we need radical changes the better. I don't know exactly who the leaders are (mods are janitors, I know that) and whether they actually realize what is going on, and to the right extent. The tools are nice, but we need more. We need to stop faking that we are a nice, loving community welcoming everyone and singing Kumbaya together. We are not, but we are afraid to admit that. We are professionals and we are here to collect knowledge about programming.
    – kapa
    Commented Jun 4, 2014 at 15:14
  • 11
    Niceness is orthogonal to quality, @kapa. The problem is one of volume - so wasting effort on indiscriminate fly-swatting is as much a part of the problem as wasting effort on indiscriminate hand-holding. The most efficient way to discourage someone asking bad questions is to just ignore them: don't fix their post, don't leave comments, don't close or answer - downvote it so that fewer people see it, and focus your efforts on questions that can be of use to others.
    – Shog9
    Commented Jun 4, 2014 at 15:19
  • 2
    @l4mpi The revamping of the close reasons and the revamping of the close vote review queue are two completely different things that happened around a year apart. You're saying that changing the close reason text didn't change the site's moderation (which of course it didn't, because it wasn't designed to, it was designed to make the existing moderation clearer to users unfamiliar with the community, not to change how content was moderated) whereas changing the review queues did change how the site was moderated, by providing more effective tools for people to actually moderate stuff.
    – Servy
    Commented Jun 4, 2014 at 15:28
  • 4
    @Shog9 You're absolutely right that new users coming to this community aren't going to determine whether or not to ask bad questions based on the existence of other bad questions, for the most part. There is one case you didn't consider though. When people come here, ask bad questions, and then get answers, it encourages them to continue posting bad questions. When people come here, ask bad questions, and don't get answers then they learn to either ask better questions, or leave. If we could actually keep users to just a small handful of bad questions each, that would help a lot.
    – Servy
    Commented Jun 4, 2014 at 15:31
  • 3
    In many cases, we do that @Servy. I should probably publish some updated stats on the q-ban system, but short of it is that it doesn't kick in early enough; Tim's been working on a design for throttling users who can't ask even one good question, which could potentially kick in after a single question.
    – Shog9
    Commented Jun 4, 2014 at 15:40
  • 1
    @Shog9 Yeah, I know Tim has been doing a lot of work on it, and I know that it's not nearly as effective as would be best, but it's at least something. It makes moderation less effective than it should be, but not useless, which is something.
    – Servy
    Commented Jun 4, 2014 at 15:43
  • 1
    If you're worried about what other people are doing, you're always gonna be worried, @kapa. Do the best you can with the tools you have, give advice to others when they're receptive to it.
    – Shog9
    Commented Jun 6, 2014 at 3:13
6

I think the real broken windows in our community are the answers to bad questions not the bad questions themselves. If you really want to get down to the cause of the problem of poor questions, you don't need to look any further than the half dozen answers found beneath many of them.

On SO we do an awful lot to try to round up and deal with problematic questions and the users who ask them, but what are we doing about the users who answer the problem questions?

Are we rounding up and punishing the addicts and letting the dealers roam free?

Perhaps as a community we need to do more to discourage users from answering bad questions, and maybe even punish them when they do.

My point here isn't that bad questions aren't a problem, but that the answers to the bad questions are the real motivating force. Until we do something to address the answers, the questions will keep coming.

1
  • I think your answer can illuminate a lot of things as an extended metaphor. Are the dealers really to blame, or are they just trying to fight off poverty? There's not enough rep to go around anymore so the high density low education areas are turning into something like ghettos.
    – Seph Reed
    Commented Apr 1, 2016 at 15:43
5

I don't think throwing any statistics at the question will help in any way. I would say that the much cited "Correlation does not imply causation" applies here to anyone who sees anything in the statistics. Why that?

Because I think the reason why people add bad content is completely disconnected from existing content.

Common ideas for why a lot of people do post bad content is:

  • They do not know about the rules of our community and don't want to invest time to learn them
  • They do not care about the rules of our community
  • They do not understand the rules of our community and have no incentive to do so
  • They are interested in getting their question answered, not more.
  • They are not interested in the SO mission statement of "building a good Q&A database"
  • They are lazy

So the question to ask at this point is: will improving the content of the site in any way influence any of the above reason? I think the answer is clearly: no. No matter if the content of SO is bad or good, mostly crap or all gems. People will still be lazy, not interested in the mission statement, only centered on their question, won't care about the community, don't want to understand it and don't want to invest anything to get to their answers.

This doesn't mean that increasing quality by close voting is bad or useless, it just means that it has no connection to the motivation of most of the people that we see posting crappy questions.

When something advertises itself as being 100% free, then it will attract people that want to use it, for 100% free.

5
  • 4
    Your fourth bullet is the root of the issue here; any solution that doesn't stand between an asker and his answer is destined to fail.
    – Shog9
    Commented Jun 4, 2014 at 15:34
  • Your answer looked good up through your bullet points, but your conclusions aren't quite right. When the moderation tools available don't prevent an answer they don't accomplish much. However, at least in some cases, questions get closed before they are answered, preventing answers, questions get downvoted enough to get less attention, preventing answers, users get question banned, preventing them from asking more bad questions (which prevents them from getting answers). So these tools aren't completely ineffective, just less effective than they should be.
    – Servy
    Commented Jun 4, 2014 at 15:41
  • If you look at my answer, the point is that reluctance to close and downvote questions can't be the cause of more bad questions. My argument is that if you don't have correlation, you can't very well claim causation.
    – Jon Ericson Staff
    Commented Jun 4, 2014 at 15:42
  • @Servy: I am not saying that this happens always, this is why I tried to use wordings such as "a lot of people". From my feeling, it would apply to the majority of people posting bad questions, but its hard to get hard facts on that. There are cases where the tools work, but as Shog9 said, we need solutions that stand between the asker and his answer to overcome this problem.
    – PlasmaHH
    Commented Jun 4, 2014 at 15:45
  • @JonEricson: My point is that starting with statistics here won't get you anywhere. This might be seen as an XY problem even. What we really want is to get more insight about people asking bad questions and how to prevent that and if our current tools are useful. The statistics won't help. Either they say "no correlation" (which they no doubtly do), which amounts to "we can't say anything about the people posting bad questions" or they do, which amounts to "we can't say anything". Starting with thoughts and hypothesis and then looking for supporting statistics might lead to insights about them
    – PlasmaHH
    Commented Jun 4, 2014 at 15:51
-1

The data provided suggests that about 1000 questions are closed a day (correct if I'm wrong).

Closing 1000 questions a day is a job for 100 voters (assuming it takes 5 to close and 50 votes limit a day: 1000*5/50=100).

On the other hand, there are over 20,000 users who can close, that is about 200x times more than needed for current job (and, just by the way, about 20x more that would be needed to close all the 8,000 questions Stack Overflow gets a day).

Above makes me feel that somehow (intentionally or not), current closing process "utilizes" less than 1% of theoretically available community moderation power:

We designed the Stack Exchange network engine to be mostly self-regulating, in that we amortize the overall moderation cost of the system across thousands of teeny-tiny slices of effort contributed by regular, everyday users...

Users with 3,000 rep can cast close and open votes...

15
  • 2
    The last full week for which we have data shows 6450 questions closed which is a bit under 1000 a day. But the question is: assuming we closed every bad question as it was asked, would that solve our problem? My guess is no. Closing questions seems to not be a deterrent for the sorts of askers who are most annoying.
    – Jon Ericson Staff
    Commented Jun 4, 2014 at 17:48
  • The thing is that we can't possibly expect everyone with the power to use actually use it. For me, I've tried to use the close vote queue. But I rarely use more than 5 close votes a day because I lose my patience very quickly. The only way I can get myself to use all my close votes is if I blindly closed everything. But of course that's not the way it should be done. (even though 80% of things in the close vote queue should be closed)
    – Mysticial
    Commented Jun 4, 2014 at 17:49
  • 2
    @Mysticial did you try aiming lower? in The Queue, I aim 10 or maybe even 5% that literally shoot me in the face screaming close me and skip the rest (possibly along with known good audits but I don't care). takes 10-15 minutes to spent 40 votes... and feels quite satisfying I'd add
    – gnat
    Commented Jun 4, 2014 at 17:57
  • @JonEricson I can't make any conclusions, only point the fact that there seems to be less than 1% participation. Hard to tell what could be if it was raised to 2-3%...
    – gnat
    Commented Jun 4, 2014 at 17:58
  • 1
    @gnat That's an approach I haven't considered. I might try that sometime. :)
    – Mysticial
    Commented Jun 4, 2014 at 17:58
  • @Mysticial ...by the way, your example (along with some stats) suggests that somehow (intentionally or not) system is designed in a way that seems to repel users who are willing, capable and qualified to participate
    – gnat
    Commented Jun 4, 2014 at 18:28
  • @gnat It definitely repels - quite strongly. Since the current system is basically asking the experienced users to clean up the crap when they could be doing other more productive things. But these experienced users are also the most trusted ones to do the cleaning. If you can't ask them to actively participate perhaps the moderation should be shifted towards passive participation. Where the lack of an experienced user getting involved is a sign that the question is bad. Which seems exactly like what the latest podcast is leaning towards.
    – Mysticial
    Commented Jun 4, 2014 at 18:34
  • @Mysticial agree, I just wonder if this is intentional or not. I sometimes think what could have happened if 100-200 (just 100-200) more users than now become regularly involved in CV reviews... and what could be consequences... and whether SE team guys ask themselves similar questions
    – gnat
    Commented Jun 4, 2014 at 18:47
  • 2
    Constantly, @gnat. That was the explicit purpose of the recent changes we made to highlight the close queue on the top tag pages (after some ad-hoc tests I ran at the end of last year). I'll pull reviewer stats once the datacenter move is done, but here are some slightly outdated ones - and you can see the uptick in raw reviews yourself.
    – Shog9
    Commented Jun 4, 2014 at 19:27
  • 3
    I stopped closing (and being active on the site also) for some time, because I felt I'm not making a difference. The binding duplicate close votes brought me back. Makes me feel my efforts are not completely worthless. Why other people do not close vote? There is a very strong and widespread misunderstanding among community members (including lots of really experienced users) that closing and downvoting is negative, and they want to be nice and polite. What could we do about this?
    – kapa
    Commented Jun 4, 2014 at 21:08
  • @kapa put yourself in shoes of SE team. Imagine there is a potential power out there, capable of closing 20x of all the questions that come in daily. Would you exercise some... caution about unleashing it? I certainly would. Though nowadays things changed in a sense that team seems to have figured ways to control that power, first of all control questions that enter queue - along with possibility to use audits to throttle things and changing review limit this makes it safer to play with closing power
    – gnat
    Commented Jun 4, 2014 at 22:18
  • @gnat They are certainly not comfortable shoes nowadays, I feel their pain. Sometimes you simply have to unleash the beast, and give it two chainguns. Caution is for times when adjustments can make a difference.
    – kapa
    Commented Jun 4, 2014 at 22:32
  • Yes, lots of users with close vote privileges still don't close questions. They have it in their head that it is "mean" or "not nice", yet they still feel that the site is going downhill and have drastically decreased their participation. It's a very difficult problem to solve. We tried renaming "closed" to "put on hold" to make it seem nicer, but didn't fool anyone. Community moderation has its advantages, but it isn't perfect. We need bigger guns. We need steely-eyed moderators that will hit that close (or delete) button without community involvement. It isn't popular, but it's necessary.
    – Cody Gray Mod
    Commented Jun 5, 2014 at 6:20
  • @Shog9 no matter how you turn it, closing 1000 questions a day is a job for 100 voters, while there are 20,000 eligible users at SO. That means, only 0.5% of "closing power" is utilised. I would be very interested to learn what could be the reason for that, especially now that SE team learned how to control what enters close queue (ie to prevent closing overuse if it happens)
    – gnat
    Commented Jun 5, 2014 at 7:30
  • @CodyGray well, it's probably not really bad that majority of 3Kers are focused on contributing / editing content. I merely wonder why it looks like this is 99.5% majority. Why not 99 or 98 or 97?
    – gnat
    Commented Jun 5, 2014 at 8:11
-5

One interesting thing I've noticed is that some closed questions actually had the answer I was looking for posted before the question was closed.

Because of that I question the wisdom of closing some of those topics and the current topic closing "rules".

29
  • 4
    Indeed, "closing" in the current "ban-on-answers" implementation is mostly counterproductive. What would be far better would be a status where an answer was still allowed, but could earn no reputation - then those who feel the question deserves an answer can still provide one, but no one has an artificial incentive to answer just to advance their reputation score. Commented Jun 4, 2014 at 14:38
  • 5
    @ChrisStratton The whole point of closing a question is because questions that are closed are very likely to produce low quality answers and very unlikely to produce high quality answers. It actually harms the site to try to answer questions that are too broad, missing necessary information, unclear, offtopic, etc. It's important to not just remove one of the incentives to answer, but the ability to answer. If/when the question becomes answerable it can be reopened.
    – Servy
    Commented Jun 4, 2014 at 15:23
  • 2
    @Servy it's precisely in the frequently erroneous determination of what is a bad question where your idea breaks down. Trying to get these questions re-opened to provide the answers that would help the asker and future visitors (such as Michael is reporting here) wastes huge amounts of time, and often fails when the mistaken closers remain stuck in unfounded opinions. Having a non-binding status instead would solve all the problems - those who don't want to deal with these questions could have a viewing filter to ignore them, while those who find them useful don't have to convince others. Commented Jun 4, 2014 at 15:44
  • 2
    @ChrisStratton Erroneous closures are quite rare in my experiences, and reopening such questions before the reopen queue was hard, now it's very easy. Zero of the original closers need to change their mind, so them not doing so is a non-issue. The problems cause by people answering questions that they shouldn't causes way more problems than the short delay of questions not being reopened very quickly in the very rare erroneous closures.
    – Servy
    Commented Jun 4, 2014 at 15:52
  • 1
    @Servy my original point is that useful and valid answers are being made to what some people deem to be poor questions. I keep finding good answers posted to questions that have been closed. Commented Jun 4, 2014 at 15:57
  • 2
    @ChrisStratton So you want people to be able to keep asking bad questions. You consider it an important aspect of the site to allow bad questions, and encourage them to be answered. That is contrary to the design of the site, it's reasons for being founded, and the values of the core members of the community. SO was specifically designed to be a place to find high quality content, where low quality contributions would not be accepted, and that users would need to learn to post up to a high standard of quality. You want to remove that standard, making SO like every other trash forum out there.
    – Servy
    Commented Jun 4, 2014 at 16:01
  • 2
    No, I want people to be free to ask good and important questions and answer them, without being harassed by ignoramuses too unaware of the topics to realize the importance and value of these questions and their responses. If such people want to have a system (short of the present answer bans) to mutually mark questions as unworthy of their attention and reputation scheme, I'm all for that, as getting unfriendly eyes off questions would do everyone a service. Commented Jun 4, 2014 at 16:02
  • 1
    @ChrisStratton If you have evidence that a significant percentage of closed questions don't actually meet the site's guidelines for closure, then you'll need to support that assertion, because I see tons and tons of open questions that should be closed and very few closed questions that should be open. If you feel that any given criteria for closing questions isn't appropriate, and that questions meeting those criteria are actually worth answering, then you'll need to explain your reasoning, in detail, and also make sure to look through previous discussions on why that reason was added.
    – Servy
    Commented Jun 4, 2014 at 16:09
  • 1
    @ChrisStratton It sounds like you simply think that most questions that don't actually meet this site's guidelines are actually good questions. You think that people should be encouraged to ask/answer questions that are unclear, too broad, opinion based, off topic, etc. (Or at least that's what you seem to be asserting.) The extensive history of the site has shown this to not be the case, that answering these questions generally results in low quality answers, and that it's far better to ensure that the question is fixed first.
    – Servy
    Commented Jun 4, 2014 at 16:12
  • 2
    @Servy - I'm not talking about off-topic questions, but I do admit that questions asking for opinions or recommendations have provided useful answers for me in the past. I fail to see why opinions are invalid answers. In a programming context, requests for, for example, a good icon editor that works with xxx seems valid to me, but are often closed. Yet I find the answers given before the question is closed to be useful. Now, I would agree that if it descends into i'm right/you're wrong type posts it is problematic, but if that is avoided the answers have value. Commented Jun 4, 2014 at 16:33
  • 2
    @Servy - no, I want to see content which is high quality as measured by it's utility and importance. To accomplish that, we need to stop encouraging people to ban material they do not understand, or better yet, let the viewer be the one to decide what they personally find to be of high quality. The problem is when some shift their attention from contributing the site, to trying to force everyone else to conform to their personal beliefs - and the bigger we get, the more fatal that problem becomes, so we must step away from that approach. Commented Jun 4, 2014 at 16:35
  • 2
    @MichaelVincent Such discussions have taken place hundreds of times on meta over the years. I can only cover the high points in a comment, and writing out a detailed explanation of exactly why is well beyond what comments can support. I suggest you look through past meta discussions on the topic to learn more about why that close reason exists, as there is considerable evidence to support it being there.
    – Servy
    Commented Jun 4, 2014 at 16:37
  • 2
    @ChrisStratton The close reasons have been added because considerable evidence has shown over an extended period of time that these types of questions don't result in high quality or useful material. That you seem to think most of these questions are of high quality tells me that you simply have a very, very low bar for quality, and that you want to see basically everything be welcome here. If that's what you want, then SO simply isn't a site well suited to you. It is specifically designed to be a place with a high standard of quality, not a place where just anything belongs.
    – Servy
    Commented Jun 4, 2014 at 16:40
  • 2
    @Servy ok ... as long as google is still kind enough to bring those topics up in the first page or so I'll still benefit. Sadly, I've just had a question closed by people who think I asked a duplicate question and point to a previously asked question - that I had referenced in my question stating it helped but not answered my question. Bizarre. It is so frustrating. Commented Jun 4, 2014 at 16:44
  • 3
    @ChrisStratton The fact remains that you're asserting that everything belongs here. We've seen what happens when you allow and encourage people to ask just anything. Take a look at Yahoo Answers, or virtually any other Q/A site outside of the SE network. Are the all entirely garbage, no; do they have some good content, yes; however they are less effective at answering questions, and dramatically less effective at helping anyone other than the OP get answers to their questions. SO was designed to be different. You want it to be like everything else.
    – Servy
    Commented Jun 4, 2014 at 16:45

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .