99

After reading "My Love-Hate Relationship with Stack Overflow: Arthur S., Arthur T., and the Soup Nazi" one of our developers wondered what would happen if we didn't have close votes. Disclaimer: I don't know of any actual plans to actually get rid of close votes. Like legalizing marijuana, a change like this isn't something that can be done overnight. It probably makes the most sense to think of this question as alternate reality fiction.

The most obvious result of no close votes is no closed questions. Since all questions would be open, that would allow more answers. While it's impossible to know how many more answers, we can make a guess:

When closed questions are closed and when they are answered

This graph looks only at questions that are currently closed and shows how long (in minutes) it took for them to be answered and closed. The x-axis is the time elapsed in minutes and the y-axis is number of events. There are more total answers than closures because I'm only counting the time of the most recent closure, but including all answers. (I've linked to a SEDE query so that you can see what I'm up to. But notice that the public data does not include deleted posts.)

Even though I'm only looking at closed questions, many of them are answered and some are answered multiple times before they are closed. Including deleted questions and answers, half of all closed questions (50.6%, in fact) do have answers. On average, closed questions have 1.2 answers compared to 1.6 for all questions. So we can estimate that each closed question "costs" roughly half an answer.

I put "costs" in scare quotes because, of course, theoretically answers to bad questions are themselves bad. I know voting isn't a perfect measure of quality, but it's the best we have:

N        question state answer Score Score > 0 Score < 0 Score = 0 
-------- -------------- ------------- --------- --------- --------- 
 1298965 closed         2.211         50.7      6.6       42.7      
14593333 open           2.013         55.7      3.6       40.7      

While closed questions are more likely to get bad answers, on average the answers on closed questions seem better than on open questions. How can this be? It turns out, the results are skewed by questions that are closed late, which I define as a month or more after initially asked:

N        question state answer Score Score > 0 Score < 0 Score = 0 
-------- -------------- ------------- --------- --------- --------- 
 838328  early close    1.099         45.2      7.4       47.4      
 460642  late close     4.236         60.8      5         34.2      
14593415 open           2.013         55.7      3.6       40.7      

There are probably other ways to slice this (such as number of answers, question score, or views), but questions are closed early when the answers are likely to be unhelpful and closed late when additional answers are likely to be unhelpful. It's the difference between a question that's unclear and a question that's been answered too many times already.

Of interest, here is the same query excluding deleted posts:

N        question state average Score Score > 0 Score < 0 Score = 0 
-------- -------------- ------------- --------- --------- --------- 
474142   early close    1.687         63.6      2.2       34.1      
252388   late close     4.741         71.1      1.9       27        
12983819 open           2.268         61.6      1.5       36.9  

The other result of no closed questions might be fewer deleted questions:

N       question state deleted % 
------- -------------- --------- 
 897851 early close    67.6      
 176921 late close     44.7      
9916795 open           14.4   

I say "might" because a good percentage of deleted questions are triggered automatically. There's no reason to not also change the triggering conditions to take lack of closure into account.

Finally, no close votes would likely drive some users away. Close votes represent a commitment to quality and revoking the privilege to cast them would signal Stack Overflow doesn't care about quality. Or at least it would in absence of some alternative method of politely declining to field certain questions. Without knowing what that alternate method might be it's pretty much impossible to guess how many people would walk away from Stack Overflow.

On the other hand, the close vote mechanism inserts a certain amount of friction in the process of asking and answering questions. If we could swap it for a method that reduces the barriers to entry without sacrificing quality, it's certainly possible we'd attract more new users than the users we lose.

We spend a tremendous amount of time collectively trying to close questions. I started this thought experiment expecting to find signs that closing questions pay back that time in terms of saving effort on the part of answerers. Now I'm not so sure there is a net benefit. What am I missing?

48
  • 18
    Closed questions have higher answer scores than open ones. Interesting...
    – Mysticial
    Commented Feb 19, 2015 at 0:14
  • 66
    vote to close: not a question
    – Blob
    Commented Feb 19, 2015 at 0:38
  • 83
    Why is attracting new users a goal in itself? This isn't a social gathering.
    – jscs
    Commented Feb 19, 2015 at 0:50
  • 18
    @JoshCaswell Probably because SE wants SO (and the other sites) to continue growing - which implies getting new users. If they didn't want new users, they would've closed registration.
    – Mysticial
    Commented Feb 19, 2015 at 0:52
  • 9
    @JonEricson Has anyone considered the idea of "showing people what they want to see"? Established users love to see bad questions closed. New users get butthurt when they get negative reinforcement. One way would be to show the closed notification to established users, but not to new users or logged out users. So they "think" their question is still open, but in reality it's closed and nobody will be able to answer. So you save them the butthurt by replacing negative feedback with no feedback. Of course this doesn't help them get out of bans, but we optimize for pearls not sand.
    – Mysticial
    Commented Feb 19, 2015 at 1:04
  • 18
    @Mysticial: That sounds a lot like a hell ban. Jeff and Joel talked about why we don't use that on the podcast once. (The conversation starts 25 minutes in or so.) The problem with the idea is that it's impossible help the user improve; even better than hiding bad askers is turning bad askers into good askers.
    – Jon Ericson Staff
    Commented Feb 19, 2015 at 1:24
  • 64
    I may not be reading enough into it, but this seems like a "what-if we were like yahoo answers just with cooler buttons to click on?" The thing you're missing, is likely how many people have became better question askers because we closed their question(s). I read somewhere, probably blog, that SE is proud to make people in general, better question askers, no matter what the subject is. Not being able to close questions would be detrimental to this SE goal.
    – CRABOLO
    Commented Feb 19, 2015 at 2:39
  • 53
    @Mysticial write once, read many -- Stack wants to attract the right kind of users. Those who have questions, sure, but mostly the experts who can provide answers. So it is true to say that Stack wants to filter its users a bit. If all you want is a quick answer fix, no need to log in, just read what is already there. If all you want is new users, at any cost, of any quality, it is doubtful any experts would stick around. Commented Feb 19, 2015 at 6:11
  • 18
  • 16
    "If we could swap it for a method that reduces the barriers to entry without sacrificing quality, it's certainly possible we'd attract more new users than the users we lose" - Reduces what barrier to entry? Reading through the tour and help center? Writing half-decent english? Understanding that a question must be somewhat specific to be answerable, or that a debugging question should include a self-sufficient piece of code demonstrating the problem? SO already doesn't have much of a barrier for anybody who is willing to read and/or lurk a bit, reducing it would mean you've got nothing left.
    – l4mpi
    Commented Feb 19, 2015 at 9:46
  • 12
    @JeffAtwood your comment hints at Shirky's A Group Is Its Own Worst Enemy #3 in the Four things to design for which starts out "Three, you need barriers to participation. This is one of the things that killed Usenet. You have to have some cost to either join or participate, if not at the lowest level, then at higher levels. There needs to be some kind of segmentation of capabilities."
    – user289086
    Commented Feb 19, 2015 at 14:19
  • 10
    @JonEricson the mean can be misleading when there are extreme outliers pulling up the scores. While its not as simple as avg(score) it might be useful to reexamine the 'average' numbers with the inner quartile mean instead. which should reduce the impact of those very highly scored historical questions and answers.
    – user289086
    Commented Feb 19, 2015 at 15:12
  • 6
    @Rachel should someone be prevented from acting to quickly close a "help, how do I make a triangle with stars in C#?" if they lack any answers in C#? If so, how do you differentiate the ones that require expert knowledge in the tag vs basic reading comprehension?
    – user289086
    Commented Feb 19, 2015 at 16:57
  • 8
    @bluet: One of the issues with the current close system is that it claims to initiate a process in which people get better at asking questions, but the process usually ends right there. People often act as if closing bad questions is a good end in itself. But that's not clear from the data I'm looking at. Closing does not stop most answers and it does not hasten the deletion of terrible questions. I'm thinking about looking at whether it encourages edits and self-improvement next. If closing helps people be better askers and programmers, it would be good to demonstrate that.
    – Jon Ericson Staff
    Commented Feb 19, 2015 at 17:39
  • 10
    I'm all for legalizing marijuana, but I don't know about getting rid of close votes. (The reason closed votes get so many answers is that they're easier to answer and thus attract rep whores.) (But I will observe that the questions that really deserve to be closed (mainly failed to RTFM) rarely are, while a fair number of "questionable" ones do get closed.)
    – Hot Licks
    Commented Feb 19, 2015 at 18:52

18 Answers 18

85

I started this thought experiment expecting to find signs that closing questions pay back that time in terms of saving effort on the part of answerers.

What? Closing questions isn't about answerers' time. Answerers have proved (and verbalized) time and again that they don't care if a question is good, coherent, useful, or answerable. They (we) love posting and trying to help. It's not about them, just as it's not about the askers. It's about polluting the "long tail".

A closed question is one that isn't going to produce a valuable document for the future. It's one that the voters think is going to clog up search results with dead ends. That is what makes user-applied closure a killer feature. Pruning.

Given that, it seems to me that the search-inbound views on closed questions, and the relationship to votes (and the "did you find this answer useful" anonymous feedback thingy) on the answers are one, maybe the, big missing piece here.

This will need to be controlled for time, of course, and the "boat programming" origins of the site. (I freely admit, though, that I don't know what you'll find if you inspect these data.)

22
  • 7
    I should point out that the one immediate effect of closing a question is to prevent answers. A distant effect is to put the question (and answers) on the path to deletion. If the goal is pruning (and that's an important goal), what benefit does closing have except to prevent answers from wasting their time answering questions destined (often months later) for deletion? (I think we both agree that duplicate closures are an all around benefit.)
    – Jon Ericson Staff
    Commented Feb 19, 2015 at 2:16
  • 1
    @JonEricson I was thinking of the anonymous feedback as "upvotes from unregistered users"; another set of indications that the answers were useful. I'm not sure that the majority of answerers care if their answers are eventually deleted, especially since they often get to keep the rep.
    – jscs
    Commented Feb 19, 2015 at 2:28
  • 3
    "It's one that the voters think is going to clog up search results with dead ends." But it wouldn't, even it would only be downvoted, if only we could teach google that a low score means that it is basically useless. Probably google found that out by tracking user behavior already. Commented Feb 19, 2015 at 10:07
  • 3
    @JonEricson "I think we both agree that duplicate closures are an all around benefit" - I disagree. I've stopped counting the amount of "Help, I have a NPE" Java questions which I close as duplicates of the canonical "What's a NPE and how to fix it". I don't think any of these duplicates are useful; and even if a few are potentially useful, the huge amount of them certainly isn't.
    – l4mpi
    Commented Feb 19, 2015 at 14:08
  • 5
    If the point of closing questions is not to stop people from getting answers but to make it easier for future answer-seekers to find the right answer then why not simply hide a closed question from Google and leave it possible for the person with the problem to get a solution.
    – cja
    Commented Feb 19, 2015 at 14:29
  • 6
    If the argument against closing questions is "but they'll only be deleted months later!" then the answer isn't "let's remove closing" but rather "let's delete them sooner". And I most certainly don't agree duplicates are a good thing; I'd rather see them gone and instead have a carefully maintained reference question. Commented Feb 19, 2015 at 15:21
  • 4
    @l4mpi what you observe with this flood of useless NPE-dupes may be less of an issue if SE team was less reluctant to implement their own ideas on more efficient auto-delete "...we dispense with the logic that preserves answers with 1 vote or an accept mark that will stay deletion at 9 days. Downvoted duplicates are also added to the mix"
    – gnat
    Commented Feb 19, 2015 at 15:39
  • 3
    @l4mpi you might wish to look at What should the system be deleting automatically that it already isn't? and the section on "Low vote meh duplicates". Note that when I wrote that (~8 months ago) there were 410 dups of NPE... now there are 1245.
    – user289086
    Commented Feb 19, 2015 at 16:23
  • 8
    "A closed question is one that isn't going to produce a valuable document for the future" On the contrary, far too many closed questions are extremely useful ones, which powerful users merely have a personal dislike for. Closing desperately needs to be replaced with a user-configurable view filtering mechanism, so that the habitually picky can stop being bothered while leaving the site useful for the rest of us who like to contribute solutions to important problems and research what others have already found. Commented Feb 20, 2015 at 21:09
  • 5
    @ChrisStratton Total nonsense. Do you really think that most closed questions are closed only because of accidental personal preferences and are otherwise useful?
    – BartoszKP
    Commented Feb 21, 2015 at 19:35
  • 4
    Not the majority of them, but far, far too many critically important instances which are closed due to shockingly ignorant biases. Ignorant people throwing their weight around that way is the #1 impediment to the utility of the site. People merely need to learn how to filter their own reading for topics of interest, and we'll all be able to get along. Trying to control what others see is a losing proposition on the Internet. Commented Feb 21, 2015 at 20:41
  • 2
    "A closed question is one that isn't going to produce a valuable document for the future." - if votes are any indication on value, then your statement is often unrealistic. There are thousands of closed and delete questions with 50+ upvotes each, let alone the upvotes for the answers. Commented Feb 22, 2015 at 2:12
  • 3
    @DanDascalescu and? If I asked something that everyone has an opinion about, yet is not actually on topic, don't you think I would achieve the same results?
    – Braiam
    Commented Feb 22, 2015 at 2:41
  • 2
    @Braiam Opinions can be useful. I would love to see more questions comparing different approaches, methodologies and technologies. Technically there is no "right" answer; but in practice there is: the one that gives a breakdown of the pros and cons. Immensely useful - but unacceptable to the soup nazis.
    – Stumbler
    Commented Feb 22, 2015 at 11:27
  • 2
    @Duncan then where would you leave things that are not opinions? I assure you that opinionated posts would effectivelly dilute any other kind of content on the site, since everyone wants/gives opinions about anything and not necesarily has the knownledge to back it up.
    – Braiam
    Commented Feb 22, 2015 at 13:00
65

You don't need to wonder what it would look like: it would be like the original beta, with fluff proliferating and rising to the top.

Only a thousand time worse because the user base is less selected.

The overall quality of the site would be profoundly compromised, and I doubt real experts would bother to hang around in any numbers.

8
  • 4
    I think there's some revisionist history in this answer. Sure there was a lot of "fluff" in the beta. But we really didn't know what we were doing. I asked a lot of fluff questions back then and the usual response I got was, "hey, why isn't this CW?" My questions weren't closed until years later. And at that point, I'd learned for myself that GTKY questions don't produce the results I was hoping for. Closing had roughly zero impact on my behaviour. As far as I can tell, there's no data showing that closing is the primary reason for quality questions.
    – Jon Ericson Staff
    Commented Feb 19, 2015 at 20:13
  • 6
    @JonEricson The mere introduction of close votes didn't change the cruft problem because we then needed to establish a consensus about what constituted a good question to get things to stay closed. And we may disagree about what happened in the early days: I always was one of the grumpy fun haters and that may color the way I recall things. That said, I've been keen on this for early on. See meta.stackexchange.com/a/51710/2509, meta.stackexchange.com/a/55973/2509, meta.stackexchange.com/a/128465/2509 (point 2). Commented Feb 19, 2015 at 20:25
  • 8
    "... I doubt real experts would bother to hang around in any numbers" This! For real-world proof, see any group on LinkedIn.
    – SiKing
    Commented Feb 20, 2015 at 19:06
  • 1
    Looking at Quora, a Q&A site with no concept of "closing", I think that on the contrary, more quality answerers will be attracted, including those who provide valuable insights that here on SO would be classified as "subjective". I'm very active in the Meteor tag, but I rarely see on SO any of the core Meteor developers. Instead, they hang out on Quora. Have a look at the Meteor tag. Commented Feb 22, 2015 at 2:16
  • 3
    @DanDascalescu different people go different places. I find trying to participate in more than about 2 communities distracting. There are others on Twitter and even some on tumblr. Should we go pro lame that they are better Q&A platforms than SO? Or would it be more appropriate to say that different questions are best addressed on different platforms, and different people enjoy answering different questions? Is the internet too small that you can't have both Quora and SO? And why must a site cater to all types of questions?
    – user289086
    Commented Feb 22, 2015 at 22:47
  • 2
    @MichaelT: SO was great back in the day when you could ask what are the best books for programmers and not have the question closed in your face and then deleted. If the reason users are leaving your site is because they've been made to feel like shit then yes, that's wrong I would say. Commented Feb 23, 2015 at 9:19
  • 2
    @DanDascalescu back in the day, SO wasn't getting 7k questions/day. It was still trying to find its focus and accepting questions of all types. Since then, it has found its focus and decided to get one type of question right - the strict Q&A focus. To further this focus, it has added and tweaked tools (comments (as second class citizens), improved community moderation, vote/reputation ratios, close reason tuning, Roomba scripts) to enhance this one type of question. It doesn't and can't do all types of questions well and maintain the quality it seeks to present.
    – user289086
    Commented Feb 23, 2015 at 13:58
  • 1
    ... and that is why there are other sites that are filling in the niches of other types of questions. Reddit, quora, slant.co - these sites each take a different part of the "ask questions" pie and focus on their part. One can try to get other types of questions to work on the fringes of the scope, but it may not cater to that type of question as well (note: quora does delete non-questions, unclear questions and does appear to have some sort of duplicate closure/redirect).
    – user289086
    Commented Feb 23, 2015 at 14:05
35

Thought experiments are good, but when they are about fundamental features then they need to be carefully executed.

This thought experiment is reminiscent of ideologically driven social arguments that use artificial constructs to define social groups or classes. You can change the definitions of those constructs in order to "eliminate" undesirable groups or outcomes. Applying this analogy to the SO paradigm I think it is better to educate and encourage users to improve their quality (so they earn their way above the poverty line) rather than lowering the quality bar to better accommodate unproductive users (they would now be considered "middle class" because we've lowered the poverty line).

The linked blog post talks a lot about elitism and the "soup nazi". The author has a point, to a certain degree. However we shouldn't lose sight of another very important point: programming and software engineering is a craft that requires a high amount of skill to be considered competent. The same could be said about doctors, but would doctors ever lower their standards to allow less proficient people into the fold? No they wouldn't, because people would die if the professionalism and elitism was reduced. While programming does have much lower barriers to entry, we are not doing the world any favors by lowering the standards - the lack of standards (quality) amongst programming oriented sites was one of the very reasons why Stack Overflow was founded.

Close votes are goneburger - what would happen?

No system is perfect, but I'm not convinced that SO has a problem with close votes. Sure some questions get closed fast, but there are checks and balances.

Eliminating close votes will mean you have to either:

  • accept rubbish, and lots of it, and then pound it with down votes instead of close votes
  • tune the delete triggers so that they work with different metrics, but then all you've done is changed the look of the goal posts and maybe moved them a little bit

If close votes are eliminated then some new users are still going to get the sharp end of the stick, but it will be because their question has been down voted to oblivion rather then being closed by the time it got to -5 or thereabouts. If questions are pounded with down votes instead of being closed, have you really achieved anything?

12
  • 1
    This answer is also a bit of a thought stream, it was rewritten a few times before it was posted. The TL;DR version is that I think we are better off concentrating on how the close votes are applied (i.e. the criteria for closing) rather than getting rid of close votes.
    – slugster
    Commented Feb 19, 2015 at 2:35
  • I agree with most what you said but I don't think it has much to do with the neccessity of close votes. Surely there are other ways/system thinkable and doable and (just my guess) they would perform similarly well, maybe even better. So yes, I agree with the call for standards but the means to achieve them are somewhat arbitrary and can probably always be changed. If you don't have close votes you can easily have something else: likes, downvotes, personal reccomendations, whatsorever... Commented Feb 19, 2015 at 10:03
  • 1
    "If questions are pounded with down votes instead of being closed, have you really achieved anything?" - It may just be the easier solution. And also one could maybe just display that a question has a score below a certain threshold, not the exact score, if it has a score below this threshold. No need in differentiating between -27 and -28. Just filter out all questions below -5 and you should never see the bad questions again. Commented Feb 19, 2015 at 10:06
  • 7
    @Trilarion that assumes that people downvote the bad questions. "Who would be the most efficacious Patron Saint of Programmers? had a positive score when it was deleted on P.SE long ago. Digging in the archives of Programmers.SE early days one can find many such examples (what is your favorite comic?) that were upvoted in their day. If down votes are the only way to get rid of that content, then you will have trouble getting rid of Good furniture for programmers which was at +23 when it was deleted.
    – user289086
    Commented Feb 19, 2015 at 14:34
  • @MichaelT I would guess that these examples are extreme outliers and most of the closed questions indeed have negative score when closed, so downvoting alone probably would be enough. Basically the mentioned examples are relicts of the old days. They can be closed, I would not care. Commented Feb 19, 2015 at 15:47
  • 4
    @Trilarion when you have an environment where anything goes (and nothing gets closed - preventing new answers), the popular watercooler questions proliferate. This is what happened on Programmers.SE. I can easilly find dozens more (or ask Yannis or ChrisF for examples) of old, popular, highly upvoted questions. Down votes are not enough to do the moderation and closing questions is critical to getting people to fix their questions so they can be reopened, preventing pile on answers on a bikeshead question, and keeping the site's scope in focus.
    – user289086
    Commented Feb 19, 2015 at 16:02
  • @MichaelT These are mostly problems of the past. Do you really imagine a lot of popular ontopic questions to appear now that should be closed? I'm not against closing offtopic questions, but I don't think there will be many new, popular, highly upvoted, ontopic questions that should be closed. This is just not happening that often. Most of the closed questions now are by far neither popular nor upvoted. Commented Feb 19, 2015 at 16:50
  • 2
    @Trilarion It is not at all uncommon for a question to get popular, hit the hot questions list and get lots of poor answers before we are able to close it. Coming from P.SE, the only thing that saves us in that manner is that many fewer people have reopen votes for popular questions (though looking in 10k tools, they show up there from time to time). Whenever a blog post like this shows up, there is often much moaning about closed popular questions (go poke on HN or Reddit). Popular questions just take the right mix of social media to get votes - it happens more often than you likely realize.
    – user289086
    Commented Feb 19, 2015 at 16:55
  • @MichaelT Okay, I have to admit I don't know or can estimate the impact of social dynamics to scoring. It can be that this is a true benefit of closing. But this also just means that the "upper class" (>3k rep) just thinks differently from the whole population (who can upvote), otherwise closing of questions and popularity should be completely inversely proportional. So could closing be seen as the rule of the upper 30000? :) Commented Feb 19, 2015 at 16:59
  • 3
    @Trilarion and indeed, that is a good thing. I strongly suggest reading A group is its own worst enemy (tangent: Shirky is on the board of directors for Stack Exchange - his writings have certainly influenced thought of the early design). The section to pay attention to here is the things you have to accept - item number two: The second thing you have to accept: Members are different than users... though read the entire thing. Its all good. And then go read meta.stackoverflow.com/a/252077/289086
    – user289086
    Commented Feb 19, 2015 at 17:05
  • @MichaelT Thanks for the links. Just want to be as clear as possible because I think discussions get better when naming things what they are. I would like to add Kicking off the Summer of Love to the list of must read articles. Commented Feb 19, 2015 at 17:13
  • 2
    @MichaelT, and that meta question you reference had two occurrences. 1. It produced very valuable conversation, and 2. ended up being closed anyway. And curious, even though it's regarded as not good enough to remain open, it's still left in place for "historical significance"? What we have here is a website of just people with all the foibles that all people have. Despite the SO charter, just as with usenet, social media, and any forum, SO is not immune to ego-driven and knee-jerk reactions, and piling-on effects.
    – user4229245
    Commented Feb 21, 2015 at 18:24
31

I have a few thoughts about this, which I've organized under roughly-independent headers...

The literal answer

First, it's worth noting that Stack Overflow had the notion of closed questions before it had the concept of close votes. So it's not necessarily true that "no close votes" would equal "no closed questions".

Given the rest of your question, I'm pretty sure this isn't what you were thinking about when you wrote this... But I think it's important to remember the history of these features: it's all too easy to get stuck in "cycles" when designing systems like this, trying the same things over and over again, reacting to each new problem by returning to a previous solution that was discarded due to other problems. More on this later...

I recently posted statistics on how questions are being closed now. Let's contrast that with stats on how questions were being closed in the 3 months before close-voting was implemented (2008-09-30 to 2008-12-30):

Total questions closed 
---------------------- 
2189                   

(1 row(s) returned)

Total questions asked PctClosed 
--------------------- --------- 
47156                 4.64 %    

(1 row(s) returned)

Name                Closed     Closed->Edited Closed->Reopened Cl->Ed->Re 
------------------- ---------- -------------- ---------------- ---------- 
exact duplicate            707        304              6                3 
not a real question        310         35              1                1 
not constructive            86         16              1                1 
off topic                 1042        126             28                8 
too localized               44          6              1                0 

(5 row(s) returned)

% of Closed Name                Closed->Edited Closed->Reopened Cl->Ed->Re 
----------- ------------------- -------------- ---------------- ---------- 
 32.3%      exact duplicate      43.0%           0.8%             1.0%     
 14.2%      not a real question  11.3%           0.3%             2.9%     
  3.9%      not constructive     18.6%           1.2%             6.3%     
 47.6%      off topic            12.1%           2.7%             6.3%     
  2.0%      too localized        13.6%           2.3%             0.0%     

(5 row(s) returned)

...and the three months following it (2008-12-31 to 2009-03-31):

Total questions closed 
---------------------- 
2444                   

(1 row(s) returned)

Total questions asked PctClosed 
--------------------- --------- 
62703                 3.90 %    

(1 row(s) returned)

Name                Closed     Closed->Edited Closed->Reopened Cl->Ed->Re 
------------------- ---------- -------------- ---------------- ---------- 
exact duplicate            586        212             18                8 
not a real question        432         81              2                1 
not constructive           176         35              2                0 
off topic                 1211        193             20                2 
too localized               39          6              0                0 

(5 row(s) returned)

% of Closed Name                Closed->Edited Closed->Reopened Cl->Ed->Re 
----------- ------------------- -------------- ---------------- ---------- 
 24.0%      exact duplicate      36.2%           3.1%             3.8%     
 17.7%      not a real question  18.8%           0.5%             1.2%     
  7.2%      not constructive     19.9%           1.1%             0.0%     
 49.5%      off topic            15.9%           1.7%             1.0%     
  1.6%      too localized        15.4%           0.0%             0.0%     

(5 row(s) returned)

Probably the most striking change there is the reduction in the number of duplicates closed (and duplicates as a % of closed questions). Even though the system didn't even require finding a link for duplicates during this period, dup-closing was already harder than closing for any other reason. As a % of questions closed, duplicates continued to drop for some time after this, finally experiencing a resurgence with the introduction of Thor's Hammer.

Finally, please go read Mike Stone's blog post about closing (also referenced in the "love-hate" blog post you cite). This post - and an ensuing email conversation between Mike & Jeff - are credited with motivating the creation of the vote-to-close system we all... uh, the system that exists today. It's worth reading just for the similarities between the complaints raised against Stack Overflow now and those tossed around 6 years ago...

Satan's Solicitor

Oh, tell me I may sponge away the writing on this stone!

Ok, now let's get back to the question you actually meant to ask: what would happen if closing - as a concept - didn't exist?

I'm not going to try to estimate actual numbers for anything here; this would be a massive change, affecting too many variables for me to even enumerate with confidence much less account for. I've provided numbers elsewhere for how many questions get closed today, as well as how many get deleted - those are useful for getting a rough idea of the scale involved here.

For starters, we could strip a lot of complexity from the system:

  • A huge number of votes, flags, scheduled tasks and associated plumbing could be discarded. No more close review, no more "recommend close" flags, Very Low Quality flags for questions could be handled the same way as they are for answers (n high-rep delete votes or n×2 low-rep reviewer delete recommendations).

  • No more tedious discussions about what close reasons we need, how to best convey such information to new users, what sorts of documentation we should be providing to help explain closures or motivate editing on the part of the askers.

  • No more reopening. No more reopen queue, no more scheduled tasks looking for edited or popular posts to reopen, no more aging away reopen votes.

  • No more public moderation; no more revenge-voting everyone who closed your question. The names of downvoters are not included in public data, and deleted questions (along with information on who deleted them) are excluded from both Google and SEDE.

In short, a massive amount of technical and bureaucratic overhead would be removed, blasting away a whopping 40% of the steps outlined in The Question Lifecycle.

And those are just the direct effects. The secondary effects are even better:

  • No comment discussions about question closure. Sure, there'd be discussions about question downvoting, but those already exist - and given that downvotes are anonymous, there's little motivation to come back and revisit questions you've downvoted to avoid having your name attached to something that doesn't warrant the vote. Unless you're coming back to vote to delete... But comments aren't allowed on deleted posts.

  • Automatic deletion could be greatly simplified; without closed questions, the notion that there should be a "grace period" for editing in order to allow questions to be salvaged becomes trivial: if there's no activity on a poorly-received question for some amount of time, just remove it.

  • Deletion in general would be greatly simplified: current restrictions are all based around a combination of question+answer score and closed date; without a closed date, deleting questions becomes as simple as deleting answers: find a worthless post -> make sure it's downvoted -> vote to delete.

  • No closed questions in search results. That "love-hate" blog post made some hay on the closure of a question back in 2011, but we get relatively few complaints about deleted questions - no one finds them! There are a lot more deleted answers than there are closed questions, but far fewer complaints - again, because no one sees them. If the only choice was to leave questions fully accessible or delete them, we'd eliminate a massive source of complaints in exchange for a few more answers to poor questions.

  • With no way to close duplicates, the conflict between folks who like to identify and close common questions and those who prefer to identify and answer common questions goes away: both sides can just post answers, one with links and one with copypasta.

  • With no way for the system to even identify duplicates, current restrictions on deleting them (by automated systems) or deleting dup-targets (via votes) become obsolete as well. Vast swaths of ugly questions, no longer needed to point the way toward an answer, can be removed.

All in all, a close-free Stack Overflow would be a much, much simpler, significantly less boisterous system. There would be a few downsides of course:

  • There'd be no path short of moderator intervention for getting rid of popular distractions.

  • With downvotes now the primary means of moderating questions, the number of "moderators" would increase by more than 10x. With downvotes now the only means of marking problematic questions, their use - along with flags - would likely see a marked increase, along with "lord of the flies" comparisons such as our dear "love-hate" author's. Note that Stack Overflow current sees more close votes + flags on questions than it does downvotes on questions and answers combined.

  • We - that is, the community here on meta, the elected moderators, and the folks like you and me working for Stack Exchange - would lose a powerful tool for influencing moderation: predefined close reasons.

  • Along the same lines, there'd no doubt be considerably more pressure to require downvoters to specify a reason for downvoting, along with calls for meta-moderation features that could invalidate downvotes that were seen as groundless by some segment of the population.

A very different system indeed...

The answer that isn't an answer

Both of the previous answers assume that there's a limited scope to this exercise, that we're cutting out one section of the question pipeline and sewing the ends of what remains back together like some sort of weight-loss surgery. In other words, I'm assuming we would not want to completely strip the concept of community moderation from Stack Overflow, Lord of the Flies comparisons notwithstanding.

But of course, that's an assumption that only makes sense here. Out in the wild and woolly world of weblogs, plenty of pundits propose exactly that - and have been since the start. When I read,

So what if some of the questions are not as good as others, and don’t perfectly fit into Stack Overflow’s long-term archive? Let people ask them, let these questions get answered. And if they’re mediocre questions, let them fade into mediocrity after a few weeks.

...I'm reminded of a question Michael Pryor pointedly posed nearly six years ago, "Why do people close questions on Stackoverflow?"

Closing a question is effectively saying "You can't talk about that here". Why do you care? What are you attempting to accomplish by closing the question?

My reply then - and now - was that you can't have a community without something shared to hold it together:

We're not a community in the traditional sense; we don't hold pancake breakfasts or sponsor 10K run/walk events, and the number of people who bother to meet off-line or even talk outside of SO is small and heavily fractured. This community is, at its core, one of shared purpose[...]

It doesn't matter if the underlying system supports multiple groups of users; without that core group and a common goal, you have nothing.

This - the notion of a shared goal - is the source of an underlying conflict that can never be resolved conclusively without fundamentally changing what Stack Overflow is. You could easily build a system that allows any question on any topic, and even things that aren't really questions about topics that aren't really defined, and others have built systems that do exactly this...

But that isn't what we've built here. I, for one, enjoy not seeing my work lumped in with discussions of pop stars or GTKY threads about food. If you like that sort of thing, great! There are places that will cater to you - we don't need to be everything to everyone. Heinz made 57 varieties of pickles yet he did not please all of the pickle lovers...

Love-hate gets so many things right that I'm reluctant to keep picking on individual quotes... But the mistake I see over and over again is the same mistake I've been seeing here for 6 years, repeated ad nauseum by countless members new and old: focusing on flaws in portions of the system without seeing how those pieces fit into the whole.

I spent the past weekend fixing the ignition system in a 30-year-old riding lawnmower. The previous owner hit on what I suppose seemed a very clever idea: replace the broken ignition system with a push-button start. He accomplished this by wiring the battery to a button and the button directly to the solenoid, bypassing everything else. And it worked! At least, it ran, and the lights turned on and off. Of course, the lights just drained the battery, which the alternator no longer charged, and the only way to turn the thing off was to stall it... But hey, it cut out loads of complexity and got rid of those safety switches everyone loves to complain about!

And this is the problem you run into when you start "fixing" things by cutting out parts without first bothering to understand the problems they were intended to solve: you inevitably find yourself - or whoever comes after you - having to solve those problems again:

  • Discouraging comments: make answering the path of least resistance, avoiding the "need to read 30 pages of forum replies to get a complete answer" problem.

  • Rewarding clear, reusable questions: allow answers to be reused, meaning knowledge isn't trapped forever in a thread no one can find.

  • Closing unclear, unwanted or apparent duplicate questions: nominate and vet deletions in public, where they can be reviewed by the entire community, discussed and even avoided completely.

Are there other ways to solve these problems? Absolutely! Could we try them? Should we? Yes! Yes! And we have, and we'll continue to, aided and motivated by discussions such as this one. But we'll do so fully aware, not just of the problems that remain unsolved, or the side-effects of our current solutions, but also of the original issues that these systems were created to solve in the first place. When I finished the wiring on that lawnmower, it was still a push-button start - but now it charges the battery, and shuts off if you should fall from the seat with the blades running. Yes, it took a lot longer to analyze and implement than the original hack - but the result is something that can continue to run for years to come. Stack Overflow is now in its 7th year, and there's no reason why it can't be made to operate for much longer - if we take similar care of repairs.

In closing

At some point, this discussion stops being about the tools we use and starts being a question about the essential nature of what we're doing...

Is this still a collaborative effort, a great leap of faith predicated on trusting your fellow programmers? Is Stack Overflow you? Or are you just canon fodder, a digital sharecropper to be milked for all you're worth and then cast aside.

And for those of us here on meta, building this system... Who are we building it for? Do we still remember?

4
  • 7
    It's okay, the important thing is the ignition is fixed.
    – user3717023
    Commented Feb 24, 2015 at 17:00
  • Wow! It's lonely down here at the end of your answer :) I think that communication is everything here. Often times, if you read carefully, people aren't complaining about the system per se, but about how they felt, and what they perceived the system was saying about them. In that sense, simply changing 'Closed' to 'On hold' was a great leap forward. But if I recall correctly, even doing that change met strong resistance when it was originally proposed. (cont..)
    – Benjol
    Commented Mar 26, 2015 at 8:37
  • So who are we? Programmer-kings who don't suffer fools lightly? Just the 20%?
    – Benjol
    Commented Mar 26, 2015 at 9:13
  • Or have we interiorised Dunning-Kruger sufficiently to give others the benefit of the doubt?
    – Benjol
    Commented Mar 26, 2015 at 9:40
25

Let's look at a subset of this experiment, as in not entirely removing closure

This thought experiment seems like a knee-jerk reaction to me. A user with a decent amount of reputation felt disenfranchised because of closures and wrote a well articulated reason for being disenfranchised with closures. The result shouldn't be to consider completely removing them.

http://www.vitamin-ha.com/wp-content/uploads/2013/08/Kill-it-with-Fire-03.jpg

Don't burn the house in order to out the spider. Simply squash it where you see it, or perhaps get a vacuum, or perhaps just humanely place it outside if you are into that sort of thing. In this case, the spider is a certain classification of closures. In this regard I believe that grouping all closures together to do analysis is terribly misleading.

Certain closures help us maintain quality, and we need them

Certain types of closures are definitely required in order to maintain the quality standards that I believe every user here can appreciate. However, it is possible that certain closure types are counter productive. Possible. If there is something to examine I believe it should be that possibility.

The new closure reasons are mature now, and perhaps need to be examined

Just as question closure reasons were changed once before, perhaps they need to be examined a second time to see if the new implementation is working. There has been a significant amount of time for them to be present and I believe that this thought experiment suggestion highlights the need to re-examine the place of some of these closure reasons.

Perhaps the new off-topic reasons are being used to close questions which contain code but are not always answerable by everyone. As in, these questions can only be answered by an expert - of which we have many - but they are not getting a chance because others close the questions as a result of what they perceive as pitfalls to the code shown. Perhaps. It is still my opinion that the reasons seem to work well, but I would like to see the data shown by Jon here aggregated more to show the individual close types.

If the reasons fit, perhaps the closing mechanism itself is what needs change

In my opinion closures should be for questions which cannot be answered and as a result serve no purpose existing. If there is no way to answer a question with something that makes sense as an answer, then the question itself has severe issues.

While it may be easy to point out by inspection, I understand the drive to provide feedback to people who have had their questions closed. As a result, it may make more sense to provide a mechanism where users see feedback explaining that their question was closed because it could not be answered.

Thought-Experiement

I could not answer this question because

  • it is not related to solving a problem by using code nor about solving a problem presented by using code.
  • there was not enough code or process shown in order to debug the syntactical error, logical error, or depicted error.
  • it is asking me to do all the work.
  • opinions vary too widely on this topic.
  • it is asking for an off-site resource and would be little more than a link only answer.
  • it has already been answered [here]().

I know you are probably thinking, "how would that be any different than using the current close reasons? They even look similar!" (or probably just a stream of wtf) But this is an experiment after-all, hopefully a thought provoking one.

Let me explain

The goal is to have quality questions and answers here, but there is also the premise presented that answers are a commodity and losing answers implies a "cost". Perhaps using this structure would be able to retain some of that cost while still providing a mechanism to prevent questions from eliciting low quality answers.

Metrics for these reasons for being unable to answer could be used to push posts off of the front page as their likelihood of being answered drops. If a question reaches a certain point of not being answerable then it is placed on hold by a script which runs with the batch no less than 24 hours after the answer is placed. If the question has an accepted answer and is above -2, it is exempt from the script. If the question is below -2, has no upvoted answers, and has at least 5 users > 3000 reputation who could not answer it then it is placed on hold and the reasons that users could not answer the question are shown to the OP.

Questions which make it through the list would be high quality

The reasons for being unable to answer are also in order for a reason. Negating the reasons, questions which could be answered would be those that are about a problem whose solution involves code, shows enough of the code or process to debug, is not simply asking for implementation of a missing section, has at least some consensus on the approach to use for the solution, is not asking for a solution which would be a link, and has not yet been answered on Stack Overflow. I believe the questions which meet those criteria are in line with the current expectations of the site's status quo and would also elicit high quality content.

Experts could still answer

Further, if some users feel they are unable to answer the question, this does not prevent any experts from posting an answer. If an expert feels they understand the scope of the question and feel they can benefit the person asking, there is no barrier to them posting an answer. While the question may be in a range of quality, most experienced expert users here only post very high quality answers because they understand the value of an answer which stands the test of time.

tl;dr; Can we see an aggregate of the data set split up by close reason? Perhaps the close reasons can be examined since they have been around for a while now. Thought experiment: change closure to reasons why users are unable to answer and extend the initial window for posting answers to at least 1 day.

14
  • 5
    Still reading this over to try to get an idea of it. How do you define expert? Should we allow Jon and Eric to answer any question? Even "what is your favorite dilbert cartoon?" There are many high rep users who have (in the past) piled on to bike shed questions - is there any mechanism to prevent that (should it be prevented)? (coming from P.SE, I know we have page 1 users who will answer bike sheds if they don't get closed - I don't believe it is any different on SO). Would this be a 20k rep priv: ignore close and locked questions when answering?
    – user289086
    Commented Feb 19, 2015 at 22:07
  • @MichaelT - You bring up some valid points and I will try to respond to each one separately because of comment length. While the bold portion states expert, I clarify in the text "experienced expert users", with which I was trying to state that anyone who has managed to reach the top 1000 list is probably well versed in contributing high quality content and probably pretty knowledgeable in their relative subject. As with all generalizations, it has exceptions I am sure.
    – Travis J
    Commented Feb 19, 2015 at 22:15
  • @MichaelT - So, there are two sides of the coin for answering any question. First, the suggestion here would be to indicate you couldn't answer because ... well the dilbert one fails a few of those options... mainly that it isn't related at all to coding, but also that opinions vary (Jon and Eric probably do not share the same favorite dilbert cartoon, and you probably have a different favorite than the both of them)...
    – Travis J
    Commented Feb 19, 2015 at 22:21
  • ...On the other side, if users are willing to post an answer this would provide a path for them to answer it. Perhaps extra metrics could be used to lock down posts which turn into little more than popularity contests, but I feel that a post asking for your opinion of dilbert would be downvoted very quickly and as a result pushed into a place where it would not be found until the script took care of it. It is hard to tell whether or not closing the question should take answers into account.
    – Travis J
    Commented Feb 19, 2015 at 22:21
  • 1
    @MichaelT - The bike shed issue is legitimate though. I believe that users should probably close these as they are asking for opinions which vary widely, and probably ask for users to do all the work of building their version of a bike shed. However, this would leave the door open for an answer of which color, how many walls, a vaulted roof versus a metal one, etc. etc.
    – Travis J
    Commented Feb 19, 2015 at 22:25
  • Should these questions be prevented? I think they should be downvoted if they ask for opinion, I wouldn't answer for one of the reasons available. I think that the downvotes and if possible, the path for closure as a result of maybe >6 downvotes and >5 unable to answer votes from >3000 rep users, would strongly discourage this type of question. Does that answer your questions?
    – Travis J
    Commented Feb 19, 2015 at 22:26
  • 4
    It isn't hard to find old examples of opinion questions with lots of upvotes and these questions contain answers from users with tens or hundreds of thousands of rep. New examples can be found too. The problem is that people are not using down votes on such popular examples... and things like this are still open (note top answer). I do not believe anything short of closing is sufficient to prevent pile on answers.
    – user289086
    Commented Feb 19, 2015 at 22:40
  • 4
    ... yes, there are good answers in those questions, but there are also very poor ones too. The questions aren't awful but rather they miss the design goals of Stack Exchange's Q&A format and thus are often a poor fit for the site structure itself. There are other formats that would handle these questions better - just people are lazy and ask on Stack Overflow as the first course of action to get the community here to answer it (because it is a great resource) rather than going to sites where the structure may work better for that question.
    – user289086
    Commented Feb 19, 2015 at 22:43
  • 6
    I like your close reasons. They cover the main reasons why we would want to close questions, but don't encourage the closing of niche questions (which I feel is at the heart of the problem).
    – ChrisF Mod
    Commented Feb 19, 2015 at 22:44
  • 1
    I like your proposed close reasons as well as they're pretty straight forward and not as bloated as the current ones, you're just missing a "that's a typo/brainfart" one. But "Questions which make it through the list would be high quality" feels like sarcasm given that your proposed scheme would let a -2 question with an accepted or upvoted answer through. Assuming an answerer often upvotes the question, the most likely scenario means three people think the question is crap, one wants to answer it, and OP is happy to accept and/or upvote the answer. That's not an indication of quality at all.
    – l4mpi
    Commented Feb 20, 2015 at 9:43
  • I think you cannot get very specific simply by giving more options to the close voter. Any specific hand written comment will always be much more valuable (for the foreseeable future). Also "it is asking me to do all the work." is not really a valid close reason right now. If anything it is a reason to downvote (no re(search)). Commented Feb 20, 2015 at 10:59
  • 2
    I liked this post until you encouraged murdering tiny animals. You animal. Commented Feb 20, 2015 at 11:09
  • 1
    @LightnessRacesinOrbit - Added the option to not murder tiny animals.
    – Travis J
    Commented Feb 20, 2015 at 19:41
  • 1
    @TravisJ: Haha now I love you <3 Commented Feb 21, 2015 at 4:25
17

Why is this a thought experiment? This already happens.

If all questions that should be closed were actually closed then there would be no need for automatic deletion of unclosed questions. There are currently 723 thousand unclosed questions with a score of 0 or less and 0 answers, i.e. those that may eventually be deleted under the current criteria.

If we take question score as a proxy for the questions that will remain if close votes are removed that shouldn't there are currently 333k unclosed questions with a score less than 0. (Yes not all of these should be closed but not all questions with a higher score should be open - it's a good enough proxy for my point).

The site is already failing to keep up with closing all questions that might need it - removing closing won't make any difference to the hundreds of thousands of questions that slip through the system.

Though good attempts are made to optimise for pearls more and more sand is creeping through the cracks in the clam's shell. By closing some, but not all, we're annoying some of both the sand-castle makers and the pearl divers.

I was going to suggest voting, as George has done, as the method of keeping things in check. Though I don't necessarily agree with the following potential methods of solving the issue, we do need a way of ensuring that someone searching for a problem can easily find the answer:

  • Idea 1 - remove from Google's index all questions with a score of 0 or less
  • Idea 2 - delete all questions with a score of 0 or less on 7 + score days after creation. If you've got your answer but aren't serving the greater good then everyone might (possibly!) be happy?

Fairly drastic, but I suspect that optimising for pearls rather than sand programmatically rather than socially may eventually become the only option. If this is done, why not annoy less people along the way?

4
  • Idea 2 is going to cause a carnage in the less popular tags. Many questions in these tags take a while before getting an upvote.
    – Louis
    Commented Feb 19, 2015 at 18:00
  • As I said I don't necessarily agree with them @Louis, my point is that a technical rather than social solution seems to be called for. Also, if you're in the less popular tags get voting!
    – Ben
    Commented Feb 19, 2015 at 18:09
  • 2
    @Louis: It's not just less popular tags. A little under half of currently undeleted questions have a score of 0. The distribution suggests -1 would be a better place to start. There would have to be a separate solution to the "programmer cartoon" problem.
    – Jon Ericson Staff
    Commented Feb 19, 2015 at 18:09
  • Yeah, I'd argue that a significant proportion are not generic, reproducible solutions to problems though @Jon. If something has low views and 0 votes then it's not being useful to people. The trick would be judging that (if it's even possible)
    – Ben
    Commented Feb 19, 2015 at 18:18
13

on average the answers on [late] closed questions seem better than on open questions

How did you measure that? Implicitly, by votes, I gather, which isn't quite right. Voting patterns correlate to question closure in ways that don't necessarily indicate useful answers. I have a hunch that a lot of late-closed questions are somewhat subjective questions, which Stack Overflow has gradually become less tolerant to. There are also popular duplicates, which get answered quickly and then closed much later when someone does a cleanup in a particular tag.


Regarding closure, I think it's a fundamental aspect of Stack Exchange. A site dedicated to a topic but without closure reminds me of Usenet. Once Usenet became popular¹, it became more and more common for a group to be overwhelmed with off-topic discussion. On Usenet, the only defense against off-topic content is a personal kill file. On Stack Exchange without closure, downvotes would allow sharing the load of sorting the wheat from the chaff. But voting is inferior, because it isn't absolute. When a question is closed, it's closed; a score-based killfile would have to use different thresholds for different combinations of tags. Thus removing closure would be a disservice to readers.

Removing closure would also be a disservice to answerers, and thence indirectly to readers. If answerers waste their time answering off-topic and unclear questions where nobody would find them, that's fewer answers that would be on clear, findable questions. Answering a question that's unclear is a waste of time because future searchers will glance at the question, fail to understand it and skip without reading the answer that might have helped them. Answering a question that's unclear or too broad is a waste of time because future searchers will glance at the question, not see how it relates to their problem and skip. Answering an off-topic question is a waste of time because future searchers won't be looking here for an answer.

Answers on off-topic questions are also dangerous because the votes are less likely to reflect expertise in the topic.

If an answer is worth keeping, edit the question to make it suitable and reopen it. Closing the question makes it clear that the question is unsuitable in its present state; reopening makes it clear that it is. The score cannot give such a clear indication.

Removing closure would also be a disservice to askers. Sure, some askers like to whine at the slightest perceived offense, but please don't forget that some askers are here to get answers. I could care less about the whiners and I want the ones who genuinely want answers to get them; I would not participate on Stack Overflow if I didn't share this sentiment with the SO community². For askers who want answers, leaving a question open when we know they're unlikely to get useful answers gives them false hopes, and fails to convey the message that they could and should do something to improve it. (Yes, people don't read the guidance, but I have little sympathy for people who don't read the guidance, and anyway you just can't succeed with these people. There is probably still room to improve this guidance though, as closure can be a bit daunting when you meet it for the first time.)

¹ Sep. 1993
² I feel that this sentiment is less and less shared; more on this later.


Regarding the blog post you cite, it makes some good points, some bad. In particular, it correctly identifies two goals of Stack Overflow that do not always align: the short-term goal of answering someone's question, and the long-term goal of being a searchable repository of answers. Where it goes wrong is that it presents closure as used today on Stack Overflow as being strongly biased towards the long-term goal at the detriment of the short-term goal.

Closure matters for both goals. It helps people get answers now by not making answerers waste time on questions not worth answering, and informing askers when they need to improve their question. It helps people who search answers later by indicating which questions are likely to have worthwhile answers.

Historically, there was a period³ during which more closures were pushed to get rid of old questions which no longer met current standards. Calling this “harassment” of askers and picturing closers as “nazis” is patently ridiculous — we're talking about old questions, which are not a priority for the asker any longer, even if it's still around.

Of late, closure has evolved on Stack Overflow. This is not a new thing but I feel that it has recently become the dominant trend. There is indeed a raise in the “no soup for you” aspect of closure. But this is not at all about long-term quality! On the contrary, it's all about askers jumping through the hoops that are expected of them. Guidance from the top emphasizes that what matters is that questions be answerable and that it's the results that matter and not the goal. And yet the issue comes up again and again of preventing questions not because they wouldn't produce useful answers, but because they do not show effort.

Closure is more and more working against the long-term goal, and with dubious effectiveness on the short-term goal. (I can't find the link now, but it's sometimes reached ridiculous height, where questions intended to be a canonical question on a frequently recurring topic are closed because the question focuses on the problem and does not demonstrate effort to solve it.)

Yet getting rid of closure would not solve this problem. It is a social problem, not a technical problem: motivations for closure have changed but the tools haven't changed. Getting rid of closure would help in that useful but non-soup-worthy questions could still be answered, but it wouldn't solve the problem that these questions would be as heavily downvoted as they are today if not more, hence filtered off and so would rarely reach potential answerers.

³ I think roughly 2011–2013, but I didn't check the dates while writing this post.

5
  • Are you thinking of How can I improve my self answered question about a task involving the use of regex?
    – jscs
    Commented Feb 20, 2015 at 2:05
  • 2
    Stack Overflow different than Usenet for reasons other than just closed questions. As a reader (most of whom find questions via search), what matters is not whether the question is closed (or even remotely ontopic), but whether the answer is correct and actionable. The data I presented in the question suggests that questions are closed after the majority of answers are in. Even after the "on hold" close changes, most closed questions are never improved. What I'm questioning is not the goals of closing, but whether closing achieves those goals.
    – Jon Ericson Staff
    Commented Feb 20, 2015 at 6:46
  • 8
    @JonEricson “Questions are closed after the majority of answers are in”: given how rare reopening is, how could it be otherwise? I really don't see how the data you presents can lead you to conclusions about the effectiveness of closing. Not that I'm doing better on the data front. To get data, we'd need to measure very hard things like user engagement and retention, and usefulness of posts (which is only very very loosely correlated to voting and views). Commented Feb 20, 2015 at 8:57
  • 1
    @Gilles: Look at the first graph. These are all questions that are eventually closed, but there are so many answers in the first five minutes that closed questions get (on average) 75% of the answers that all questions do. Would you use a killfile system that showed you 75% of a user's posts? (This is skewed by the dual-use of closures. When I throw out the outliers, the rate will be lower.) The closing system promises certain things that I do not believe it delivers.
    – Jon Ericson Staff
    Commented Feb 20, 2015 at 16:08
  • 4
    @JonEricson: Perhaps it doesn't quite deliver because deletion is too rare, and with any answers just about impossible? Commented Feb 20, 2015 at 20:28
11

In this alternate reality, what if the way to determine quality was solely based on voting? Voting is already free on questions; If we surmised that questions that were below 0 were not useful and the system automatically deleted those after 30 days (even if they have answers), then it's likely the effect would either be:

  1. People not answering bad questions because they'll lose their reputation anyway
  2. People upvoting more questions, even bad ones, because they don't want to lose their own reputation.

This is a stream of consciousness answer, so I haven't yet pulled any alternative data to extrapolate the actual result of such a policy, but I feel like the assumptions would be something like the following:

  • Any 'close vote' in today's terms would turned into a downvote.
  • Any user with an answer to the question would (at least half the time) vote up the question
  • Any question with a negative cumulative score after 30 days would be deleted (regardless of the answers)

Some of this data can't be figured out by the public dataset alone; but there's probably enough there for a WAG to see how it would affect the numbers.

11
  • 21
    Because popular questions aren't necessarily indicative of the quality of the question. Everyone who wants a book list would have one at +100 rep and a new answer every few days. Questions that don't work in the Q&A format may not be bad questions, but, well, you saw what P.SE was before it got changed around. What is your favorite cartoon would get upvotes and answers as easily today as it did years ago. That doesn't mean it is a quality question.
    – user289086
    Commented Feb 19, 2015 at 1:04
  • 3
    I have two immediate thoughts. Without closures and only votes, I wonder if SO would look a lot like the new tab on a programming subreddit after a while. Secondly, I wonder if this would make some communities utilize downvotes more. From my perspective, Math.SE is a very downvote-light community, and it would perhaps benefit from more upvoting and more downvoting. Commented Feb 19, 2015 at 1:23
  • @mixedmath I guess that begs the question: What do new users hate more, closure? or downvotes? I'm going to guess it's closure, but I say that without any evidence.
    – Mysticial
    Commented Feb 19, 2015 at 1:31
  • 6
    @Mysticial downvotes. They don't fear closure because by the time their question is closed they've already rage quit because of the downvotes.
    – user2629998
    Commented Feb 19, 2015 at 1:45
  • But a downvote only reduces rep by -2 while and upvote is +10. So just modify the ratio even more and the bad consequences mentioned in this answer do not have to come. Commented Feb 19, 2015 at 9:56
  • So this would allow a single user to search for questions with is:question score:0 created:2015-01-19..2015-01-19 then unilaterally delete them complete with answers? Commented Feb 19, 2015 at 12:55
  • @MartinSmith No, negatively scored questions.
    – George Stocker Mod
    Commented Feb 19, 2015 at 13:06
  • 6
    It is within one person's power to convert a zero scored question into a negatively scored one. Commented Feb 19, 2015 at 13:07
  • 4
    @MartinSmith It's also within one person's power to convert a negatively scored question into a zero scored question.
    – George Stocker Mod
    Commented Feb 19, 2015 at 14:20
  • 3
    Sure - if you happen to be in the habit of monitoring 30 day old questions just in case they happen to be downvoted. You'd just end up creating a system where questions can be deleted with little consensus and no accountability - as votes are private. Commented Feb 19, 2015 at 14:25
  • @MartinSmith Over the years we've expanded queues for closing; I would imagine there'd be a queue for these sorts of questions as well. Again: I'm not looking to get into an extended discussion about this approach in my answer. I'm just presenting an alternate reality. It's not meant to be a complete reality; just a picture. If you'd like to debate the merits of this approach, feel free to raise a meta question about it, or propose an alternate reality.
    – George Stocker Mod
    Commented Feb 19, 2015 at 14:53
9

I would imagine that without closing, it would be harder if possible at all to establish and grow specialised sites like CR, CG, SR...

If you check help center articles at these sites, you may notice that their norms on asking and answering differ a lot from those at Stack Overflow (and also differ a lot between each other). You will probably notice that questions that fit their norms would likely be closed at Stack Overflow (and, conversely, SO kind questions would likely get closed at these specialised sites). If you dig deeper, you will also find that these sites have fairly active, strong and mature communities.

If there were no closes, questions of this kind would likely stay at Stack Overflow, first of all because there are solid communities supporting these.

In particular, establishing separate sites would be much harder than now, because of tension that inevitably would be there for it would be unclear why adding and maintaining a dedicated site when questions are okay at Stack Overflow (and they would be okay, again, simply because there is a solid and active community behind these).

Would that be better than what we have now? Let's see...


Imagine someone willing to ask a good ("good" in common sense, not as "one that fits SO norms") question about troubleshooting, or about code review, or about code golf, or about software recommendations. As of now, they can pick the appropriate site and look at their Help Center to check what they need or what they could missed.

If these questions would be part of Stack Overflow, askers would have to search and identify guidance applicable to their kind of questions among many different ones. If, in addition, there would be a separate site, it would complicate things even further, because askers would have to additionally decide where to post and why. As far as I can tell, current way is simpler and more convenient for askers.

Now, let's look from answerers perspective. Imagine someone preferring to focus on answering particular kind questions, be it troubleshooting or code golf or code review etc.

As of now, answerers simply pick respective site and stick with it (additionally filtering questions by tags that match their technical proficiency), fine. If their kind questions would all be on a Stack Overflow, they would have to find a way to somehow filter these (looking through 100 troubleshooting questions to get to one about code review, give me a break).

To make sure that expert answerers stick (see pearls-not-sand), system would likely have to adopt meta tags, along with additional burden to determine and maintain "necessary" from "useless" ones (why homework isn't okay when software-recommendations are).

Not to mention additional friction that would be there because of inevitable tagging mistakes.

Wow, what a great software recommendation question! Oh, why it's voted down to -10? Ah, that's because it's tagged troubleshooting. Okay, let's retag, fine. Now... how to get its score recovered from -10. How to convince readers that this is only because of wrong tag and not because it was bad from beginning?

If there would be additional "specialised" site, this would also somewhat complicate answerer's life for they would have to track both "their" meta tag at SO and that site. Given my experience of answering at different sites, that probably wouldn't be much harder but still, current way looks simpler and more convenient - you pick a site and stick with it.


Now that we're done with less important folks, :) let's think of the most important ones - the readers.

Current way is simple and straightforward for them: go to appropriate site, find appropriate content.

If it would be all at Stack Overflow, it wouldn't be much harder: they would have to additionally filter by meta tags (give or take inevitable tagging mistakes), fine. The issues here could be for answerer's reputation. In a perfect world, it wouldn't matter much for readers, but it really does (and reasons for that make some good sense, if you think of it).

In our thought experiment though, this could bring additional problems. If you see an answer from 100+K reputation user, how could you tell that their reputation is relevant? What if that answer is just a random drop into code-troubleshooting from someone who acquired their 100K by giving software recommendations?

Note how adding a specialised site would make life of readers even harder, for they would have to look for stuff they want at two places instead of one. They would have to search through both the specialised site and respective meta tag at Stack Overflow.

FWIW readers needs pressure would probably make introduction of meta tags inevitable, even if SO team would somehow decide to ignore interests of answerers. Thing is, forcing web search audience to look through random mix of golf/review/recommendations in order to find needed troubleshooting question and answer would make a fairly severe disadvantage (and I bet there would be competitors out there, ready and willing to leverage it).


As far as I can tell, closing is typically considered a means to only shut down any content that doesn't fit site model. It is probably worth thinking about how it works in a really long term.

Given the history of specialised sites mentioned in the beginning of this answer, one can see it also as means to motivate building more appropriate place(s) for the kind of content that is good per se but has the only drawback of not fitting the Q/A model and norms of a particular site.

5
  • ...for one willing to ask, answer and read good questions (of different kinds), current way seems to have a solid long term advantage over "experimental" one. One of the factors that can put this advantage at risk is wrong / overzealous closures; maybe system could do a better job in detecting and fixing these... though that would be a different question :)
    – gnat
    Commented Feb 26, 2015 at 22:16
  • 1
    There are actually two different types of readers. The type you are talking about go to the site every day and maybe vote. There are almost certainly more of this sort than most of us suspect. But there aren't nearly as many of those readers as there are readers who arrive via Google (and other search engines, I guess). I'm not sure how big an impact specialized sites and ambiguous reputation has on those users. I hadn't thought of site splits as a possible consequence of closing. I'm curious if we can find some evidence for the theory . . .
    – Jon Ericson Staff
    Commented Feb 27, 2015 at 1:23
  • 1
    @JonEricson point about googlers is very well taken, thanks. I wouldn't expect much direct difference for these. Content separation at SO would likely be handled just fine by meta tags, just like it works now with regular tags. Extra specialised sites would be not much of a direct problem either: I bet Google could handle one more site in addition to millions they track already. :) I'd rather be interested to figure indirect impact. Thing is, googlers depend on a smaller group of readers who rate (vote) content, what would be difference for voters and how it could impact googlers...
    – gnat
    Commented Feb 27, 2015 at 6:47
  • ...Wonder how often we miss that "SE food chain" has four groups, not three (my own answer here is example of this happening). Askers -> feed answerers -> feed voters -> feed googlers
    – gnat
    Commented Feb 27, 2015 at 6:48
  • ...there even seem to be a popular belief that there are only two groups: askers and answerers :)
    – gnat
    Commented Feb 27, 2015 at 8:07
8

First I wanted to thank everyone who answered and commented on this question. I particularly appreciate that George Stocker played along with my thought experiement. The other answers also helped me to understand the role of close votes and their goals from various points of view. And I appreciate the thought people put in considering the consequences of removing this one feature. I think we are at a crossroads of closing and nobody is quite happy with the situation as I'll explain below. If we are going to smooth out the experience for everyone, we need a lot of input from a wide variety of users.

Throw out the outliers

MichaelT suggested running the averages in the inner quartile of answers by score:

N       state  average Score Score > 0 Score < 0 Score = 0 
------- ------ ------------- --------- --------- --------- 
 535070 closed 0.783         64.6      0         35.4      
7437633 open   0.721         60.2      0         39.8      

N       close_time  average Score Score > 0 Score < 0 Score = 0 
------- ----------- ------------- --------- --------- --------- 
 363200 early close 0.669         57.9      0         42.1      
 171870 late close  1.024         78.7      0         21.3      
7437633 open        0.721         60.2      0         39.8   

The data with deleted posts excluded looks a lot worse for closed questions. Even when including deleted posts, answers to open questions don't look much better than those to questions closed within a month. If you are looking for the crap predicted by Sturgeon's revelation, don't look at closed questions alone. Closing requires attention from an uncommon set of users. Crap is mostly found in questions that don't get any votes at all:

Question score distribution

This x-axis is question score from -9 to +9 and the y-axis is the count of questions with each score in the various states. The image includes deleted questions, but the linked query does not. What the graph doesn't show is the untold number of questions that are never asked because of rolling rate limits and other quality measures that happen behind the scenes. I'm showing this graph to demonstrate that first line of defense against low-quality questions is ignoring them.

Lately there's been a formalization of that plan. Don't answer bad questions or the askers will keep coming back for more. People go so far as to downvote good answers to bad questions. I've seen comments publicly shaming the answerers. Until I looked at the data, I couldn't figure out why people weren't just closing these questions. But it turns out that since it takes five people to close a question and only one to answer it, askers are often satisfied before their questions are closed. So they come back the next time the have a problem. (Often this ends in a question ban, but what you see is all there is and most people don't see question-banned users.)

Another manifestation of the problem is the "huge rash of "cv-pls" in the chat rooms". In order to speed up closing, folks coordinate their closing activities. To me, that feels pretty cheap, but I can see why it happens: closing is just too slow.

Anonymous feedback

Josh Caswell suggested looking at anonymous feedback. It turns out that anyone can analyze this on SEDE by querying the PostFeedback table. Intially, I didn't think it would make any sort of difference, but here are answer averages using anonymous voting scores:

N        close_time  average Score Score > 0 Score < 0 Score = 0 
-------- ----------- ------------- --------- --------- --------- 
 841222  early close 0.057          6.6      3.8       89.6      
 461596  late close  0.339         14.5      6.7       78.8      
14655977 open        0.219         12.7      5.7       81.6      

Now the contrast between early and late closes is stark. Anonymous feedback is very sparse and strongly correlated to views. Questions closed within a month of asking are just not viewed very often. Questions closed after a month are some of our most-viewed questions. To judge by anonymous feedback, they have some of our most useful answers as well. There aren't many sites that would allow popular pages like these to be put on a path to deletion.

Different people have different "fun" thresholds

One of the frustrations I have talking about this issue is that we can't seem to keep wildly different types of closed questions separate:

  1. Questions that are so poorly written we can't expect the answers to be meaningful to anyone (least of all the original asker).
  2. Questions that ask the same thing that's been asked dozens of times before.
  3. Questions that start off interesting and fun, but eventually get boring.

For the most part, knowledgeable people can come to some sort of agreement about #1 and #2. But #3 is where reasonable people can and do disagree. If you go back and listen to the early podcasts, you can tell that Jeff and Joel were on nearly opposite ends of the spectrum. (In fact, hearing them disagree politely was one of the attractions of listening for me.) But even in Stack Overflow: Where We Hate Fun you can see a concession to finding a proper balance between serious and not-so-serious questions:

I know that we're all programmers, so we love thinking of the world in absolute, binary terms—either fun questions must never be allowed, or fun questions must always be allowed. Well, I hate to be the one to break this to you, but the world is more … floating point. We will sometimes allow fun questions that meet the three broad guidelines I outlined above, but even then, only a limited amount.

Truthfully, it's been a long time since Stack Overflow has tolerated even a limited amount of fun questions. Instead, we've facilitated fun on meta (to a degree), chat, and with events like Winter Bash. When people say that closing works, I suspect that's what they are thinking about. I'm a little sad that the community landed in this place, but it hasn't seemed to hurt the site. Counterintuitively, banning fun has probably contributed to Stack Overflow's growth.

Redesigning the throttle

Shog brought up his jury-rigged lawnmower and that reminds me of a pontoon boat my Minnesota uncles bought so my grandparents could enjoy the lake behind their house. Before we could go out, someone had to start the engine and it would invariable get flooded. Then my uncles would argue about the precise way to set the choke. They'd try again and the engine would become flooded again. Repeat until by some miracle the engine would stay running. This was in the late 1980s when most new cars used fuel injection and I think there was an element of nostalgia for my uncles.

Our close system is like a choke on a carburetor throttle. During normal operation questions remain open and their fate is decided by voting and the answers they receive. But in exceptional situations, it's necessary to close questions down. Like the choke, question closing is really a hack to get around a problem introduced by other parts of the system.

Modern engines don't need a choke because instead of relying on conditions being a certain way, fuel injection engines automatically adjust. On a cold morning you don't have to fiddle with a choke anymore: your car just starts up. We've been putting some time into doing something analogous. Instead of seeing all the new questions (many of which are bad in the sense of #1 above), we'd love to show you questions that you are most likely to appreciate reading and answering. If we can do that successfully, I suspect there will be a lot less need for closing questions. Bad questions won't need closing; they'll just fade away.

Now a change of this magnitude is monumental undertaking and will need to be phased in over time. It's entirely possible that closing is necessary for "fun" questions. We certainly need closing (or something like it) for duplicates. But maybe it's time we stopped thinking of closing as our only defence against poor questions.

7

I would suggest that close votes form an important part of negative reinforcement of poor quality questions. A newcomer to the site doesn't have reputation to 'lose', and they may not have read the 'how to ask' guidance.

If they ask a poor question and get an answer from someone who's enthusiastic (Or y'know just repwhoring) - then they're 'rewarded' for doing it, and will consider doing so again.

If their question is closed unanswered, then that's a clear sign that they need to improve their standards.

And for future users - having a vast morass of poorly articulated questions and substandard answers greatly diminishes the value of StackOverflow as a resource. Far better to close, delete, mark as dupe etc. so it becomes a sort of mega-FAQ, with every answer a useful and valuable one, even if it is very niche.

5
  • 3
    It's not uncommon for people to use the contact form to complain that their question was rejected. When we investigate, occasionally we discover that the question is open, has a useful answer and some helpful comments, but also has a single downvote. I think it's difficult to overstate how effective downvotes are in communicating that something is wrong. We also see situations where not getting an answer results in the asker posting a duplicate of their original question. (This is true whether or not the original is closed.)
    – Jon Ericson Staff
    Commented Feb 19, 2015 at 19:13
  • An anonymous god-knows-who from god-knows-where with god-knows-what credentials, who has no life and spends all time upping has reputation on this site alone has power to yell "shut up" at the rest without any requirement to actually prove their point. You call that reinforcement? At least the web forums are usually run by one or handful of the subject matter experts - here anyone with higher reputation can do whatever they want.
    – ajeh
    Commented Feb 20, 2015 at 14:37
  • 3
    Yes. It takes 5 anonymous god-knows-who who've spent enough time on the site to clock up the required rep to shout 'shut up'. And if they do, another set of people can shout 'nyuh-ha' and vote to reopen. Practically speaking - there aren't many good questions that get closed when they shouldn't be. Aggressive moderation means that in order to be sure a post is 'safe' it needs to be high quality in the first place. Which is good for everyone concerned.
    – Sobrique
    Commented Feb 20, 2015 at 14:47
  • 4
    @JonEricson There's a serious observation bias here. If a user got their question downvoted but also answered, and does not really care about the "downvoted" part.... they are not going to write you a letter to tell about it.
    – user3717023
    Commented Feb 20, 2015 at 20:13
  • 2
    @FamousBlueRaincoat: Sadly, that's not the case. Their questions are very often answered. But they: 1) don't understand the answer, 2) have follow-up questions they left in the comments, 3) are now question banned, 4) think their boss/co-workers will think less of them, 5) are simply offended by the -1 next to their question, or 6) a combination of the above. People hate being downvoted even when it doesn't matter.
    – Jon Ericson Staff
    Commented Feb 20, 2015 at 20:20
6

You've made a great job analysing the data, but it's not about the current data, it's about the feedback between the system and the users.

While downvoting is a strong signal that the question is of poor quality, I suppose most of the poor question owners are perfectly aware of that. They often write they have no idea what they are doing, they don't understand the basics, and they don't have time, or are unwilling, or are unable to learn them.

Often they admit having only basic (or no) English skills and using translators.

As long as they finally get the answer, even only once a time, they will continue to post. Even if their post get ignored, those desperate enough may post them again and again. And there are enough rep whales* out there to answer even the most lazy question to get some rep, giving those help vampires a positive feedback and encouraging them to post again, no matter how many downvotes they get. Early closing stops rep whales* giving positive impact to help vampires.

*some people are very touchy about some words, and I find X-word notation utterly stupid, so let's refer to some other animal instead

7
  • 1
    i.sstatic.net/MW9QR.gif
    – gnat
    Commented Feb 19, 2015 at 14:54
  • 1
    The most striking discovery I made is that if we frame the problem as a race between question closers and question answerers (in essence, the graph) answerers are winning. If the goal is to prevent people from getting answers (and that's a strange goal in my opinion), closing fails to do the job. Thanks to the rolling rate limit, downvoting is more efficient. Unfortunately, once a question is closed, it tends to stay that way. Not even a heroic edit is sufficient. [citation needed]
    – Jon Ericson Staff
    Commented Feb 19, 2015 at 19:05
  • 3
    @JonEricson the goal, as I understand it, is to provide good and meaningful search results for the next person with that question. Its not about allowing or preventing a single person asking a question from getting an answer. We fail just as hard when closing a good question as we do when a user from google finds a poor quality answer because the question is poorly asked. The poorly asked and answered question also impacts the quality of Stack Overflow's brand from being "a place to get good answers" to "a place to get your homework done for you faster than Yahoo Answers."
    – user289086
    Commented Feb 19, 2015 at 20:07
  • 2
    @MichaelT: I don't know about you, but I don't often find those questions via Google. More often, I get results of mediocre questions that were closed for reasons that have nothing to do with whether the answers helped me. For instance, on Nov. 6, 2014 I was looking for a Unicode arrow. (Don't ask me why; I can't recall.) One of several SO results was: HTML unicode character for a “tall” right chevron? As someone trying to find a bit of information, I didn't care whether the question was closed.
    – Jon Ericson Staff
    Commented Feb 19, 2015 at 20:48
  • 2
    @JonEricson my visibility into the anon-feedback is a bit hampered by not having it in the 10k tools. One might look into how much feedback (positive or negative) there is on posts that are closed (and deleted?) as one bit of insight into the "what do random people who hit the site from outside think?" and the frequency of such events.
    – user289086
    Commented Feb 19, 2015 at 21:20
  • You wrote: "As long as they finally get the answer, even only once a time, they will continue to post." And that is a big threat to the Internet because?... Because you want to save 2KB on the Internet?
    – ajeh
    Commented Feb 20, 2015 at 14:40
  • 1
    @ajeh: Why do you think the capacity of the servers and network bandwidth has anything at all to do with his argument? It's not eve a peripheral consideration. Commented Feb 21, 2015 at 17:29
5

In a world where close votes don't exists...

I see several differences immediately, that would have to be considered. There are privileges at 50, 250, 500 and 3000 rep that become obsolete or confusing.

  • 50: Flag a post. What would the purpose of flagging be in this case? Currently, a flag by a user with lower reputation dumps the question into a review queue where higher rep users can cast close votes (if needed). Without close votes, I see the review queues being used only to further down vote poor questions or cast deletion votes. The middle step of closing/putting a post on hold, is gone. The reputation gap between flagging and deleting is a VERY wide gap. Flag a post remains relevant for spam and offensive posts.
  • 250: View/cast close votes on your own post. This new privilege no longer exists without close votes
  • 500: Access to first post and late posts review queues. Much like the new privileges at 50, this allows users to see what others have flagged but all they can really do is upvote or down vote content. They can't flag the post for closure of poor questions.
  • 3000: Cast close/reopen votes. This privilege no longer exists without close votes.

Without close votes, what is the next step in the process after a user flags a post? The community aspect of moderation loses a big part of the quality control that exists.


In my opinion, quality control would become much more focused on the up/down votes. But, I'm not sure that would be enough. There are over 8,700 questions on Stack Overflow with a score less than -5 that are open right now. Does this make the question bad and should be closed? Of these 8,700, only 300 don't have an answer at all. The other 8,400 managed to gather at least one answer. Clearly, a score of -5 isn't enough to ward off answers to questions received poorly by the community.

The other disadvantage of utilizing only upvotes/downvotes to determine the fate of a question is overcoming years of inertia when community standards change. Several of the old "book" questions, that are closed now, would fall into this category. How does the community turn around 100s or 1000s of upvotes to say that such a question is no longer appropriate for Stack Overflow? Instead of popping to an appropriate chat room and requesting a cv-pls, users will pop in an ask for downvotes on a question. Many sub-communities around Stack Overflow utilize this process to keep their corners of the site clean. If close votes are removed, the people that want to help have one less method to do so.

Where do duplicates fall into all of this then too? Gathering enough duplicate votes on a question will close a question. If this closure no longer takes place, we'll be (more) swamped with the number of duplicates that are seen. NullPointerExceptions and floating point arithmetic errors will be all that we ever see.


Close votes are a vital part of what keeps the lowest quality questions off the site. They are a vital part of a user's progress from lurker to new user to trusted user (and everything in between). They are the aspect of the community that users have the most say in. From the time a user gets 50 rep, they can indicate a question should be closed for various reasons.

8
  • Yes, questions with score -5 or below can probably easily be closed if somebody would take their time. On the other hand why even bother to close vote. Just downvote and be done with it. Commented Feb 19, 2015 at 9:59
  • 3
    @Trilarion close vote so that the question gets closed. The Roomba will delete closed questions more quickly than open ones. Sufficent (2) comments will prevent the Roomba from deleting unanswered low scoring questions. If the question really needs to go, closing is a key step in that process - either to help automated processes or to allow 10k users to cast delete votes.
    – user289086
    Commented Feb 19, 2015 at 14:24
  • @MichaelT In the current system yes. But most probably you could also just tweak the system so that downvoting is the key step to deleting questions instead of closing. Commented Feb 19, 2015 at 15:50
  • 2
    @Trilarion but fun/popular questions that don't fit the Q&A format don't get down voted. Just look at reddit /r/programming and /r/programmerhumor and consider that what is popular isn't necessarily a good question for the Q&A format. Up/down votes and close/reopen votes measure different things. "What is the best furniture for programmers" had +23 votes when it was deleted on P.SE. I don't want to even think about "What is your favorite dilbert cartoon". Even things that are more professional such as "what topics should be covered in a college class?" get lots of votes.
    – user289086
    Commented Feb 19, 2015 at 15:56
  • @MichaelT But all these questions are offtopic. In my answer below I proposed to close only offtopic and duplicates and to deal with the rest with pure downvoting. So can you also find heavily positively voted ontopic questions that should be closed? Commented Feb 19, 2015 at 16:02
  • @Trilarion eaislly. programmers.stackexchange.com/questions/84278 is on topic, but too broad. programmers.stackexchange.com/q/102090 is on-topic ish, and opinion. programmers.stackexchange.com/questions/111633 is on topic ish, and a poll. programmers.stackexchange.com/questions/163185 is a discussion about a quote by a well known person. programmers.stackexchange.com/questions/91629 is a book list. Just scroll through this search.
    – user289086
    Commented Feb 19, 2015 at 16:06
  • 1
    ... and while I am linking to Programmers.SE there - remember that P.SE started off because SO was having trouble with the 'fun and popular' questions on this site. Not closing questions returns to the days of SO before P.SE was created (and the days of P.SE before it was realized that watercooler questions don't work on a technology site).
    – user289086
    Commented Feb 19, 2015 at 16:13
  • @MichaelT Yes, these are popular questions, but also from the past and not very many, certainly not the majority of closed questions. programmers.stackexchange.com/questions/111633/… even could probably made ontopic by reformulation. The book list is just plainly offtopic. Maybe we should have a discussion just about these cases ("Should popular questions really be closed?"), but I suggest to rather discuss whether unpopular questions (which do not end up with large numbers of upvotes) really need to be closed. Commented Feb 19, 2015 at 16:55
5

The analysis posed in this question roughly corresponds with my experience, and I wonder if others feel the same way. That is, I often find that one of the following is true when I happen upon a closed question from a particular search:

  • I find it relevant and wish that it would have been left open for more answers;
  • I find the question to be reasonable and I have an answer to it that I am unable to post.

I'm not thinking about the case of duplicates here, which I believe are reasonable to be closed and/or cross-referenced. However, I think there is a general tendency to vote to close questions due to lack of knowledge of the subject matter or a rush to pass judgment on new users rather than the question itself being badly written.

The end of the article My Love-Hate Relationship with Stack Overflow: Arthur S., Arthur T., and the Soup Nazi (referenced above) also seems to corroborate this experience.

We've experienced a lot of this for posts on the meteor tag, which is an up-and-coming web framework that is understandably unfamiliar to many SO users. To counter this effect, we actually have a group of people that is ready to vote to re-open questions that we believe were closed in error.

One productive thing that could come out of this conversation are potential changes to the voting system that would maintain the positive benefits of closing duplicate, "give me the code", etc. questions while reducing the impact of "soup nazi" behavior that is at best unfriendly toward new users and at worst preventing useful knowledge from accumulating.

9
  • 6
    It's extremely important to note that there's selection bias inherent in what you're talking about: closed and deleted questions don't show up in your search results at all. You cannot judge closure to be erroneous on balance without seeing the balance.
    – jscs
    Commented Feb 19, 2015 at 17:31
  • 1
    Agreed, good point. But consider this idea, then: questions that are closed but not deleted by some interval revert to being open.
    – Andrew Mao
    Commented Feb 19, 2015 at 17:57
  • 2
    "revert to being open" not a good idea, as the reason for closure presumably still applies. Though that might be mitigated if the votes of asker and answerers, if any, are ignored for deletion-criteria. Commented Feb 19, 2015 at 18:34
  • I see where that idea springs from, but because close voting requires semantic information, I'm skeptical that reopening could be automated.
    – jscs
    Commented Feb 19, 2015 at 18:58
  • @AndrewMao do you really want old posts like this, this, or this to get reopened after some interval because they haven't been deleted yet? (and I can easilly find tens of thousands more...)
    – user289086
    Commented Feb 19, 2015 at 20:19
  • 1
    @MichaelT I was just brainstorming. I'm sure I could find many closed posts of the other variety as well; anecdotes are not statistics.
    – Andrew Mao
    Commented Feb 19, 2015 at 20:30
  • 2
    @AndrewMao possibly. is:question closed:yes locked:no answers:0..2 duplicate:no views:0..2500 returns back 166k results and the median score is 0. The boundary between the highest 1/8th and all the rest has a score of 1. Even browsing around the 500 highest scored Sturgeon's law is still in force (and Sturgeon's long tail is on my side there) - they may be well written, but they aren't good questions for the Q&A format.
    – user289086
    Commented Feb 19, 2015 at 20:40
  • 2
    @AndrewMao You are another victim of drive-by downvoting. Proves my point.
    – ajeh
    Commented Feb 20, 2015 at 14:33
  • As another Meteor answerer, I can fully corroborate Andrew's story. I've often seen animuson, CasperOne and others close questions they simply didn't understand. When the question was finally reopened, the comments from those asking why it was closed and explaining why it shouldn't have, had been deleted. Commented Feb 22, 2015 at 2:32
-3

What am I missing?

EVERYTHING

The linked article (http://www.embeddedrelated.com/showarticle/741.php) don't claims that questions should not be closed, so there is no point in analysing numbers about question closures to claim that question closures are good. We all know that closing crap is good and the article author knows that also.

The problem with your analysis is the implied strawman argument that he supposingly opposes closing question and then you analysed a bunch of number to prove that closing question is good, which indeed you proved. But he is not saying that closing question is bad, so you are entirely missing the point of the article!

What he is arguing about is closing legitimate questions that shouldn't be closed, like the question about getting rid of the this keyword and the angles (mis)calculation question in python. What he is arguing is that closing good valid questions due to minor easily-fixable problems scare people away (specially newcomers) and gives a bad name to StackOverflow. Lets quote this:

Closing someone’s question sends a message that their work isn’t good enough, that they aren’t wanted here. From my point of view, even if only 1% of the close-votes are wrong, that’s too many. Even if a tiny percentage of people are harassed by Soup Nazis, it has a pervasive effect that just encourages hostility and banishment, without any accountability. It sends the message that it’s okay to continue the practice.

Again, he is not claiming that closing is bad. He is claiming that even a tiny percentage of wrong-closing questions is too many.

6
  • 5
    And how would you reccommend fixing this problem? The only sure-fire way to get rid of false-positive closures is to get rid of closing altogether, which is what the rest of the question/discussion is about. Commented Feb 26, 2015 at 18:53
  • 1
    @BradleyDotNET No. You are now commiting another fallacy, something like all or nothing, black or white. I didn't came to say what we should do to fix the problem, I don't know how to fix it. I came here just to say that we (or at least the OP) were looking to the wrong problem and getting a conclusion that is irrelevant to the real described problem. Now that you are asking for suggestions, we need to find novel ways to avoid false-positives. This could be closure-reviews and audicts, or something else entirely novel. Commented Feb 26, 2015 at 18:59
  • 6
    I would claim that none of those methods are fool-proof (though we would all be interested to hear a way to make it better) so if you accept the claim that "a tiny percentage of false-positives is too many", then you are forced to conclude that 0 is the only acceptable number, and the only way to guarantee that is removing all closure (the premise of this question). Clearly suggesting this has had the desired effect, as other suggestions have been offered and discussed. Commented Feb 26, 2015 at 19:05
  • 1
    @BradleyDotNET I really disagree with you. I don't think that we are forced to conclude that the only way is removing all closure. Commented Feb 26, 2015 at 19:32
  • Then we disagree. Worse things have happened :) If you have a different conclusion, I'm sure Meta would love to hear about it! Or come by the Tavern and chat about it. Commented Feb 26, 2015 at 19:33
  • 1
    So the funny thing about the How can I get rid of this keyword in local functions? Even after it was reopened (likely as a result of the blog post), it hasn't gotten any more answers. I suspect that's because the answer that came in before the question was closed was about as good as one could hope for. There are a lot of things I liked about that post, but the examples of "bad closures" are not ideal in my opinion. Closure signals community consensus even when people disagree. In other words, we can't even agree on the 1% that's wrong.
    – Jon Ericson Staff
    Commented Feb 26, 2015 at 19:41
-4

The vast majority of my questions are very specific to a certain aspect of an API or language.

Anecdotally, some of my questions have been closed by multiple people with high point counts, after they received good responses. I think this is because people are very close-happy on Stackoverflow and don't take the time to read through a question carefully. I did manage to get most of them re-opened. Many of my questions are down-voted for the same reason, I think.

The mechanism doesn't always work.

I find that Stack Overflow is very good at getting "easy questions" answered. One of my questions, which actually had to do with something else, was closely related to an extremely common question Javascript beginners make. It got a large number of responses very quickly.

Search engines (mostly Google) play a very important role.

So, even slightly niche answers to questions are difficult to find (get) on Stack Overflow at present. Despite that, the great value in Stack Overflow is that it has many common "natural language queries" about a certain topic. That makes answers to such queries much easier to find with Google, for example.

You do have a lot of false positives, in my experience. I'm not sure what would happen if questions were not closeable. I suppose it could make things more uncomfortable for people asking common questions as there may be far more such questions. But it would make it less of a hassle for me when I find a recently asked and answered question was stubbornly closed for no good reason.

Stack Overflow is both a topical site about the most recent programming APIs, and a language reference.

Many people search Stack Overflow content based on date. That means old stuff is typically not going to be visible. The proliferation of close votes would make it less useful as a topical site.

I suppose many users want Stack Overflow to be a reference site, thus the seeming need to close questions they don't think make sense or are "useful".

-5

Before you downvote this answer (I know you will), ask yourself how "value" can be measured. Three proxies for value come to mind:

  • upvotes and stars
  • time and effort
  • money

If we look at upvotes, there are thousands of questions with more than, say, 20 upvotes (let alone upvotes for the answers) that have been closed, or even deleted. (Someone please link this to the appropriate stats; I've made the answer CW).

Time and effort: there are mirrors of deleted questions from StackOverflow, some "legendary". Clearly someone cares enough to put time into that, and clearly there's also demand for it.

Money: Quora, a system without the concept of "closing" (except of course for spam, illegal materials and such) is valued at over a billion dollars.

Sure, Quora has its junk, but instead of closing or deleting content, they bank instead on good search algorithms. From my experience, search on Quora works very well, and the voting mechanism also surfaces good answers to the top.

What Quora understands is that search will only get more intelligent, but alienated users will never come back; worse, they'll write about their frustration, and pull others away with them, as happened with the guy who wrote that explodingly popular "My love-hate relationship with StackOverflow" post.

1
  • 4
    The "legendary" posts you've linked to are the poster children for GTKY timeburning fluff. "What is the best comment you've encountered?" "What is your favorite cartoon?" "Best programmer joke"? This is why we shouln't have closure?? N.B. that, of the four in the top handful that actually seem to have technical value, three are not deleted. programmers.stackexchange.com/q/103807 programmers.stackexchange.com/q/46716 stackoverflow.com/q/9033 The first two were moved to Prog.SE, and the last is closed because it already has too many "answers" and doesn't need more.
    – jscs
    Commented Feb 22, 2015 at 8:41
-10

Very simple. Users would downvote instead (because downvoting and closing are partly synonymous) and request to filter out negatively scored questions. And otherwise this site would mostly pretty much move on.

I would actually only close for offtopic questions and duplicates. The remaining questions are sufficiently dealt with by downvoting and commenting.

And I would actually not show the score if the question has been downvoted to below -5. Just display "<=-5" and that is enough.

And the key philosophy would be: It is better to efficiently ignore bad stuff instead of fighting it.

What we would need to do in order to check this is taking some bad ontopic questions ("too-broad", "unclear", "debugging help") and not close them and wait what the score will be after some time. If the score is negative then the proposition that downvoting can replace closing is probably true.

15
  • The real question behind this question here is: Why don't we try it out? Commented Feb 19, 2015 at 9:55
  • 13
    why do you think that users would downvote? that certainly wouldn't be my case: if I knew that site allows questions like "does one debug better when sitting in the toilet", I would vote these up. Oh, and I would definitely vote up most funny answers to questions like that
    – gnat
    Commented Feb 19, 2015 at 13:57
  • 1
    @gnat Come on. I clearly wrote I would close offtopic questions in the second paragraph. You must have misread. Commented Feb 19, 2015 at 15:57
  • 3
    but of course! I thought my example is on-topic, isn't it. An even stricter on-topic variant would be like "efficient debugging for NPE in Visual Studio when sitting in the toilet"
    – gnat
    Commented Feb 19, 2015 at 16:02
  • 1
    Irony, this answer got voted down. I thought I'll share my opinion, but not after this kinda behavior..
    – Siddharth
    Commented Feb 20, 2015 at 10:34
  • @Siddharth Oh, I have no hard feelings about the downvoting. After all it just indicates that there are different opinions with a strong majority but not that the answer is written badly in any way. I'm too old to get discouraged by opposing views. A lively discussion always draws from them. Of course if you play with the thought of posting another similar idea I already can forecast the voting result quite accurately, I think. Commented Feb 20, 2015 at 10:53
  • @Siddharth, you took the words out of my mouth. Downvoting, when it means "I disagree", but yet still affects the non-meta SO scoring is only making people with valid opinions stay clear. How many times in our lives have we seen valid opinions too quickly dismissed not because they were wrong, but because they were merely complex in nature? I've seen such opinions turn the most ardent of naysayers around in time. But not here. HERE, we end up with people being too afraid to say anything.
    – user4229245
    Commented Feb 21, 2015 at 16:58
  • @tgm1024: downvoting here on meta only affects the answers score, showing readers opinions of a) the clarity and b) the usefulness of the post / merit of the expressed view (that those are comingled is unfortunate, but separating them would be really hard). It does not reflect on reputation or anything else; Thus, if disagreement makes you afraid, instead of leading you to rethink, carefully present and underpin with data and citations your position, you deserve being ignored / overridden. Commented Feb 21, 2015 at 17:25
  • @Siddharth: Why be afraid of downvotes? See my above comment. Commented Feb 21, 2015 at 17:27
  • This answer is a opinion, and a good opinion. May not be good enough for accepting right away, but good enough to talk about this as a idea. The statistics clearly state that closing questions is tougher than you think. The fact that we have 12k questions marked for review is alarming. The fact that closing a question needs 5 people to review is JUST crazy. But no one is ready to discuss, just state a opinion and get downvoted from answering here ever again. Immature bunch of rogues..
    – Siddharth
    Commented Feb 21, 2015 at 17:42
  • @Deduplicator, when I posted here on meta, I received downvotes that affected my SO score. And as such it dipped below 50 preventing my commenting elsewhere.
    – user4229245
    Commented Feb 21, 2015 at 17:53
  • 1
    @Deduplicator, You said: " It does not reflect on reputation or anything else; Thus, if disagreement makes you afraid, instead of leading you to rethink, carefully present and underpin with data and citations your position, you deserve being ignored / overridden." ---- This is only if the people judging the question are capable of setting aside their own egos and rushes to judgement (and piling on), and pervade all human discourse. Given what I've seen thus far, SO is far from immune to this.
    – user4229245
    Commented Feb 21, 2015 at 17:57
  • 1
    @tgm1024: Nope, that's not affected by such. Anyway, your main-profile does not show any downvotes at all, though I concede that in any sizable group there is always an idiot who has to break the rules, like voting on the user (and searching out unrelated posts on main). Still, even if you got a downvote (and subsequently deleted) on main, it could be unrelated, so try to avoid rushing to judgement as well (and accept this is the internet). Commented Feb 21, 2015 at 18:00
  • 1
    @Deduplicator, I deleted the question I got a lot of downvotes on so that I could return to discussion in other posts. About "always an idiot"; Some of the most learned, careful, and thoughtful folks I've known on the planet make the knee-jerk mistakes that I'm talking about. Don't start with the idea that SO is somehow the first place in history to be immune to human foibles. Start knowing that SO will encounter it routinely and consider that piling on and knee-jerk reaction will happen.
    – user4229245
    Commented Feb 21, 2015 at 18:06
  • This dovetails directly into allowing the editing of other's questions outside of trivial formatting. That is likely the worst of all ideas: it has an arrogance that allows one to judge what is in the person's head, and further is a public chastising that he's not forming his own question properly.
    – user4229245
    Commented Feb 21, 2015 at 18:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .