Skip to main content
1 of 6
Jon Ericson Staff
  • 21.5k
  • 27
  • 153
  • 182

First I wanted to thank everyone who answered and commented on this question. I particularly appreciate that George Stocker played along with my thought experiement. The other answers also helped me to understand the role of close votes and their goals from various points of view. And I appreciate the thought people put in considering the consequences of removing this one feature. I think we are at a crossroads of closing and nobody is quite happy with the situation as I'll explain below. If we are going to smooth out the experience for everyone, we need a lot of input from a wide variety of users.

Throw out the outliers

MichaelT suggested running the averages in the inner quartile of answers by score:

N       state  average Score Score > 0 Score < 0 Score = 0 
------- ------ ------------- --------- --------- --------- 
 535070 closed 0.783         64.6      0         35.4      
7437633 open   0.721         60.2      0         39.8      

N       close_time  average Score Score > 0 Score < 0 Score = 0 
------- ----------- ------------- --------- --------- --------- 
 363200 early close 0.669         57.9      0         42.1      
 171870 late close  1.024         78.7      0         21.3      
7437633 open        0.721         60.2      0         39.8   

The data with deleted posts excluded looks a lot worse for closed questions. Even when including deleted posts, answers to open questions don't look much better than those to questions closed within a month. If you are looking for the crap predicted by Sturgeon's revelation, don't look at closed questions alone. Closing requires attention from an uncommon set of users. Crap is mostly found in questions that don't get any votes at all:

Question score distribution

This x-axis is question score from -9 to +9 and the y-axis is the count of questions with each score in the various states. The image includes deleted questions, but the linked query does not. What the graph doesn't show is the untold number of questions that are never asked because of rolling rate limits and other quality measures that happen behind the scenes. I'm showing this graph to demonstrate that first line of defense against low-quality questions is ignoring them.

Lately there's been a formalization of that plan. Don't answer bad questions or the askers will keep coming back for more. People go so far as to downvote good answers to bad questions. I've seen comments publicly shaming the answerers. Until I looked at the data, I couldn't figure out why people weren't just closing these questions. But it turns out that since it takes five people to close a question and only one to answer it, askers are often satisfied before their questions are closed. So they come back the next time the have a problem. (Often this ends in a question ban, but what you see is all there is and most people don't see question-banned users.)

Another manifestation of the problem is the "huge rash of "cv-pls" in the chat rooms". In order to speed up closing, folks coordinate their closing activities. To me, that feels pretty cheap, but I can see why it happens: closing is just too slow.

Anonymous feedback

Josh Caswell suggested looking at anonymous feedback. It turns out that anyone can analyze this on SEDE but querying the PostFeedback table. Intially, I didn't think it would make any sort of difference, but here are answer averages using anonymous voting scores:

N        close_time  average Score Score > 0 Score < 0 Score = 0 
-------- ----------- ------------- --------- --------- --------- 
 841222  early close 0.057          6.6      3.8       89.6      
 461596  late close  0.339         14.5      6.7       78.8      
14655977 open        0.219         12.7      5.7       81.6      

Now the contrast between early and late closes is stark. Anonymous feedback is very sparse and strongly corrolated to views. Questions closed within a month of asking are just not viewed very often. Questions closed after a month are some of our most-viewed questions. To judge by anonymous feedback, they have some of our most useful answers as well. There aren't many sites that would allow popular pages like these to be put on a path to deletion.

Different people have different "fun" thresholds

One of the frustrations I have talking about this issue is that we can't seem to keep wildly different types of closed questions separate:

  1. Questions that are so poorly written we can't expect the answers to be meaningful to anyone (least of all the original asker).
  2. Questions that ask the same thing that's been asked dozens of times before.
  3. Questions that start off interesting and fun, but eventually get boring.

For the most part, knowledgeable people can come to some sort of agreement about #1 and #2. But #3 is where reasonable people can and do disagree. If you go back and listen to the early podcasts, you can tell that Jeff and Joel were on nearly opposite ends of the spectrum. (In fact, hearing them disagree politely was one of the attractions of listening for me.) But even in Stack Overflow: Where We Hate Fun you can see a concession to finding a proper balance between serious and not-so-serious questions:

I know that we're all programmers, so we love thinking of the world in absolute, binary terms—either fun questions must never be allowed, or fun questions must always be allowed. Well, I hate to be the one to break this to you, but the world is more … floating point. We will sometimes allow fun questions that meet the three broad guidelines I outlined above, but even then, only a limited amount.

Truthfully, it's been a long time since Stack Overflow has tolerated even a limited amount of fun questions. Instead, we've facilitated fun on meta (to a degree), chat, and with events like Winter Bash. When people say that closing works, I suspect that's what they are thinking about. I'm a little sad that the community landed in this place, but it hasn't seemed to hurt the site. Counterintuitively, banning fun has probably contributed to Stack Overflow's growth.

Redesigning the throttle

Shog brought up his jury-rigged lawnmower and that reminds me of a pontoon boat my Minnesota uncles bought so my grandparents could enjoy the lake behind their house. Before we could go out, someone had to start the engine and it would invariable get flooded. Then my uncles would argue about the precise way to set the choke. They'd try again and the engine would become flooded again. Repeat until by some miracle the engine would stay running. This was in the late 1980s when most new cars used fuel injection and I think there was an element of nostalgia for my uncles.

Our close system is like a choke on a carburetor throttle. During normal operation questions remain open and their fate is decided by voting and the answers they receive. But in exceptional situations, it's necessary to close questions down. Like the choke, question closing is really a hack to get around a problem introduced by other parts of the system.

Modern engines don't need a choke because instead of relying on conditions being a certain way, fuel injection engines automatically adjust. On a cold morning you don't have to fiddle with a choke anymore: your car just starts up. We've been putting some time into doing something analogous. Instead of seeing all the new questions (many of which are bad in the sense of #1 above), we'd love to show you questions that you are most likely to appreciate reading and answering. If we can do that successfully, I suspect there will be a lot less need for closing questions. Bad questions won't need closing; they'll just fade away.

Now a change of this magnitude is monumental undertaking and will need to be phased in over time. It's entirely possible that closing is necessary for "fun" questions. We certainly need closing (or something like it) for duplicates. But maybe it's time we stopped thinking of closing as our only defence against poor questions.

Jon Ericson Staff
  • 21.5k
  • 27
  • 153
  • 182