150

The size of the backlog in Stack Overflow's close review queue has been an issue for a long time now:

I find that having such a huge backlog, makes it very hard to properly deal with closures.

What steps or system changes can we make to get the huge close backlog on Stack Overflow, under control?

The additional visibility granted to this backlog by the review system has sparked a lot of discussion here recently. The problem with most of these discussions is that they don't agree on what the problem to be solved is... Other than that it must somehow involve the existence of a backlog.

In order to understand the problem, we first have to agree on a goal. Here's what I think the goal should be:

When you vote to close a question, if you've correctly identified a serious problem with the question and the problem isn't corrected, the question should be closed. And if you haven't, your vote should go away.

If you don't agree with that, you should probably stop reading here... But I'm hoping this is something we can agree on; that even if /review didn't exist, this would still be a good baseline expectation for close-voting. We've built up an awful lot of behavior that depends on this assumed goal.

We can further refine that goal a bit; ideally,

  • Problematic questions should be closed quickly - if it's a real problem, letting it hang around stinking up the place isn't doing anyone any favors, and even conscientious users are less likely to put time into correcting them long after they've been posted.

  • Questions that aren't causing problems shouldn't have close votes hanging off of them, like some terrible blade threatening to drop at any time.

If we're in agreement on the goal, then I think we can all agree on the real problem:

People are voting to close questions, and their votes are hanging around for months - even years - but the questions aren't getting closed.

As with the goal statement, we can refine this by observing that:

  • Some lousy questions are hanging around for long periods of time, clogging up search results and with little or no feedback given to the asker.

  • Some questions are being closed years after being asked, and often after attracting perfectly good answers whose very existence casts doubt on the need for closure.

Again, if you agree with the goal you should agree with this statement of the problem; if you don't, then you should've stopped reading already. Stop wasting time you could be using to tell me how wrong I am!

And again, this isn't specific to /review - it's been a problem ever since we instituted the 100-view requirement for close-vote aging, at which point it largely supplanted the previous problem: close votes expiring before anyone else who could close had seen them.

So now...

About review... A deficit and a long, long tail

Here's a picture:

The entire history of the modern close review queue

You see the problem, right? No, it's not the 57K backlog we started with; there are only 11K or so of those left. It's that we've only been "in the black" for 5 months out of the last 16. When most days - most months - end with a deficit, you never pay down the debt you started with... And eventually, folks get discouraged.

There are a bunch of ideas for addressing this problem out there, some of them good, some of them awful. I'm not going to discuss most of them tonight, though I'll try to do a better job of at least commenting on them in the next few weeks. The goal of this post is to try and focus everyone on the same problem, so I'm trying to keep it relatively brief... so to save on words, here's one more picture:

The long tail

Don't worry, you're not supposed to be able to read the text on that; just look at the shape. The single largest block of questions in the queue right now is attached to the tag, amounting to about 10% of the entire queue. Fast on its heels is , at which point it starts to flatten out really fast. By the time we hit , we're at 1% of the queue.

This is an opportunity...

See, the problem with a lot of the suggestions floating around here right now is that they make a couple of shaky assumptions:

  1. Most of the questions in the queue actually need to be closed.
  2. A handful of people working REALLY HARD could close them all in no time, if we just gave them more privileges / required fewer close votes / skipped the whole "review" thing and just closed them all automatically / etc.

#1 I'm just not seeing. Oh, for sure there's a lot of crap in there... But there's also a lot of stuff that's just in the queue because someone didn't know what the hell they were looking at and decided to flag it, or thought "minimal understanding" meant "already solved the problem and is just posting here for typing practice". Especially once you get outside the PHP tag.

#2 is true in theory, but... We've kinda been trying to move away from that - the big hope for review was that it would empower folks enough that we wouldn't need 15 moderators closing stuff all day long to keep up. And the truth is, it's a lot harder to review stuff when you don't know jack about the topic. I'm pretty comfortable in , but pretty much everything in makes me suspect the author was drinking heavily when he wrote it; trying to distinguish bad from just weird is taxing... And, I'm fairly certain, less accurate.

I'm going to try and use this thread as sort of a scratch-pad for tracking changes as we propose, discuss and implement them, as a way of staying focused. If you disagree with the goal (or the problem) statements above, please discuss them in answers here, but save discussion on the proposed solutions for their own threads.

Progress so far

A couple of important changes have been made:

  1. Allow direct linking to a filtered /review queue has enabled folks to jump directly into their tag of choice when reviewing.

  2. https://meta.stackexchange.com/questions/199879/fuzzy-the-number-of-questions-in-the-close-review-queue-a-dopamine-for-the-shut/209774#209774 has reduced the visible backlog, helping to focus efforts on questions most likely to be closed or left open.

Coupled with these, we has a bit of a shindig to try and get more folks involved in the process: https://meta.stackexchange.com/questions/223497/lets-burn-down-the-close-queue

The results so far have been promising. Here's a graph of the number of active reviewers by week:

active reviewers by week, past 30 weeks

And here's a variation on the "deficit" graph above, this time comparing questions collecting their first close vote or flag to those where the last vote or flag was dismissed (by either closing or aging):

first vote/flag vs last vote flag, last 30 weeks

It's not all roses though; the average time between the first vote and completion of review for reviewed questions flagged in is 28 days - up from 12 days a month ago. We have to assume the bulk of this is due to the work being done tearing down the backlog, but we'll want to keep an eye on this to make sure we're not dramatically delaying closures (and thus depriving askers of timely guidance) long-term.

Action items

We need to continue this two-pronged attack here: increase the number of active reviewers, and focus their efforts on areas most likely to benefit. I believe the next steps in these directions are:

Keep up the good work!

32
  • 4
    your "refinement" is incomplete, it lacks a bullet that would address reasonably good questions, flagged / voted on incorrectly and polluting the queue for long weeks for votes / for eternity for flags. Any passer-by who managed to randomly get 15 rep can drop a senseless flag onto a perfectly good question and have their garbage infinitely hang in the queue waiting for random vote to come. This game sucks, that's why I'm out
    – gnat
    Commented Nov 21, 2013 at 5:23
  • 5
    I just know someone's going to make a joke about why we're reading posts about the close votes queue instead of actually doing our daily 40 XD. Commented Nov 21, 2013 at 6:13
  • 1
    @Shog9 privileges page says "Flag to close" at 15 - that's like 500K users by the way (or 300K, just in case if privileges doc is wrong and it's granted at 50 indeed - 9000*36~=300,000). Other than that, I merely pointed that your refinement lacks a bullet - how you'd address it is sure up to you, it's just missing from the list
    – gnat
    Commented Nov 21, 2013 at 7:51
  • 1
    and again, talking about it but doing nothing about it - it's what keeps us here, talking, arguing, laughing => wanting to come back.
    – user221081
    Commented Nov 21, 2013 at 8:28
  • 2
    @Shog9 This post is a hufe step forward. Defining THE problem and outlining a strategy to fix - this is exactly what I wa looking for. Keep posting. If there's anything I can do to help - do not hesitate to ask!
    – Shai
    Commented Nov 21, 2013 at 9:25
  • 5
    A statistic which would significantly help in analysis is the raw number of active reviewers, as well as the number of potential reviewers; is there a convenient way to find this information? (This could either address or justify concerns about the psychological effects of the queue size.)
    – user1131435
    Commented Nov 21, 2013 at 20:34
  • 1
    I seem to be missing the purpose of this post / what to do with it / whether any answers should be posted and what they should be about (are answers not supposed to be posted, or are we just supposed to duplicate all the (decent) suggestions here?) - most, if not all, of us already know that there's a problem. Oh, and most suggestions that come to mind do not assume #1 (most of the questions in the queue actually need to be closed). Commented Nov 22, 2013 at 17:35
  • 4
    The true underlying problem is too many users asking inappropriate questions (which, now that I think about it, hasn't been addressed enough during all this). Ignoring that for the moment, the underlying problems are being addressed. Underlying problems - (1) Too many questions that shouldn't be closed are flagged or close-voted (should be easy enough to figure out if this is the cause). (2) Too few users are involved. (3) The involved users aren't doing enough. (4) The system is broken. Most proposals suggest long-term solutions to 2, 3 or 4, which would likely fix the symptom forever. Commented Nov 22, 2013 at 17:59
  • 1
    Things like vote ageing and number of votes required to close questions falls under #4. To fix #0, (1) we probably need to redesign "how to ask", assuming they're not going to read a whole lot or click on any links we give them, (2) we also need to get rid of robo-reviewers on the "First Posts" queue (or just those who don't know what's appropriate, or aren't willing to go through the effort required), (3) perhaps extend "First Posts" to include a little more, (4) perhaps set some 'traps' (e.g. tools)). Commented Nov 22, 2013 at 19:39
  • 2
    @Dukeling side bit, at 500 rep, one can do first post reviews and flag items for the close queue, but can't cast close votes. So while these people may not be robo reviewers, they aren't helping with closing the questions more quickly (as they can't cast a vote) and instead effectively increasing the size of the close queue (while at the same time removing the first post from the queue). And no, the answer to that isn't make first post queue a 3k limit.
    – user289086
    Commented Nov 22, 2013 at 23:27
  • 2
    @MichaelT The point of all my "To fix #0" ideas are not to close questions more quickly, but rather to, over time, reduce the number of new questions that need to be closed (that one specifically is intended to provide more guidance to new users as opposed to simply trying to close their questions). If we can do this, we should fairly quickly be able to clear out the close vote queue at the rate we're currently processing items from it (and, ironically, the rate is likely to increase as the queue starts to reduce in size). Commented Nov 22, 2013 at 23:34
  • 1
    @Shog9 On #0, I believe I've pointed out two big problems in my recent Meta posts: non-native speakers with poor English who don't read the /about stuff and so on, because it's written in English and they have trouble with English, and Jeff Atwood's refusal to countenance more noticeable warnings triggered by tags or key words. And yes, I do think the latter problem is the more serious one. Since Jeff has left, I suppose what he has decreed in the past has no binding effect on you now? Commented Nov 23, 2013 at 14:43
  • 1
    Do you have any idea what changed in July to cause the deficit to expand sharply the way it did? Was there a change in the code, or just extreme user apathy?
    – Bill the Lizard Mod
    Commented Nov 23, 2013 at 16:39
  • 2
    Yes - we changed all the close reasons and redirected flags at the end of July. Both contributed to an increase in the queue (voting increased, flagging went down but wasn't handled as quickly). @Bill.
    – Shog9
    Commented Nov 23, 2013 at 16:45
  • 3
    There is another problem there. The more you close, the more likely you are to get blocked for failing badly chosen reviews. It's still an open issue, discouraging people from reviewing: meta.stackexchange.com/questions/168374/… Commented Dec 11, 2013 at 8:24

14 Answers 14

31

Below request for data has been posted per discussion with Shog in comments

I think having more data about reviewers would help to better understand what goes on.

Specifically, it would be useful to see sort of "histogram" showing how many reviewers performed certain amount of reviews, like 2000 did 1 or more, 1000 did 10 or more, 100 did 100 or more reviews etc.

users grouped according to the number of reviews they've done

Raw data (if you don't like the sizes I chose for my bins)

Sharp drop at 10-50 likely indicates that many users find it difficult to work in review queue. Drop after 250 (silver badge) suggests that even after substantial amount of reviews, many users still fail to discover a way to work productively (filtering 1, 2, 3). About half of ~500 users who have gotten golden badge at 1000 reviews indicated an interest to continue using queue even without badges.

9
  • 5
    I like histogram. The word. Oh; and the thing.
    – Andrew Barber Mod
    Commented Nov 29, 2013 at 19:28
  • 4
    I played with the size of the bins a bit to try to get a clearer picture jsbin.com/oVeHiYUZ/4 its a little depressing...
    – apaul
    Commented Nov 30, 2013 at 1:11
  • 1
    Just guessing, there are badges at 250 and 1000. Commented Nov 30, 2013 at 1:22
  • 5
    Review queue has poor interface, is the most buggy part, working poor on older browsers, and review audits are chosen at random, many of them being the opposite of good audit. Diagram is not suprising. Commented Dec 11, 2013 at 8:27
  • 6
    I don't find it really difficult to work with the queue, its just that well, the whole task of reviewing stuff is really boring, and I just do it when in the right mood, and that wears of quickly. So would see the sharp drop at 10-50 as that: people are bored by reviewing.
    – PlasmaHH
    Commented Dec 11, 2013 at 20:49
  • 1
    Where did you pulled this data from? SEDE?
    – Braiam
    Commented Jun 4, 2014 at 18:36
  • 1
    @Braiam I "pulled it" from Shog. Ask some 10Ker at MSE to check revisions history of original post, it was inserted at rev 2 or 3 by Shog, just like similar data in my other "answers" here. I merely provided a placeholder for it (our discussion about this is probably still buried along comments here in the question or answers:)
    – gnat
    Commented Jun 4, 2014 at 18:43
  • 1
    I wouldn't read too much into the drop on the histogram. It is likely due to bin size being too wide. This follows a pretty good exponential distribution, which is exactly what you would expect in this type of data. I would be concerned if it did not look like this.
    – theMayer
    Commented Jan 13, 2015 at 19:17
  • @theMayer one can check how it looks with smaller bin values at jsbin.com/oVeHiYUZ/1 - I did just that before settling on this histogram
    – gnat
    Commented Jan 13, 2015 at 19:44
27

I agree with the real problem: people are voting to close questions, and their votes are hanging around for months - even years - but the questions aren't getting closed. And I believe you really nailed it in a comment:

[...] closing must be fast - and much as I hate to admit it, that ship may have sailed. Still, when it works it's pretty useful - so worth looking for a solution, even if it only helps a fraction of the questions.

That's what I see on SO, closure works well when it happens fast. But the mechanisms to deal with questions not closed fast – vote aging, and the review queue itself – don't work well enough. Aging is sort of invisible, but the queue is the opposite of that, it stands there in the top of the /review page, as if it were the most important thing reviewers should be working on. In my opinion, it isn't – because it's not manageable, and I'm not sure if it will ever be.

I've been thinking, if fast closure works well, why don't we focus on that? Make all close votes expire after a certain time, regardless of the number of views. If a question gets closed fast, great. If it's not, it's removed from the queue after some time. If later it's noticed again and someone votes to close, it comes back to the queue, so it gets another chance of being closed fast. Unlimited chances could be allowed. This implicitly admits that we'll always have a certain volume of crap hanging around, but don't we currently have that anyway?

2
  • 16
    if close votes should expire this fast, can we at least have the ability to recast them? It's kinda stupid if five or more people see a recently closed question, all of whom have their votes expired because they had voted before. Commented Nov 24, 2013 at 7:45
  • 4
    I agree, @Jan. We should be able to recast them as many times as necessary.
    – bfavaretto
    Commented Nov 24, 2013 at 19:29
22

For the most part, I agree with shog9's problem statement.

I don't know the statistics on this, but perhaps one small way to help would be to add notifications for when a question you've voted to close is edited. That way you have a more convenient opportunity to go back and retract your vote. This would only apply to questions that are in the review queue, not to questions that have already been closed.

The thought here is that I feel a lot of people, myself included, tend to vote to close questions shortly after they are posted -- e.g. when looking for new unanswered questions. However, questions tend to be very active during that period right after they are posted, meaning other users will probably leave comments like "What are you asking?" or "Please include X" which prompt the poster to clean up the question.

The idea is that since questions frequently get close votes when they are new, but since there's a high chance of the question being improved during this same time period, a notification would draw your attention back to the question and give you a chance to retract that vote.

One potential problem: This is also the time period when "minor" edits such as code formatting and tag/title cleanup are occurring. Since those edits would produce notifications too, users may tend to get irritated at receiving them and learn to ignore them or come here to complain about them.

However, I think a solution for that is that the notifications should only be sent if the edit is made by the original author, and if the edit makes a sufficiently large change (in terms of number of characters edited).

I don't think it's a problem that you can't vote to close again after your vote has been retracted. I can't conceive of a situation where you would vote to close a question, then retract it because an edit was made to make the question valid, then for whatever reason decide the question should be closed again. So I don't see that as a downside.

This wouldn't help clear existing items out of the queue, but I think it will increase the chances of new items not sticking around, and so help slow new queue growth a bit.

Not sure how much this would help. I feel that I would find it useful -- but again I don't have any statistical information on just how many bad questions are actually improved to be valid early on. It might be worth implementing this as an experiment, if anything, to see how much of an impact it would have and gather some more data points.

5
  • 1
    The other option is people stop being so trigger happy on the close vote. If you know new questions from new users have a good chance of improving in the first 12hrs after they are asked then leave them alone and let them improve.
    – tacaswell
    Commented Feb 17, 2014 at 23:39
  • 1
    I see. Do you feel that it is easier to implement a UI feature on SO vs. commanding all people to stop being trigger happy on the vote and having them obey your command? Do you feel the latter is a legitimate "option", as in you or I get to choose to modify the behavior of all people as a feasible alternative to the above? If so, I suggest we start at scifi-real.com/mind-control-weapons.
    – Jason C
    Commented Feb 17, 2014 at 23:47
  • It is part of the push to be friendlier to newer users (which is one of the themes that is showing up in the moderator comments). There are all sorts of discussions about the tone of the community and how we want new users to behave. No reason we can't have the same discussion about high-rep users as well.
    – tacaswell
    Commented Feb 17, 2014 at 23:55
  • Well sure we can discuss it until we're blue in the face. The fraction of users who read those discussions, the fraction of those people who take them to heart, the fraction of those people that change their behavior, and the fraction of the previous group who change their behavior permanently are sure to make a huge dent. OR we could just opt to set up an environment that naturally encourages users to behave that way instead. Or both! It's like having a big red button that nobody is allowed to push. You could either convince everybody not to push it, or just not have the button.
    – Jason C
    Commented Feb 17, 2014 at 23:59
  • It's more like a credit card, it can be very useful, but if used improperly can be catastrophic.
    – tacaswell
    Commented Feb 18, 2014 at 0:07
21

Tag-oriented analysis would be terribly incomplete (up to the point of total failure) without a corresponding data breakdown by closure reasons.

I mean, say, if a question is like "recommend me a library / tutorial", it doesn't really matter if it's or , because it falls straight under simple, standard, tag-agnostic close reason:

Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam...

Particular tag may be technically complicated / obscure, or it may have a lot of questions in queue, but without knowing breakdown by close reasons, it doesn't make sense to believe that it makes a trouble in a CV queue.

Say, if 99% queued questions in this tag are resource-requests, it would be only a matter of getting more reviewers / reviews to clean it up, straight and simple.

Note for the case if it turns a matter of getting more reviewers / reviews, it would be also useful to get the data that could help analysing the situation. Some stats that come to mind are:

  • how many users are eligible to work in queue? (that one is easy to estimate: about 15,000 users)
  • how many users have at least once tried the queue?
  • how many users have demonstrated a strong interest, by performing substantial amount of reviews?
  • how many users have proven ability to perform a lot of reviews?

Three latter data points can be roughly estimated by amount of users having respective Custodian, Reviewer, Steward badges in the queue. Quite a pity that badges pages are useless in helping to find this out because of indiscriminately counting badge holders in all queues.


Side note - as far as I can tell, tag-oriented analysis currently looks for the long tail in the wrong place. An example of what I would qualify as true long tail, would be not questions in particular tags (indiscriminately aggregated, no matter what's the close reason), but duplicates. Per my experience, duplicates are tough and take much effort, even in the tags I am well familiar with. Maybe there are other "difficult" close reasons besides duplicates - I can imagine minimal understanding, or too broad, or unclear being tricky, especially in tags I'm not familiar with.

15

Proposal:

Allow lower-rep users to review simpler close-votes

If more users can review CVs, then more reviews can be completed in the same timeframe. Some close vote reviews are simpler than others, particularly the "recommendation" and "no repro code" off-topic reasons.

Users with a lower reputation threshold should be able to evaluate "recommendation" and "no repro code" CVs. 500 rep allows a user to review first posts and late answers, so these users could help with the some CV options. Of course, these CV reasons will be subject to audits just like any other part of the review queue.

1
  • Good candidates: "no repro code" or "no attempted solution" -- Off-Topic subtypes (that are not really "Off-Topic", anyway). (CodyGray agreed, in the first comment at APaul's question: meta.stackexchange.com/questions/191540/… .)
    – Doug_Ivison
    Commented Jan 13, 2014 at 19:04
15

tl;dr Apply Lean principles to this problem

When you have a "system" like SO with lots of circular feedback mechanisms, I think it's hard to agree on a single "end goal". That said, I very much agree with the focus of Shog9's post, which I look at as timely and appropriate resolution of the per-item close process, one way or another.

Similarly, I completely agree that our priority ought to be on dealing with the steady state process and ensuring that we can readily handle it. I view this through the lens of "lean manufacturing" and think this Womack article is highly relevant. Rather than first tackling the obvious "muda" (waste) (e.g. the backlog), we ought to be dealing first with "mura" (unevenness of operations) and then "muri" (overburden). Only after we've set up a system to handle the incoming flow reliably without burdening our people should we be tackling the backlog

While I have opinions like everyone else as to what structural changes ought to be introduced to get us to the point where we are reliably handling the incoming flow, let me just say that I think other Lean principles apply, as summarized in http://www.craiglarman.com/wiki/downloads/scaling_lean/toyota-house-larman-vodde.pdf.

For example, "Simple visual management" would seem to be particularly critical in getting our arms around this problem. Shog9 and others have done good work in creating charts that helps us understand the situation better. Others have submitted good ideas for making visible to users the current state of the input queue that they are interested in tackling.

Another potentially helpful Lean principle to leverage, I think, would be use of teams and tag-oriented teams in particular. Not only is the close process more efficient and effective when participants are technically knowledgable, but there already is a sense of "ownership" for various tags within the community. We should leverage that.

One final note is that I think it's important to remember that the actual incoming flow of questions-that-people-want-to-close may actually be much higher than the flow of questions into the close-vote review queue, since many people are undoubtedly not voting-to-close in the current situation. That's yet another reason to address the steady state first.

13

Proposal for prevention:

Question template(s)

The purpose here is to DECREASE the portion of the close review queue, that is due to low-quality questions, by making common requirements clearer.

Like, instead of one question box, provide separate boxes for:

1) The goal:  what is to be accomplished. 
2) Platform, development language/environment/tools, operating system, all w/ versions.
3) Initial work -- tried so far. 
4) Sample code. 
5) Results (data). 
6) Messages (errors/warnings). 
7) Resolution work -- tried so far.
8) Questions -- anything you'd like more information on -- details, principle or practice.
9) Ideas -- where you think it might need to go next. 

Some boxes might be required, but in any case make it clear to the user that better use of the boxes speeds up the process of helping them... and more empty boxes can lead to a question being put on hold.

THEN, this will also help speed up the review of the close queue -- it will be quicker/easier to IDENTIFY gaps/issues, to EXPLAIN the reason for the close, and at review time, to DISCERN if it's a valid close.

Aside:
Consider possibly taking questions on hold as low-quality questions OUT of the queue of visible questions.... and when certain boxes (or enough boxes) are not filled in, possibly AUTOMATING the close review.

If the template idea seems too restrictive, they might select "Freeform text or Template" with a radio button... but again, communicating to them that the template can make help more likely, and faster.

Thanks,
---Doug

P.S./edit: A freeform text box would probably still be helpful, after the template boxes. Also, it might even help to have "edited" notifications -- who and when -- as now, next to each template box.

10
  • 1
    Would be better to have different templates for different tags or types of questions, but that opens up a potentially enormous redesign can of worms. Commented Nov 23, 2013 at 14:57
  • 1
    @RobinGreen (1) A generic define-the-problem template, like the above, could be used initially... and tag-specific tweaks (a great idea) could come later. (2) The design changes would be much smaller, if mostly at question creation time -- like, after the template has been used to CREATE the question, the resulting posted question could still be simple formatted text.
    – Doug_Ivison
    Commented Nov 23, 2013 at 20:55
  • 1
    I think having "insert template" button at the new question form would be trivial to implement, and could easily have a few different templates, like "My code does not compile", "My code crashes", "My code gives wrong results" and "I need help solving an other problem".
    – hyde
    Commented Nov 24, 2013 at 10:49
  • @hyde Yes, I see how that also could be valuable -- in addition to question structures (what I described -- spaces), to also have answer structures (what you described -- stuff in the spaces).
    – Doug_Ivison
    Commented Nov 24, 2013 at 13:31
  • 7
    Interesting idea, but have you taken (say) 20 good existing questions to see if they would fit your template?
    – Benjol
    Commented Nov 25, 2013 at 6:53
  • 1
    Interesting. Should there be an item in the list for 'platform — operating system and software versions'? Commented Dec 2, 2013 at 6:00
  • @JonathanLeffler Thanks... GREAT idea... yup, ran into that omission in the past 24 hours... (I'm hearing SNL characters saying "Don't you hate it when that happens?...") I added it as point#2, and bumped up the numbers after that.
    – Doug_Ivison
    Commented Dec 4, 2013 at 18:03
  • 1
    A variant on this idea could be a "Ask a question" wizard, which could guide people towards asking a 'good' question. Though whether any of the "I can't be bothered to read anyway" users would be helped is a moot point...
    – Benjol
    Commented Dec 9, 2013 at 9:05
  • @Benjol I haven't yet done the due diligence analysis (20 examples) you recommended, though a quick review of some good questions shows that some questions about programming language characteristics are very highly voted, even though they do not have a "my specific problem" format at all. So, my suggestion could have drawbacks... and I wonder if incentives would work better than requirements: for example, tell first-time-posters they will get FOUR POINTS (or maybe even abilities, like comment-leaving) for a question that follows the template, instead of 2 points.
    – Doug_Ivison
    Commented Dec 9, 2013 at 15:16
  • @Benjol Re' who would skip a template: I do think many of us have a "jump in and try it" attitude, that's helpful in this profession. For that approach, "reading instructions" sometimes seems to slow down productivity, whereas "script-generators" (or templates) can speed things up. So yes, there are those who "can't be bothered", but also those who would see it as a tool. Also, I think that the process of breaking-the-question down, in the template above, would actually lead to some people answering their own question, through the clarification steps the template calls for.
    – Doug_Ivison
    Commented Dec 9, 2013 at 15:19
12

Proposal:

Allow filtering by CV reasons inside Off-Topic

There are sub-reasons of Off Topic for closing a question

  • general computing hardware and software question
  • no repro code
  • no attempted solution and expected results
  • server or networking infrastructure question
  • recommendation question
  • leave a comment

A user can more quickly complete CV review tasks by refining them to focus on one specific off-topic reason.

2
  • Entries into the CV queue that come from flags don't appear to have sub-reasons associated with them (and its been commented that many times, flaggers don't have a good idea of what is off topic or not). Btw, you forgot the migration routes as a sub off topic reason.
    – user289086
    Commented Nov 22, 2013 at 23:20
  • 1
    meta.stackexchange.com/questions/191540/…
    – apaul
    Commented Nov 23, 2013 at 2:08
11

Below request for data has been posted per discussion with Shog in comments

I think having more data would help to better understand what goes on. Specifically, it would be useful to see how the queue is split by close reasons (preferably including split by off-topic sub-reasons)

Breakdown by close reason and OT reason

Raw data / chance to show off your D3 skills

This is a bit misleading, since a given post might have more than one distinct close reason attached to it; here's a rather quick and dirty visualization to demonstrate:

euler diagram

script to generate the above image, containing link to raw data

6
  • 1
    @Shog9 33K dupes + 25K minimal-understanding seem to support your focus on tag-oriented analysis, working with these is harder without relevant tag experience. Also...
    – gnat
    Commented Nov 30, 2013 at 8:34
  • 4
    ...61K off-topic items look like 61K upvotes for Close Vote Review, additionally filter off-topic close reasons
    – gnat
    Commented Nov 30, 2013 at 8:35
  • 1
    I suspected that off-topic would would take the cake, but not by such a wide margin... Thanks for the data.
    – apaul
    Commented Nov 30, 2013 at 15:14
  • 1
    This is all a bit misleading, I'm afraid... You should've asked for a Venn diagram
    – Shog9
    Commented Nov 30, 2013 at 15:17
  • @Shog9 yeah, the way it's presented now, readers need to permanently keep in mind that there are intersections. Venn diagram would probably make it easier to grok... but somehow I still prefer that plain list, even though it involves additional mental adjustment
    – gnat
    Commented Nov 30, 2013 at 15:33
  • @apaul34208 The diagram is great. Ironically, the use of "off-topic" itself sometimes seems "too broad": things get marked as off-topic, for being OFF-STYLE, even when the topic (content) is within scope. It gets used when reasons such as "too broad", "too localized", or "primarily opinion-based" fit, and would be more informative. It seems to me that off-topic is a good description for content outside of scope... otherwise, use another reason.
    – Doug_Ivison
    Commented Dec 24, 2013 at 18:07
7

I have suggested this before in relation to edits, and I think the idea, though it didn't seem to be well-received there, sounds good to me here too.

The PHP tag, with the most questions in the review queue, has a "close vote posse" (I don't like the term myself). There's a script and a backlog page. And we're still buried.

The "cv-ring", as we call it, is usually a pretty effective process, though it seems to make some people uncomfortable. I think it is effective; the people in the room know their stuff, so when ircmaxell, for example, posts a question that needs to be closed, I can be reasonably sure the question warrants closing. So I pop over to the question, read it over. 90% of the time, I agree with ircmaxell's assessment and vote to close. I might post a comment. If there are 3 other people online who are game, the process happens pretty quickly, and I view that as a success.

We could skip this process if we gave users with experience in a given area more weight when they evaluate a question. And the great thing is, we already know who has experience in a given topic, because we have a fairly deep bank of statistics regarding their adeptness at answering questions in any given tag.

Had ircmaxell, with a gold PHP badge and 9 Great Answers in the tag, been thus empowered, the chat room stage and the waiting for 4 other users and the PHP cv backlog are all unnecessary, as is the queue deficit. Ircmaxell can just take care of it. Or me, with a silver badge, and one other silver badge-holder. Or 3 bronze badge-holders, or 5 non-badged users. Something to that effect.

We have evidence of, if not expertise, at least adeptness at answering questions in a given subject. People who have shown they can extract meaning from a question, and a willingness to try and provide an answer. They aren't here to close questions if they're answering, and answering well. Why not tap into that, and more importantly, trust it?

7

You have to incentivize review.

I don't see a way around it, and I also don't see why not. You're rewarded to make high quality contributions. Shouldn't you be rewarded for keeping the low quality contribution out?

Something symbolic like +3 on successful closure (i.e. 4 other people or a moderator agreed with you) should get people involved. I'm aware it isn't perfect, and that sometimes good questions are wrongly closed. But we're talking about mainstream here, not edge cases. (We can revoke the points if the question gets reopened for example).

Also, regardless of that:

  • The tag filtering and close reason UI must become more obvious. I didn't know about the feature until someone told me about it.
  • Duplicate question handling isn't easy right now. It's sometimes not clear which is a duplicate of which. Also, introducing system recognized canonical questions can help.
6
  • 4
    @Shog9: Well, the site is built on incentives in the first place. Sure, most regulars end up not doing it for the rep, but for fun/education/self fulfillment/etc. But if you really want to attract people, not just allow the existing ones to do a better job, you need to gives some sort of incentive. That's just how I see it. Commented Feb 11, 2014 at 20:58
  • When we had the Winter hat thing going and people were once again rewarded to review to get a hat, I saw the queue drop by ~10K, keep in mind this is in addition to offsetting the growth. The moment it was over, the queue once again began to grow. Maybe it was a fluke, but Madara Uchiha is right, people need to be rewarded or they lose interest in doing it. Even if you don't want to give reputation, then at least make the "one-time" review badges no longer one-time and that will make people review to earn that next gold badge.
    – cryptic ツ
    Commented Feb 11, 2014 at 21:04
  • I saw the queue drop too, @crypticツ. I also saw a bunch of crazy flags, questions closed that didn't need to be, pointless edits, outright fraud and other assorted nonsense. The hat thing is fun, but if you wanted an example of how extrinsic motivation can make folks do stupid things without caring about the outcome, you could do worse.
    – Shog9
    Commented Feb 11, 2014 at 22:46
  • 2
    There are a few badges, @Madara. Generally, that's small enough of an incentive to be rewarding without overly-motivating - although for a handful of people they still manage to override any sane notion of "I'm here to make the site a better place". Incentives like this aren't some magic fairy dust that makes people do what you want without any down-sides: if you make something into a game, folks will treat it like one and damn the collateral damage. Please read: xoxco.com/clickable/i-love-my-chicken-wire-mommy
    – Shog9
    Commented Feb 11, 2014 at 22:52
  • @Shog9 Well comparing giving another gold badge to giving a hat is not the same thing. To get the hat you had to review a few items. But to get to the next gold badge you would have to review 1,000. Not something fraudsters or people wanting a quick reward would so easily want to tackle as the time and effort involved is much greater.
    – cryptic ツ
    Commented Feb 16, 2014 at 9:04
6

Shog9 said:

  • Some questions are being closed years after being asked, and often after attracting perfectly good answers whose very existence casts doubt on the need for closure.

Some of these, at least, are questions which are duplicates of other questions of similar or greater age, also with perfectly good answers. The answers to the question marked for closure say much the same thing.

Should those questions be tagged for closure?

If not, should I have stopped reading some time ago?

8
  • 6
    Duplicates are, in many cases at least, an entirely separate beast from the rest of the questions nominated for closure. They really need their own queue, or at least a more direct way of filtering.
    – Shog9
    Commented Nov 23, 2013 at 3:37
  • I'd go with 'method for filtering' or 'separate queue' — either would work for me. I suspect they're a fairly small percentage of the queue anyway, though not non-existent. Commented Nov 23, 2013 at 3:40
  • 21.7K out of 95.5k - not insubstantial.
    – Shog9
    Commented Nov 23, 2013 at 3:43
  • 5
    Wow — I am surprised; indeed, not insubstantial. Breaking them out into a separate queue would at least give two separate measures of progress (or allow dedicated people to work on twice as many questions a day by virtue of them being separate queues). Commented Nov 23, 2013 at 3:45
  • @Shog9 I probably won't vote to close a question as a duplicate once it has an interesting answer b/c I have seen highly voted duplicate questions deleted sometimes for no other perceivable reason than it was a highly voted and so that has tainted my view on this aspect of closing. Which is perverse b/c they should be marked as a duplicate but it won't get a close vote from me unless it has some other problem that makes it bad and then I will vote to close on that reason. Commented Dec 13, 2013 at 13:48
  • 1
    When you see a duplicate question with a good answer, @Shafik, flag it for moderator attention and recommend merging with the original (include a link). This'll not only preserve the answer, it'll preserve the question as a "signpost" pointing to all of the answers that address the same problem.
    – Shog9
    Commented Dec 13, 2013 at 16:48
  • @Shog9: One more advantage of a separate Duplicates queue could be that it could support a 'merge answers' check box to automate the flagging for moderation attention. Commented Dec 13, 2013 at 17:13
  • @Shog9 from what I understand most dups are not good candidates for merging, so good answers can be in bad merge candidates. Looking into some topics I had to look at many dups sometimes to find all the bits I needed and many times a good answer was one in many really blah answers, so I am not going to flag that thread as a dup b/c I won't be able to search for it if it is deleted later on. I have requested merges before but it is opaque and takes a long time, most are good but the last one I have no idea why it was rejected, what criteria is used or even why the original question was deleted. Commented Dec 13, 2013 at 22:13
6

I've no idea why so much a fuss about closures. And especially about the queue.

Closing a question does not a slightest good to the site. Even if you manage to solve a long queue problem, you won't help site at all:

  • bad question always gets an answer faster than you can close it. Means no harm neither for the OP nor rep-whunter.
  • even closed, it still hangs around.
  • worst part: only 1 out of 10 really bad questions manages to get a vote. This is a real problem.

Out of good site you made just a game with scores and badges. It attracts gamers, not programmers. Most participants are after only rep and badges, not sharing knowledge. As long as this status quo persists, anything you do will fail. Your motivation system is rotten.

Rep'n'badges do not encourage writing good answers for complex questions.

Only desire to share a knowledge do. If one wants to share - they'd share without no badges.

While reputation IS the thing that encourage writing fast answers for lame questions. As long as your only concern is rep points, you'd like as much lame questions as you can answer. And most silly questions you'd like most - for they take you no time to copy and paste from other answer or a manual page.

And it creates so strong positive feedback-based circuit that it never even notice none of your silly struggle with closures.

3
  • 10
    I don't really disagree with any of this; in all honesty, I don't lose any sleep over the existence of a backlog. I would really like closing to be a tool for the folks answering questions, something that could remove half-assed questions from play before they collected half-assed answers, and restored when a reasonably useful answer can be provided. But for this to work, closing must be fast - and much as I hate to admit it, that ship may have sailed. Still, when it works it's pretty useful - so worth looking for a solution, even if it only helps a fraction of the questions.
    – Shog9
    Commented Nov 23, 2013 at 3:34
  • 5
    I disagree that closing questions doesn't help the site. Closed questions are (very probably) factored into the question ban algorithm. There are many very bad questions which are closed quickly enough to prevent answers.
    – KatieK
    Commented Nov 23, 2013 at 22:30
  • @Shog9: I don't think the ship has sailed.. that's quite a solution you have actually. Commented Nov 27, 2013 at 13:16
5

I have only one quasi-quibble to offer. I don't see how to distinguish a legitimate close vote that languishes in the giant tide from an invalid one. If I vote to close a stinker, and my vote just hangs out there in the queue, is it because I was having a grumpy day, or because the inflow rate is such that the question is already lost in the sauce?

In the 10k pages we used before the review queues, there was something of an immediate community feeling: I could visit and see what other people had nominated for close, and I could quickly join in when I agreed. Maybe one of these subdivision of work ideas will recreate that atmosphere.

Maybe the flag business was a mistake. We now have a ton of people with enough rep to out and out vote. Perhaps allowing people below that rep level to suggest closures via flags is adding more noise than signal?

One idea I have is this: by all means, time out close votes. Just time out the 'you already voted to close this question' with them. Most of the time this will be uninteresting, but every so often it will allow some of us a second attempt to clean up actual trash.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .