The size of the backlog in Stack Overflow's close review queue has been an issue for a long time now:
I find that having such a huge backlog, makes it very hard to properly deal with closures.
What steps or system changes can we make to get the huge close backlog on Stack Overflow, under control?
The additional visibility granted to this backlog by the review system has sparked a lot of discussion here recently. The problem with most of these discussions is that they don't agree on what the problem to be solved is... Other than that it must somehow involve the existence of a backlog.
In order to understand the problem, we first have to agree on a goal. Here's what I think the goal should be:
When you vote to close a question, if you've correctly identified a serious problem with the question and the problem isn't corrected, the question should be closed. And if you haven't, your vote should go away.
If you don't agree with that, you should probably stop reading here... But I'm hoping this is something we can agree on; that even if /review didn't exist, this would still be a good baseline expectation for close-voting. We've built up an awful lot of behavior that depends on this assumed goal.
We can further refine that goal a bit; ideally,
Problematic questions should be closed quickly - if it's a real problem, letting it hang around stinking up the place isn't doing anyone any favors, and even conscientious users are less likely to put time into correcting them long after they've been posted.
Questions that aren't causing problems shouldn't have close votes hanging off of them, like some terrible blade threatening to drop at any time.
If we're in agreement on the goal, then I think we can all agree on the real problem:
People are voting to close questions, and their votes are hanging around for months - even years - but the questions aren't getting closed.
As with the goal statement, we can refine this by observing that:
Some lousy questions are hanging around for long periods of time, clogging up search results and with little or no feedback given to the asker.
Some questions are being closed years after being asked, and often after attracting perfectly good answers whose very existence casts doubt on the need for closure.
Again, if you agree with the goal you should agree with this statement of the problem; if you don't, then you should've stopped reading already. Stop wasting time you could be using to tell me how wrong I am!
And again, this isn't specific to /review - it's been a problem ever since we instituted the 100-view requirement for close-vote aging, at which point it largely supplanted the previous problem: close votes expiring before anyone else who could close had seen them.
So now...
About review... A deficit and a long, long tail
Here's a picture:
You see the problem, right? No, it's not the 57K backlog we started with; there are only 11K or so of those left. It's that we've only been "in the black" for 5 months out of the last 16. When most days - most months - end with a deficit, you never pay down the debt you started with... And eventually, folks get discouraged.
There are a bunch of ideas for addressing this problem out there, some of them good, some of them awful. I'm not going to discuss most of them tonight, though I'll try to do a better job of at least commenting on them in the next few weeks. The goal of this post is to try and focus everyone on the same problem, so I'm trying to keep it relatively brief... so to save on words, here's one more picture:
Don't worry, you're not supposed to be able to read the text on that; just look at the shape. The single largest block of questions in the queue right now is attached to the php tag, amounting to about 10% of the entire queue. Fast on its heels is javascript, at which point it starts to flatten out really fast. By the time we hit wordpress, we're at 1% of the queue.
This is an opportunity...
See, the problem with a lot of the suggestions floating around here right now is that they make a couple of shaky assumptions:
- Most of the questions in the queue actually need to be closed.
- A handful of people working REALLY HARD could close them all in no time, if we just gave them more privileges / required fewer close votes / skipped the whole "review" thing and just closed them all automatically / etc.
#1 I'm just not seeing. Oh, for sure there's a lot of crap in there... But there's also a lot of stuff that's just in the queue because someone didn't know what the hell they were looking at and decided to flag it, or thought "minimal understanding" meant "already solved the problem and is just posting here for typing practice". Especially once you get outside the PHP tag.
#2 is true in theory, but... We've kinda been trying to move away from that - the big hope for review was that it would empower folks enough that we wouldn't need 15 moderators closing stuff all day long to keep up. And the truth is, it's a lot harder to review stuff when you don't know jack about the topic. I'm pretty comfortable in winapi, but pretty much everything in heroku makes me suspect the author was drinking heavily when he wrote it; trying to distinguish bad from just weird is taxing... And, I'm fairly certain, less accurate.
I'm going to try and use this thread as sort of a scratch-pad for tracking changes as we propose, discuss and implement them, as a way of staying focused. If you disagree with the goal (or the problem) statements above, please discuss them in answers here, but save discussion on the proposed solutions for their own threads.
Progress so far
A couple of important changes have been made:
Allow direct linking to a filtered /review queue has enabled folks to jump directly into their tag of choice when reviewing.
https://meta.stackexchange.com/questions/199879/fuzzy-the-number-of-questions-in-the-close-review-queue-a-dopamine-for-the-shut/209774#209774 has reduced the visible backlog, helping to focus efforts on questions most likely to be closed or left open.
Coupled with these, we has a bit of a shindig to try and get more folks involved in the process: https://meta.stackexchange.com/questions/223497/lets-burn-down-the-close-queue
The results so far have been promising. Here's a graph of the number of active reviewers by week:
And here's a variation on the "deficit" graph above, this time comparing questions collecting their first close vote or flag to those where the last vote or flag was dismissed (by either closing or aging):
It's not all roses though; the average time between the first vote and completion of review for reviewed questions flagged in is 28 days - up from 12 days a month ago. We have to assume the bulk of this is due to the work being done tearing down the backlog, but we'll want to keep an eye on this to make sure we're not dramatically delaying closures (and thus depriving askers of timely guidance) long-term.
Action items
We need to continue this two-pronged attack here: increase the number of active reviewers, and focus their efforts on areas most likely to benefit. I believe the next steps in these directions are:
Make it more obvious that the queue can be broken down by interest, highlighting areas mostly likely to benefit from the reviewer's attention.
We've seen the effect that bite-sized pieces can have: when the progress you're making is visible, it's much more gratifying to keep making progress. Let's give folks the ability to observe their progress through a specific area of the queue.
Keep up the good work!