Update: SE is looking at feature requests now! It's hopefully just a matter of time before review/moderation-related ones start being worked on.
Background
On MSE, there are ~700 unimplemented, review-related feature requests. In general, there's over 13k feature requests without any [status-*]
tags on MSE. On SO, there's nearly 4000 overall, with roughly 200 aimed at review. Those are just using the generic tags.
The main problem, as far as I see, is that there is a ton of stuff that needs to be moderated, but way too few votes and active reviewers to do so. There's a lot of incorrect reviews, and not to forget various "fine prints" on editing and the common plagiarism pitfall with tag wikis that cause additional work and reviewing reviewers.
Duplicate CVs are specifically tricky because of the domain expertise some of them require. Flags aging away is also a problem, but one that is specifically visible to users who flag. Close votes expiring isn't as visible.
The problem is better demonstrated in this post - the queue size fluctuates, but never goes specifically low.
The current situation
The burn of the close vote review queue shows we have the technical amount of people required to actually pull it off again. But what one of the answers unfortunately shows is a drop in reviewers after a review spike. At its peak, there were 12000-14000 reviews per day. So far today, there has been 658. To handle the 8152 pending questions in the queue right now, over 40000 reviews are required, or roughly 800 reviewers with all 50 votes.
There is, however, one problem with this calculation: the number isn't real. SE has an algorithm (that I don't know the exact details of, and therefore can't reverse) which reduces the number of questions that appear to be in the queue.. The real number of reviews required to keep the queue down is likely exponentially higher, further increasing the need for review enhancements.
However, the queue isn't sitting there static, and the site isn't blocking users from asking new questions. If a full-scale strike is put into action, the amount of posts in review will spike significantly.
The problem
We don't have enough reviews.
And this is a three-part problem:
- The reviewers
- The review system
- The posters
This question asks for a strike in light of the review system. However, if we don't have reviewers, we still won't get enough reviews. If we lower the close votes required to close a question to three, the amount of reviews drops to 24000, or nearly cut in half.
Which brings us to the last problem: The posters. This is being worked on through things such as the wizard, but this only covers questions. Answers are more complicated, but bots such as Natty are helping flaggers with that. The major difference between VLQ review and close vote review queue is 1000 reputation.
Talking in user count, 0.9% of users have access to VLQ, but only 0.65% have access to CV review. (source). How many of those are active though? That's a very different question, and one I can't answer accurately through SEDE. But given the size differences, even though there are volume differences in questions and answers, the answer review queues are very close to empty.
Based on today's (17.05.19 at 19:55, +0 GMT) 658 reviews, and for the sake of math and missing information I will have to assume all the reviewers have burned through their reviews. Doing so gives 658 / 40, which means 16 reviewers, and one who used about 20 reviews would be able to keep the exact count listed earlier. And, again, there were near 800 during the CV queue burn.
At the same time, the queues are filling up, and it will get worse when there are fewer close voters. The strike is barely starting up, so to what extent it affects the review queues is yet to be seen.
However, I do feel I need to give a shoutout to the moderators, and especially the one I've seen most of, Bhargav Rao, who've made a noticeable impact on the VLQ review (and indirectly on other answer-related review queues) by handling NAA flags. Without the moderators, the VLQ queue would likely contain more posts than currently does.
And if they, as Yvette mentioned, are trying to tackle the close vote queue as well, that's on one hand good news (because of binding votes - which means more effective reviews), but on the other hand, additional proof the CV queue has fundamental design flaws when it comes to efficiency. While it's most noticeable in the CV queue, it wouldn't surprise me if this affects other review queues some day, or with the strike, soon.
TL;DR: We don't have enough reviews per day.
Secondary problem
Aside the lack of reviews compared to what's required for an efficient system, there's one pretty big problem affecting the queues.
As listed in the start, there are ~900 unimplemented review feature requests on MSO and MSE combined (possibly with some overlap). These not being implemented likely contributed to this.
I've talked to some people who have stopped reviewing because of missing features, or otherwise have issues with the review system - and in my opinion, with good reason.
Statistics
Update: These numbers are wrong. Without access to the close votes themselves, I wrote the query under the (wrong) assumption posts closed outside the queue completed the review task. According to Shog, the number is closer to 42%, which I still think is too high. On the bright side, it wasn't as bad as I thought.
Update: These stats cannot be corrected without Community Managers or other people with database access. Accurate statistics will eventually be available here
Update: The accurate numbers are between 10-20%. See this answer.
The original state for historical purposes:
Note: I am not sure how the fuzzied numbers affect SEDE
I get that it's hard to see the problem. I've created a SEDE query (fork-ish off this query) to better demonstrate this (thanks to rene for helping me with sanity-checking the query):
The question and answer post stats is just to demonstrate the volume. On a weekly basis, there's (on average) close to 50000-60000 of each post type posted. Every week, there's about 40000 reviews done, of which, only 8000 are on close vote review.
5k are on answers (Very Low Quality and Late Answer. First Posts is excluded because it also includes questions).
Now, this is where it gets tricky. Each post has one ReviewTask for each time it enters the queue. Which means the 8000 posts on CV Review Tasks In Week are all posts pushed into the queue for that given week. The line right under, CV Review Tasks Invalidated in Week, shows how many of the review tasks created that week that have been invalidated (which, according to both observation of an invalidated close vote review, and rene, includes expired close votes and flags). And the last line, CV Review Tasks Completed in Week, shows how many of the tasks created that week that have been completed.
Note that one ReviewTask does not signify how many user reviews were made on that task - it only shows how many, in this case, questions were in the queue with a creation date for a given week. Each of those can have any (sensible) number of reviews. To find out how many user reviews have been wasted, that requires a different SEDE query that I don't have available at the moment.
The vast majority of questions in the CV review queue aren't handled. The completed review items are those that have either been left open or have been closed in CV review. On some of the worst weeks, ~94% of CV reviews were made for nothing. I have no statistics on how many flags and close votes were essentially lost on this, but it's probably significantly more than one per review task.
The past few weeks show fewer invalidations, among other things because they haven't gotten old enough to expire, but also because the amount of reviews has spiked since about mid-April. This might be because of more reviewers, but it could also be because more questions now are worth closing than earlier (this is at least something I've observed on some tags).
Either way, those thousands of expiring close votes is a huge problem because they may contain questions that should be closed. And they can also contain the questions we actually want on the site, but that were handled poorly/incorrectly.
For those of you who aren't familiar with SEDE, completion type 3 on reviews (named "invalidated") means natural removal from the queue, and includes expired close votes/flags. This is verified in this SEDE query, as outlined in the SEDE structure post on MSE.
Note that this is an overview of what happens to review items by the review item creation date, and not necessarily the date it was handled. Because of expiration, it's fair to assume most of them were handled (or expired) relatively quickly
Personally observed situations
I don't review in the CV queue at all. I haven't for a long time because of one issue: 50 close votes doesn't let me cover the tags I watch, and review at the same time. I've seen several questions that should be closed getting answered rather than closed (and this might be a side-effect of the massive queue size for all I know).
For me, it's a question of whether review is worth it, compared to finding and tackling low-quality questions head-on as they emerge. Ironically, that doesn't work properly without review (there's relatively few active closers on the tags I'm watching, from what I can tell).
The CV queue doesn't work without people flagging or voting to close on the questions in the first place, but posts getting closed doesn't work effectively without review.
The partial lack of a coordinated effort (looking aside SOCVR) likely contributes to a lot of flags aging away. A few hundred reviews per day barely helps tackling backlog of closed questions, with new questions piling on at the same rate as review handles questions, if not faster. If I remember correctly, the CV queue was at 9000 relatively recently. I'm not sure whether the reduction is because of the fuzzy algorithm, if it's because votes and flags are aging away faster than review can handle it, or if it's because SO in general is getting fewer questions worth closing.
Ways to handle it
I can't use this answer to propose all the possible solutions, because there's far too many good ones that could be implemented. One that would massively help the active reviewers, is the one that enables regenerative close votes. I personally think we should let willing reviewers review as long as they want to as well - and if it helps on gamification-related problems, remove badge progressing for extra reviews per day. And the one Yvette mentioned: fewer close votes to close.
But the CV queue isn't alone. Help and Improvement gets a lot of posts from other queues that actually belong in the close vote queue. VLQ flags have helped, but instead of reviewing and actually helping, it's skipping and flagging that's (at least my) most used action in that specific queue.
There's essentially three ways to handle it:
The first, and by far most complicated one, is reducing the amount of closeable posts being posted. The wizard has likely had an effect (considering the CV queue is down to 8000 now), but there's still answers, although they fortunately haven't spiraled out of control in review to the same extent as the CV queue.
The second option is getting more reviewers. This has been done before with the CV queue burn, but getting more long-term reviewers is hard when the unimplemented queue improvements keep pushing reviewers away.
The third, and the one I've focused on: let the active reviewers review! Give the reviewers regenerative close votes and/or require fewer close votes to close questions, or implement something else to make sure those who review actually have a chance of making an impact, rather than browsing through a queue with 40 reviews per day.
And if the issue is a concern for abuse, why not let gold badge holder votes have more weight?
If the issue is with "zombie reviewers" clicking buttons for a badge, why not remove progress towards review badges after 40 reviews (or another number for different queues)?
Final notes
Please implement one or more feature requests that make review efficient. Assuming this comment is correct, last time there was a strike, the only action taken was reducing the visible number. While this may help motivating reviewers, it is useless alone. You can hide the truth all you want, but if nothing is done to tackle it, it's essentially like sitting in a burning house and ignoring it.
I mentioned the review burn earlier, but there is one thing I haven't mentioned yet, and that is this answer. The post itself isn't actually the main point, but before I go on, look at the numbers: all of them, including CV review, says 0.
However, under two hours later, it was back at 14000 questions. I can't verify the legitimacy of these images (given they weren't created by me, and it was 5 years ago), but if they are correct, that also means one thing for the review queue right now: there are guaranteed more than 8000 questions in the review queue.
I mentioned later the review queue size is "obfuscated" by an algorithm that makes it appear smaller. The comment I mentioned in the previous paragraph was likely made when the algorithm wasn't as aggressive. When it spikes by 14k questions in under two hours, and with activity only increasing since, it serves as further proof 8000 isn't a realistic number.
TL;DR:
If no review enhancements are made, we will not be able to moderate content efficiently given the current amount of votes, available reviews per user, and reviewers. We currently aren't either, given the massive volume of posts SO is getting.
With, on some of the worst weeks, up to 94% of review tasks expiring, this is a problem obfuscating the numbers further cannot fix. However, in light of a recent post here on meta, it has come to my attention SE actually isn't working on features for core members. That may make the purpose behind striking pointless if this push for change isn't enough (and in my opinion, based on previous posts, this probably isn't enough unless there's commitment to it). These past few days, there have been several feature requests with status changes. Some declined, some completed, some being worked on - this is progress into the massive pile of seemingly ignored feature requests. I'm not sure how long it takes until moderation-related feature requests start being implemented, but what SE stated earlier is no longer the case (fortunately). Just the fact that feature requests are being implemented is a good sign after a long time of nearly no implementations.
Related