152

Users are freaking out over the number of questions with close votes. More so when they look at the review queue and have to grab a newspaper to fan themselves as they do declare that the high number is giving them palpitations.

Currently that number represents all questions waiting for a pillow to come and make it a salt night. Those with three votes to close. Those with a single vote to close. You get the Tokyo Drift.

The number should be fuzzier, or at least abstracted to various levels of hurt.

Instead of saying there's 80, 90 or 120,000 (knocking close voters over with a feather from the neck crane), show the count of questions sitting at four close votes. That should be a lower number. And a lower number, as studies show, will deflate the sense of doom users see when they jump into the review queue.

When there are no four-voted questions, count the threes, and so on and so on until we have a sweat shortage from users relaxing a little more at the now not so large number asking for a ride home.

And when the four-voted questions are up, jump to that number. Because as we all know from dieting, nothing beats a yoyoing number.

20
  • 11
    Not sure about the assumption (that article doesn't seem to reference any authority whatsoever) that seeing smaller numbers will result in more haste. It will just hide the fact that there is a huge number of reviews left. Automatically filtering by the user's best tag or some other metric which could help users review quicker would probably be more efficient than merely hiding the problem.
    – Travis J
    Commented Oct 7, 2013 at 22:56
  • 25
    We should just show the natural logarithm instead! "11,325 questions left!" (11325= 1000*ln(82832)) Edit: Wait, actually, that's not a terrible idea.
    – user1131435
    Commented Oct 8, 2013 at 0:19
  • 2
    The issue with this suggestion is that not many questions will have three or four votes in the queue. Since people vote on questions until their reviews are complete, not many questions will have four votes, and those that do will leave shortly. When they do, the next question is going to bump up to take its place. I like your idea in principle, but I'm not sure that method will work well.
    – user1131435
    Commented Oct 8, 2013 at 1:29
  • 11
    The second request is to change the number to
    – random
    Commented Oct 8, 2013 at 1:35
  • 12
    ∞ is too intimidating. How about lots?
    – Charles
    Commented Oct 8, 2013 at 2:44
  • 14
    @Charles: Yeah, but you have to show progress somehow. Maybe progress from lots and lots and lots => lots and lots => etc. Commented Oct 8, 2013 at 3:21
  • 38
    Or ∞ that changes in font size.
    – random
    Commented Oct 8, 2013 at 3:30
  • 17
    I have no idea what this feature request is about, but I love it's wording, so I support it.
    – Andrew Barber Mod
    Commented Oct 8, 2013 at 3:53
  • 1
    This is only until the size gets down, right? If it says there are 10 items in the queue but you can use all 40 of your reviews without clearing it, it would confuse a lot of people.
    – ughoavgfhw
    Commented Oct 9, 2013 at 16:38
  • Ne, ne... the queue is reaching 100k... We need this NOW
    – Braiam
    Commented Dec 2, 2013 at 22:37
  • 1
    Couldn't help but read that second sentence in Jon Stewart's Lindsey Graham impression.
    – JDB
    Commented Dec 3, 2013 at 20:04
  • 4
    I personally dont bother with the CV queue for 2 reasons. 1. its over 100k, why bother when you wont make a dent when limited to the number of votes you can cast. 2. why am i being shown questions with 1 or 2 votes? why am i not being shown only questions with 4 votes so those questions can actually be removed from the queue? The only thing stoping thousands of posts hitting 5 CV's is the stubornness of the admin who are just blindly saying that only showing 4's (then if there are none, show 3's and so on), will make no difference.
    – bizzehdee
    Commented Dec 3, 2013 at 20:32
  • 9
    Although this option bothers me greatly, they did a great job at doing nothing for a long time, that I'm just glad they finally did something different, even though it will not make any difference in the long run.
    – CRABOLO
    Commented Feb 27, 2014 at 14:15
  • 1
    As a cheerful inspiration of what's possible, over at CrossValidated the ratio of reviewers to queue is so good that the queue is often empty, making one reviewer think he'd been blocked
    – smci
    Commented Mar 2, 2014 at 13:02
  • 1
    Has the number of close vote reviews per day gone up since this change? Edit: nvm, found the close queue rallying post which has numbers, but also means we don't know yet if this change happening under the hood would have been enough alone to psychologically soothe people into reviewing more.
    – jball
    Commented Mar 3, 2014 at 18:15

6 Answers 6

82

This is now implemented (build rev 2014.2.27.1403 on sites) as lined out in Shog's answer. We now have a per-site configurable close vote threshold for questions to be enqueued in the close vote review queue. Review tasks with fewer close votes and no do not close review results will be gradually (performance...) removed from the review queue.

Expect the SO close vote review queue to start shrinking down to ~5k tasks as soon as we crank that threshold up to 4 (on SO only, obviously). Once the queue is cleared we'll decrease it to 3, and so on...


Update 3/3/14

Queue with threshold 4 was empty, we decreased the threshold down to 3, queue size was back to 14k.

Update 3/9/2014

Threshold decreased to 2, we're almost there!

39
  • 20
    And I thought the recently elected moderators worked overtime when I saw fallback to 99k questions in the queue :)
    – TLama
    Commented Feb 27, 2014 at 11:27
  • 57
    So, uh, if you stop adding things with 1 close vote to the queue, doesn't that mean that if I cast the first close vote on a new, bad question, it's much less likely to actually get closed, and more likely to just expire without anyone even seeing my vote?
    – Wooble
    Commented Feb 27, 2014 at 12:39
  • 14
    Does the threshold go all the way to 11?
    – Bart
    Commented Feb 27, 2014 at 12:46
  • 4
    how does this interplay with filtering? I mean, did you test 1) what happens when one's filter is such that it involves only questions that are fuzzied away? 2) Will 0 be displayed to this user? 3) Will user be blocked from entering the queue? 4) If they enter the queue, will they see questions that match their filter? and, 5) will next question that matches their filter load to them after review completion? (that's a rough test plan for such a change by the way)
    – gnat
    Commented Feb 27, 2014 at 13:08
  • 6
    But that's... that's... that's cheating! Commented Feb 27, 2014 at 15:06
  • 2
    So we should end up with a queue full of (1.) questions where the sum of close votes + close flags is greater than four, and (2.) questions that have previously received a "do not close" review vote (regardless of the number of close votes / flags). Right? Or am I interpreting this incorrectly? Commented Feb 27, 2014 at 15:13
  • 3
    So when we've cleared the 4 vote questions the queue will jump back up to 10k+ (say) three vote questions, but what happens then when people start adding the 4th vote. Won't it suddenly drop back down to single figures as there won't be that many 4 vote questions, and then snap back up to 10k+ as soon as they're cleared? Or will you change the number of votes down to 3 and keep it at 3?
    – ChrisF Mod
    Commented Feb 27, 2014 at 15:33
  • 5
    This breaks filtering pretty badly, @gnat. If we can get the threshold down to 2 or even 3, it should be a lot better though. This is strictly a temporary measure to clean up some of the backlog.
    – Shog9
    Commented Feb 27, 2014 at 16:42
  • 6
    "Review tasks with less close votes and no do not close review results will be gradually removed from the review queue". Does that mean that a leave open vote may actually increase the chance for a question to be closed? Since one such vote means it will not fade from the queue, thus increasing the chance it will gather more close votes? This seems counter-intuitive.
    – HugoRune
    Commented Feb 27, 2014 at 16:53
  • 4
    The test plan is as follows, @gnat: m0sa turns this on while I'm asleep, and all of you test it. So far, so good...
    – Shog9
    Commented Feb 27, 2014 at 17:12
  • 7
    @Shog9 believe it or not but there's even a widely used term for this: scream test. You push it live and just wait if anyone screams WTF
    – gnat
    Commented Feb 27, 2014 at 17:13
  • 3
    but don't forget that they'll be back
    – m0sa
    Commented Feb 27, 2014 at 19:17
  • 4
    @m0sa: Looking at what Wooble mentioned about new single-vote closes not being prioritized in the queue; maybe it's possible to have questions that received a close vote (for any number of close votes) in, say, the past 12 hours bypass the threshold? That way the priority is both the 4-vote low hanging fruit and questions that are getting attention in the present (personally, in this situation, I think dealing with recent activity takes priority over cleaning up old garbage - it's much more likely to cause unhappiness if somebody who is active now doesn't see an action taken).
    – Jason C
    Commented Feb 27, 2014 at 20:27
  • 3
    So much for election promises!
    – pianoman
    Commented Feb 28, 2014 at 2:47
  • 2
    @msh210 all sites except SO have the threshold set to 1
    – m0sa
    Commented Mar 3, 2014 at 18:00
67

Counter proposal: why not show how many questions are awaiting review in my part of the site?

Rather than providing a somewhat abstract meaning of "number of questions in the close queue", why not focus on what people really care about; how many questions are awaiting closure in their part of the site?

So if I've filtered the queue to and , the number of close votes isn't 92.5k, it's 1.5k1. Not only is that number psychologically better; it's also more useful to me. What do I care if the PHP guys have 10k1 of questions to close. That doesn't affect me. The Java questions on the other hand do!

1: Numbers made up

10
31

Note: Based on Shog's answer to this question, reviews actually are prioritized based on number of past reviews. I am leaving the below for posterity, and redoing the math based on the numbers he shared in the comments

Here are the current amount of posts in the queue by number of existing votes:

Reviews #
0       62777
1       14747
2       3871
3       955
4       150
5       32
6       5

I will assume:

  • Items with zero reviews start at 1 close vote
  • 100% of reviewers vote to close
  • Therefore anything with 3+ reviews would currently have 4 close votes
  • Close votes expire after 4 days, and 100% of questions have over 100 views
  • We get 600 reviews a day (all votes to close)
  • We will get 600 more questions in the close queue

Note: These are assumptions for the sake of running the numbers.

If we were to implement this idea, here is what the queue would look like if they implemented it currently:

Votes #
1       62777
2       14747
3       3871
4       1142

Because questions with fewer votes take more reviews to clear, we would be able to clear out the 4-vote questions in 2 days. It would take 13 days to clear the 3-answer questions. And then it would take 74 days to clear out the 2-vote questions. By the time we were down to the 1-vote questions, we would need 774 days to clear the queue (and would never catch up).

While it would feel good to be able to clear out the low-hanging fruit over the next couple of months, we would just run in to the same problem again once we got to the 1-vote questions.

I think that using smoke and mirrors to make the queue appear smaller is just a cosmetic change that doesn't solve the fundamental problem with the size of the queue right now. I would much rather priority spent actually solving the underlying issues.

Previous answer remains below


Executive Summary

I strongly disagree with this idea because:

  1. It doesn't solve the problem
  2. It doesn't reflect how the queue is actually handled

I would much rather the developers spend time on fixing a problem rather than dressing it up and creating separate issues.

It Doesn't Solve the Problem

The issue is that the queue is not decreasing in size. As Shog9 points out this is because the number of review tasks (the number of flags and close votes outside the queue) are increasing.

Let's assume this increases motivation to use more close votes. This will only affect:

  1. People not using all their close votes each day
  2. Who are looking at the queue size (actually look at the review queue)
  3. Who only avoid the close vote queue because of the size

In other words, it is a drop in the bucket and will likely have no real impact.

It Doesn't Reflect How the Queue is Actually Handled

If you show the amount of review tasks with 4 close votes, that doesn't mean you will get a review task with 4 close votes. As gnat has pointed out the issue is that the queue doesn't work that way, and you are as likely to get a question with 1 vote as you are to get a question with 4. Let's say that 1/4th of the questions in the queue have 4 close votes (I strongly doubt that is the case, but let's assume anyway), that means when you click a question you have:

  • A 50% chance of not impacting the number at all (get a 1- or 2- close vote review)
  • A 25% chance of increasing the size of the queue (get a 3-close vote review and vote to close)
  • A 25% chance of decreasing the size of the queue (get a 4-close vote review and vote to close)

So this 'motivational tool' will actually result in you pushing up the number, which certainly doesn't make a whole lot of sense.

Example

There are 80k questions in the queue (give or take).

Assuming even distribution:

  • 20k with 1 vote
  • 20k with 2 votes
  • 20k with 3 votes
  • 20k with 4 votes

This proposal would show the queue size as 20k (number of 4-vote questions).

However, due to the review task you get being random, you are no guaranteed to get a question you can vote to close and eliminate from the queue.

If you get a 1-vote or 2-vote question, even if you vote to close there will be no impact on the 20k (number of 4-vote questions in the queue).

If you get a 3-vote question, and vote to close, that 3-vote question will become a 4-vote question. The 20k number will become 20k + 1 -- the size of the queue will increase as a result of participation.

If you get a 4-vote question, and vote to close, one question will be eliminated from the queue.

If we add up the probabilities in regards to impact on queue size we get:

( 50% * 0 ) + ( 25% * 1 ) + ( 25% * -1 ) = 0 + 1 - 1 = 0

In other words, there will be no net impact from reviewing on the queue if we just make the numbers fuzzy. This is not motivating.

19
  • 4
    Meh. The reason I disagree with you is that motivation is a key tool for success. While it's not numerically precise, it's good enough.
    – user1131435
    Commented Oct 8, 2013 at 1:15
  • 5
    You have a 25% chance in increasing the number, which is equivalent to your chance of decreasing it. And a 50% of having no impact at all. This is not motivating. Net effect on queue: zero. Even worse if there are fewer 4-vote items in there (which I have to assume is the case since close votes expire).
    – jmac
    Commented Oct 8, 2013 at 1:18
  • I don't understand what you mean by this. Also to your last point, that isn't how the review queue works. You aren't assigned a random question from the queue. The queue works like... a queue. Everyone sees one of the next few questions and takes action on it until it's removed from the queue. If questions were distributed randomly, nothing would get done.
    – user1131435
    Commented Oct 8, 2013 at 1:20
  • 2
    The close vote queue is the red pill, showing users the reality of how much junk the site needs to be KFC
    – random
    Commented Oct 8, 2013 at 1:20
  • 1
    @Emrakul I edited my post to include an example. Changing the display without changing behavior will entirely eliminate the link between your voting and progress in the queue. Instead of going up, it will stay steady, but this doesn't add motivation and we would never get to a point where it only displays 3-vote questions.
    – jmac
    Commented Oct 8, 2013 at 1:25
  • Once again, to your edit, that's not how the close queue works. There would never be 20K questions with any number of votes. The number of questions getting close votes in the queue is highly limited. Maybe a dozen. That means that almost all questions will have only one close vote. Only the ones which are currently being voted on acquire votes from the queue. Additionally, the formula can change the display number; see the logarithm. It just won't change much, but people expect that anyway.
    – user1131435
    Commented Oct 8, 2013 at 1:25
  • 2
    Yes, I think almost all questions will have only one close vote. That means that showing people the number of questions with 4 close votes, especially if you never actually get a review task of the 4 close vote questions, will not be a motivating thing to display at all and won't help solve the problem.
    – jmac
    Commented Oct 8, 2013 at 1:26
  • 1
    Hmm... sorry, I'll post this comment on random's OP. I think the suggestion is a good one, if slightly misplaced by mechanics. Even so, I disagree with your calculations.
    – user1131435
    Commented Oct 8, 2013 at 1:28
  • 2
    @Emrakul If there are fewer 4-vote questions than 3-vote, the math gets even more in favor of not implementing the suggestion as participating in the queue itself will increase the number that is displayed. Unless 4-vote questions are prioritized, and we knock off low-hanging fruit first, this suggestion will not increase motivation as participating in review tasks will increase the number (which is about as demotivating as you can get).
    – jmac
    Commented Oct 8, 2013 at 1:38
  • 1
    Like I said, I agree with the suggestion in principle. As in, I agree that such a change would be helpful. As I have noted already (and agree with you) the specific change listed wouldn't necessarily be helpful.
    – user1131435
    Commented Oct 8, 2013 at 1:41
  • 2
    Just using a lower number will not impact the direction of the number of items in the close votes queue (up). In addition, any suggestion to use some formula (like your ln()) to make it smaller will also decrease the effectiveness of reviews in eliminating questions from it. Smoke and mirrors that do not solve the problem.
    – jmac
    Commented Oct 8, 2013 at 1:45
  • 1
    FWIW, there's nearly always a better chance of getting a task with previous reviews than one without. Current breakdown of pending review tasks by # of previous reviews: 1->14747, 2->3871, 3->955, 4->150, 5->32, 6->5
    – Shog9
    Commented Oct 8, 2013 at 4:55
  • 1
    No, 62777 have had no reviews (add 'em up, and you get the current size of the queue). I'm afraid there's no easy way to get this data right now.
    – Shog9
    Commented Oct 8, 2013 at 5:01
  • 5
    Strictly-speaking, it's higher-reviewed - regardless of what the outcome of those reviews were (ignoring "skip"). This has two side-effects: 1) very few tasks with 6 previous reviews awaiting the "tie-breaker", and 2) the occasional 4-close-vote question languishing in the bowels of the queue because none of those votes came from /review. The actual breakdown by # of close votes is: 0->8320, 1->40711, 2->20193, 3->9812, 4->3498
    – Shog9
    Commented Oct 8, 2013 at 5:11
  • 2
    we don't want to solve the number, we want that people don't get hearts attack when seeing the review queue (and don't get depressed either) so we can focus in prevent questions from being asked in first place (I'm sure 80% of the review queue is duplicates).
    – Braiam
    Commented Dec 2, 2013 at 22:46
23

I agree.

Think about it from a pure UX perspective: do users really need to know there are 90 trillion questions in the review queue? Or do they simply need to know that there are a metric crapload and they should dedicate a little bit of lovin' time to the queue?

But wait, there's more... let's expand on the idea. Can we order the way the questions are presented to a reviewer; i.e. show the 4 vote questions first, then the three vote ones, etc.? Obviously the four vote questions are more in need of attention, can disappear off the radar sooner, and give a reviewer a greater sense of satisfaction than voting on a single vote question that may not be reviewed again for some time.

Currently the palpitations and feelings of inadequacy I experience when I see the close review queue registers a big fat 9 on the Richter scale, which is the same as what I experience when I see Vicki Pollard:

enter image description here

1
18

This... Actually makes a lot of sense.

Here's the current breakdown of review items by the total number of active flags or close votes associated with each:

flags/ review
votes  tasks
------ ----- 
8      1     
7      15    
6      34    
5      246   
4      4466  
3      12116 
2      25306 
1      57980 

If we removed everything with less than 4 flags or votes from the queue today, we should be able to burn through 4762 items that require only one vote to close fairly quickly. Then the 12,116 items (or whatever it is by that time) that would require two votes would take a bit longer, but hopefully not too much.

This should be fairly trivial to implement too... But there's a slight problem: if we dequeue all of those posts with a small number of flags or votes, we throw away the active Do Not Close reviews on them too. Assuming we later add them back to the queue, the votes are preserved but the DNC responses aren't. Here's how tasks with Do Not Close reviews break down:

DNC reviews review tasks 
----------- ------------ 
4           2            
3           4            
2           604          
1           5414         

That's a bit awkward. We could just leave anything in the queue that has more than N flags or at least one DNC review though. That would add 6K or so to most of the initial thresholds.

A more elaborate solution would involve reactivating old, invalidated review tasks when the threshold was lowered. I'm not sure this is really worth the added complexity, however.

Once we're down to a threshold of 2 reviews, I suspect getting to 1 is going to require a lot more effort; in the meantime, posts with only a single vote or flag aren't going to be getting much attention. That's when we're going to need to bring in some more complicated ideas for prioritizing things.

And in the meantime, we can be working on getting more folks involved in reviewing.

5
  • 1
    Are questions with more close votes shown first/ahead in the queue?
    – random
    Commented Dec 3, 2013 at 21:14
  • Right now? No; questions with more reviews are shown first/ahead, with the caveat that selection is heavily influenced by the age of the review task as well.
    – Shog9
    Commented Dec 3, 2013 at 21:16
  • as far as I can tell, you don't really need to physically dequeue currently existing posts with a small number of flags or votes. It would likely suffice to just temporarily block these from feeding to reviewers - this will have an effect on reviewers focusing solely on higher flag / votes counts. Whether to (temporarily) prevent or postpone adding new posts like that to the queue is another matter
    – gnat
    Commented Dec 4, 2013 at 15:39
  • Sure, @gnat. Dequeuing them just happens to be the most straightforward way to implement that.
    – Shog9
    Commented Dec 4, 2013 at 15:41
  • ...yes, I only mentioned this since you concerned about preserving existing Leave Open votes - blocking would keep these intact (ignoring various effects of votes expiration for simplicity)
    – gnat
    Commented Dec 4, 2013 at 15:43
5

I love this idea. Since it was just implemented (as per m0sa's answer) I have an additional suggestion, depending on how this goes.

Along the lines of the psychological effects of showing a decreased vote count: If this change does not sufficiently increase the review rate, a way to further increase participation would be to display the number of close review votes you have remaining in a little read box up in the title bar (or integrate it into the review tasks count that is up there now).

The idea is to create a sense of "todo" by drawing attention to the tasks and encouraging users to complete them instead of passively relying on them to go to the review area. This would have the most effect on users that may previously have seen and ignored the "review" link without much appreciation for the importance of community reviews. Adding a notification changes it from "check out 'review' and see what's there, if you're bored and in the mood to explore" to "you have some review tasks left to complete today". People will generally respond to notifications that give a sense of urgency and/or importance.


As an aside, I had meant to do as much of a study as possible on the database dumps to see if there was a distinct correlation between queue size and reviewer participation, to present an idea similar to this. So, I'm really happy to see that this was implemented (mostly because I don't really need to sort through all that data any more to convince anybody), and I think it will have a good effect.

It's also why I posted that unpopular question (which some of you may have seen, and was appropriately closed, of course) asking what effect the displayed queue size had on people's actions on the site. In reality, knowing the size of the queue isn't particularly useful information and, theoretically, did more harm to morale than good -- I was looking for reasons to support reducing or removing it entirely (in combination with the notification idea).

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .