21

Currently the close vote queue is up from 7.2K to 8K. I'm always left wondering what a healthy close vote queue would be for StackOverflow. Common sense tells me that a queue that is linearly growing with the number of questions asked - certainly not faster - is a healthy queue. If the queue is shrinking everything is fine; if it keeps growing it will grow out of hand.

Do we still need additional measures to counter the growth of the queue? Do we think it takes too much effort to keep it in check? Or are we actually happy with the current situation and is 7-8K a healthy number given the huge amount of traffic we are handling?

8
  • 2
    OK, manageable would probably be a better word than happy. A close vote queue of 0 would make everybody happy. I see that it takes way too much time for crypto questions to close, so I'm not that happy myself for this very particular area. Commented Sep 15, 2015 at 14:16
  • 3
    related: Enough fuzzying: let's let everything into the close queue and age out questions that don't reach a threshold. TL;DR - currently close queue is tuned in a way that is considered healthy by Stack Exchange team
    – gnat
    Commented Sep 15, 2015 at 15:42
  • 1
    @gnat Thanks for the link. I'm still interested in what the active reviewers and users of SO think of course. As much as I admire the SO team, SO is made up of more than just the team (and they would be the first to acknowledge this). Commented Sep 15, 2015 at 15:49
  • 3
    well, then you probably need to somehow take into account that most users are unaware of how many questions with close votes are there really. Last time I checked, it was something 30-50 thousands, are you interested to know what they would think of this number or how they feel about that artificial number shown to them as queue size?
    – gnat
    Commented Sep 15, 2015 at 15:52
  • 30-50K seems excessive to me and it sounds like we are still having trouble in that case. The queue I see is considerably smaller than this value, does that mean I see only part of the story at the moment? How come? Commented Sep 15, 2015 at 15:54
  • 1
    showing you only "part of the story" is one of the main points of current design, this is discussed in much details in the question referred in my first comment
    – gnat
    Commented Sep 15, 2015 at 16:14
  • Hmm. Loss of transparency should not be what we're after. What is the difference with a prioritized close queue? I.e. we see all close votes, we just don't get to vote on them? Or is there another way of seeing the amount of questions that have been voted on for closing, mainly for discussion on meta? As said, on crypto many questions that should really be closed, aren't - I understand now that they are only briefly in the close queue. All is not well. Commented Sep 15, 2015 at 18:34
  • 6
    If you want to get that queue down you'll need more voters. Some of them are gathering in the SO Close Vote Reviewers room which apply special tactics to make sure no close votes age away.
    – rene
    Commented Sep 16, 2015 at 8:53

2 Answers 2

20

On average, roughly 2500 questions attract at least one close vote or flag every day on Stack Overflow. This has gone up a bit over time.

On average, roughly 800 questions are actually closed every day. This has been more or less constant for quite a while.

Oh... And about 50 questions that enter close review exit with a "Leave Open" result each day.

As you can see, there will be somewhere around 1700 more questions each day that enter the close queue than those that leave. The votes that keep them there age away after between 4 to 14 days, assuming they're not retracted in the meanwhile. Also, some significant number of these questions get deleted. I don't know how many, but tons of questions get deleted all the time, so you gotta figure...

Anyway, 1700*14 is 23800. You can probably assume that that's more or less the real size of the close queue, in the sense that that's how big it would be if every pending close vote or flag kept a question in it - but that'd be a waste, so after a few days (currently 4), questions with less than some threshold of votes or flags (currently 3) are dropped out of the queue; they can be re-added if they get more votes, but after four days of not being closed this is increasingly unlikely and there's a good chance the existing votes will start aging away soon anyway.

So the count reflects the delta between questions that are identified (by someone) as needing to be closed, and questions that are actually closed. While the exact range of time this delta is allowed to accumulate is subject to some variation, it's rarely ever more than 2 weeks and usually a lot shorter.

Thus, the queue will only increase in size by a significant amount if a lot more questions get voted on or flagged and/or a lot fewer questions get closed each day. Both of those could happen. Generally though, the size of the queue fluctuates naturally throughout the week and throughout the year as the volume of questions asked goes up and down.

12
  • Thanks for the insight, Shog9 Commented Sep 16, 2015 at 2:54
  • 5
    1700 escaping via aging vs 50 via Leave Open is quite a difference. One could conclude that Skip is the new Leave Open; the effect is about the same except Skip doesn't count toward review limit.
    – user3717023
    Commented Sep 16, 2015 at 3:42
  • I really like what you guys did with the close vote queue. Definitely an improvement over the six figure stuff that we were beaten over the head with for a while.
    – Travis J
    Commented Sep 16, 2015 at 5:33
  • 5
    @TravisJ It feels a bit too much like ignoring the problem to me. This figure tells me that alternatives may still be needed and could / should be discussed. It also explains why a lot of the crypto questions I try to close remain open (outside of the odd dupe with encryption of course, got a GM for that). Commented Sep 16, 2015 at 11:57
  • 1
    What @Maarten said. "Aging" of votes is not a solution to the problem. The problem is: someone thinks the question needs to be closed, but not enough people know enough about the subject, or don't understand the close reason enough to agree with that, hence the question stays open.
    – CodeCaster
    Commented Sep 16, 2015 at 12:07
  • @CodeCaster: Yes and in some cases the vote of a user who knows enough about the subject is not given the level of importance that it can be afforded. For example, the dupe hammer could be given to silver badge users (say 3 silver users to close), opinion based and typo error closure reasons could also do with a hammer like behavior because surely a gold badge user should know enough about them.
    – Harry
    Commented Sep 16, 2015 at 12:28
  • did you manage to handle that old issue with close-worthy triaged questions hanging in limbo? (FWIW 23800 looks like a really healthy amount to me)
    – gnat
    Commented Sep 16, 2015 at 14:54
  • @gnat: Not sure if sarcasm... but any routinely non-zero, much less non-zero-able, queue size seems instinctively and inherently dysfunctional. "We're going to randomly toss about two thirds of initial votes and flags, because they didn't accumulate enough buy-in" sounds more like a problem of scale than a fundamentally sound principle, made more prominent by the pride SE (rightly) takes in solving such problems of scale everywhere else. Commented Sep 16, 2015 at 21:30
  • @NathanTuggy it's not a sarcasm (I remember times when there were 60K, 80K, over 100K questions with close votes) but your point is an interesting twist, I need to give it a thought. As for making close queue near zero, I am not very comfortable with this, at scale of Stack Overflow this feels somewhat unstable. Near zero queue would likely mean insta-close of almost any question that gets flagged / voted, it doesn't feel right for closing to be that easy
    – gnat
    Commented Sep 16, 2015 at 21:51
  • @gnat: Well, rapid close or leave open, one or the other. That would make on hold a bit more significant and healthy by requiring quick reopens once a question had details added or whatever. On the whole I think that would be good, and all other flags/votes tend to move on that timescale. Commented Sep 16, 2015 at 22:25
  • 1
    I'm in favor of fast closing when it's actually necessary, @Nathan. You can't fake it though; folks seem generally disinclined to close certain questions, and increasing the size of the queue by withholding aging just ensured that those which did need to be closed got lost in the noise. Remember, the existence of a backlog was a created problem - and the problem it was created to solve still exists if questions aren't actually getting closed.
    – Shog9
    Commented Sep 17, 2015 at 0:17
  • @Shog9: I'm not going to say that the current system should be scrapped for one of its predecessors. But I will say I don't think it solves everything. The existence of a massive backlog is a design smell, and the fact that it would take three times (or more likely, six or seven times, counting the halving of reviews at <1000 entries) as many reviewers to have any hope of getting it down to zero is fundamentally disturbing. I do have some ideas, but the margin is too small to contain their proof. Commented Sep 17, 2015 at 0:23
-5

I think there are two fundamental pieces that keep the close queue from being manageable:

  1. The 40-per-day limitation on close reviews
  2. There's no incentive to spend time in the queue once you've received the Steward badge.

Consider:

  1. Once you've received the Steward badge (and have a minimum rep of e.g. 5k or 10k), it stands to reason that you know what you're doing. For Stewards, the 40-per-day limitation could increased per day, or a weekly (or hourly) limitation could be set instead (or the limitation could even be lifted altogether?)
  2. Adding a new badge into the mix isn't likely to be effective. Instead, why not offer reputation in exchange for voting in the review queue?

    For example: if a user performs at least 100 reviews in a week, they're then awarded an extra 10 reputation. For each 10 additional review done per week, an additional reputation point is awarded. Reputation bonuses would be calculated and awarded on a weekly basis.

13
  • 8
    Adding in rep will just make the robo reviewer problem worse then it already is.
    – Joe W
    Commented Sep 16, 2015 at 20:52
  • 1
    Adding a new badge can't hurt. Make it so you can get it multiple times, so there is always an incentive to go on.
    – Anders
    Commented Sep 16, 2015 at 20:53
  • @JoeW that's a problem that can be solved with software. Enforce a limitation per hour, or whatever - the design is up for debate for sure. But adding some form of incentive that encourages others to help? Perhaps there should also be a minimum reputation as well - e.g. 5k or 10k. Commented Sep 16, 2015 at 20:56
  • Also, small amounts of reputation start becoming a lot less useful past 3k reputation, so it's not even that great of an incentive. It's thousands of rep to the next privilege.
    – ryanyuyu
    Commented Sep 16, 2015 at 21:00
  • @ryanyuyu sure, but it's pretty easy to hit 1000 reviews in a week - with this model, that's 100 rep, equivalent to 10 up votes. Not a lot, but not a little either. Commented Sep 16, 2015 at 21:01
  • 1
    Limiting how fast they can go through a queue will not change the fact that they are doing robo reviews and not doing a proper one. Add in a reputation gain and you will see the same reviewers complete all possible reviews in the queues that award reputation while at the same time providing bad reviews. The simple fact is that the more incentive there is to do the reviews with something like reputation gain the more robo reviewers will continue to to bad reviews rather then stop once they get the badges. Also reputation does not mean you are a good reviewer.
    – Joe W
    Commented Sep 16, 2015 at 21:05
  • @JoeW and there's no programmatic way of flagging robo reviewers? Commented Sep 16, 2015 at 21:06
  • 1
    If there was would they not be using it now since there is already a problem of robo reviewers? The problem isn't a new one and adding in reputation gains for reviews would just make it worse. Not to mention that a feature like this would most likely be network wide and would have a lot larger impact on a beta site.
    – Joe W
    Commented Sep 16, 2015 at 21:08
  • 1
    It would also help if every user that has moderation privileges, like close vote reviewing, use all their reviewtask every day. It is not that hard I would say.
    – rene
    Commented Sep 16, 2015 at 21:10
  • @rene: A lot of users (or at least a lot of voices) have given up reviewing because they can't stomach audits. Making those more palatable without making them significantly worse at cutting roboreviewers down to size would go a long way toward taming the queue. Commented Sep 16, 2015 at 21:32
  • @NathanTuggy in the cvq you can always filter on a specific tag. You do notice when a Haskell question pop-ups... and if not, failing that audit seems fair. I failed most of my audits out-side of the CVQ. And this question and answer is about the CVQ. I would say to those voices, just try it for a week, every day, and then speak up again...
    – rene
    Commented Sep 16, 2015 at 21:44
  • But again, what's the incentive to keep doing it? Commented Sep 16, 2015 at 21:45
  • I have no idea but obviously I don't need an incentive...
    – rene
    Commented Sep 16, 2015 at 21:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .