Alright, well, shoot. Whether it’s reverse psychology, the Hawthorne effect, or some other fun social phenomenon, I am only human, and I am too curious for my own good. So why not? I decided to go back into the review queues after being thoroughly discouraged since my initial post on the subject and see what happened.
First of all, yay. Thank you. It is nice to know that my request was heard, if perhaps a year late. Having more close votes sure does feel different, and it’s actually pretty nice. Unfortunately, this implementation actually kinda missed the point of my original post.
I'm not really complaining about a failure to close all the questions. That won't ever really be fixed. But frankly, I want to be able to close questions that I deem poor. When I run out of close votes, I stop trying to answer questions in poor-quality tags because I'm frustrated I can't close the awful ones. This actually reduces my positive contributions to the site, and I don't imagine I'm alone in that.
It’s not about the review queues
When I posted my initial request for more close votes, I scraped together a scary graph displaying the number of unclosed questions per month:
This graph shows a symptom—questions getting some close votes but never enough to seal the deal—a symptom that the close vote queue is pretty much explicitly designed to fix. With that in mind, it might seem like this issue is really about the review queues. It isn’t.
Here’s what that graph looks like today, in March 2016:
What happened? Well, Shog happened. New close vote aging logic was implemented to age out close votes that weren’t getting enough attention, replacing the old logic which let them stick around. With this in place, the effects were sudden, and the effects were total. The review queues were no longer a problem.
Reviewing posts is a chore
So, let me be honest: I do not like going through the review queues. I haven’t been doing it much lately, but that is not because things have changed, I’ve just been a lot busier. That’s not the point. I don’t know if anyone actually enjoys clicking through posts in the queue (almost all of which are terrible), but I don’t, and I would expect I am not alone in that feeling.
So then why would I use the queues? Why do I use the queues? Well, there are some motivators:
- Stinkin’ badges. I already have both the close vote queue badges, so this is less relevant to me, but some good gamification can’t ever really hurt, especially just to get things going. (Generally, though, I think the motivation factor of these badges is overstated.)
- Small, easy goals. The review queues are generally nearly instant gratification. Just take a few minutes out of your day, and boom. Twenty posts reviewed. This is an easy goal to work towards—it isn’t hard, but it feels good to have finished it.
- Feeling like you’ve done your part. Related to the above, getting that “come back tomorrow” screen feels good. It says “I did my part for today”, and that’s it. I can feel a little tiny bit better about myself, and I can get on with my life.
Those last two points are especially important because they create a positive feedback loop to produce a self-sustaining system. First, the goal feels achievable, so I’m willing to invest some time to just knock out the queue. Second, I get a feeling of gratification when I’m done. This makes me want to do more, which I can do consistently because it is so easy to do each day. Et cetera.
This balance is delicate, and it almost certainly isn’t the same for everyone, but erring on the side of caution is important to maintain that positive loop because...
...queue burnout is a real thing that happens. I just ran through the queue and reviewed sixty—that’s right, sixty!—posts. That is a big number. And you know what? I’m not sure I would want to come back and do it tomorrow, even though I might if it were a paltry forty. Now, one might point out that I could just do twenty and call it a day, but that’s not what my brain’s reward centers crave: I want that positive reinforcement, dammit! So I am possibly doing even less by being given the option to do more.
So with that in mind, was Shog right and was I wrong? Was increasing the cap a bad idea with potentially disastrous consequences? Well... I still don’t think so. It isn’t quite that simple.
It was never about the queues, anyway
Let me select some quotes from my question and its associated comments. First, that original quote once again, from me:
When I run out of close votes, I stop trying to answer questions in poor-quality tags because I'm frustrated I can't close the awful ones.
From Jim Garrison:
After I run out I see no point in slogging through the muck looking for the occasional gem if I can't help clean up the muck. I really believe in StackOverflow as a resource and I'm not here for the rep. The vote limits are contribution limits.
From Josh Caswell:
For myself, I know that I only hit the cap when I use /review -- and (paradoxically?) that's part of the reason I don't use /review much. Burning through the queues -- and thus my votes -- doesn't take all that much time compared to the rest of the normal browsing day. Sure, I could stop reviewing early, and save myself more votes depending on the time, but then I'm putting way too much strategic thought into, as Will likes to say, mopping up the sick on the floor. (And I'm not even sure there's a displayed counter of daily close votes.)
From tchrist:
Despite being the top CV reviewer there, I "seldom" use the CV queue on ELU because then I run out right away. Yes, I get a few more but I still run out of CVs there on most days I'm an active voter. I bet if you look at the site stats, you'll see this isn't rare. I try to treasure them up and look for canidates in the 10k queue as need be. Just too many need closing to squander my CVs, which means I don't use them enough as much as I should.
The people who are hitting the caps and getting frustrated don’t want more time in the trash compactor, they want the privilege to throw the litter they find in the trash.
The people who spend time in the queues are on cleanup duty, but they are really a necessary evil. Ideally, we wouldn’t need to waste valuable human resources on validating close votes. If all close votes were cast intelligently and fairly, the close vote queue would be pointless—but we’d still need the initial person to cast that first vote. Ultimately, this is where the frustration seems to happen the most.
Close votes are a panic button
In Stack Overflow’s most popular tags, there is an extremely low signal to noise ratio, and the “close” button is a way out. It’s a way of saying “eep, this question is not worth my time or anyone else’s, please get it out of my face!” (Or for you power users, it’s pronounced mc24enter.)
The close vote is comforting to the user searching to answer good questions because it marks that question as adequately managed and has signaled to the system that it is of low quality. Theoretically, this system (along with ordinary up/downvotes) helps good questions rise to the top as the sand sinks to the bottom. Whether or not this is actually true in practice is mostly irrelevant to this particular post; the point is simple: being unable to report poor questions causes a surprising amount of psychological friction on the part of the benevolent user. The barrier is not being unable to close all the questions, it is being deprived of the way out: the panic button.
What does all this have to do with this question, again?
Alright, so the issue is that people ran out of close votes, and we increased the number of close votes people have. Alright! Good work, team. But wait, no, we tried to solve a problem but instead we made two problems:
- Going through the close vote queue is more of a chore than it used to be.
- After going through the queue, I am still out of close votes, and this makes me more likely to stop sifting through crap.
Well, fortunately for us, Shog has an answer:
Also, I recommend not reviewing until you're about done for the day - I used to leave that sort of thing for about 23:00 UTC.
Of course, that’s not really a perfect solution, either:
Not reviewing until the end of the day, dose not work well for people in different time zones. I think part of the problem is that the RISK of running out of close vote is putting people off form reviewing a lot, even when most of them do not run out of close votes.
(Plus, it interacts poorly with the psychological gamification tricks mentioned earlier.)
What am I arguing then? Well, my opinion isn’t really anything new—Félix Gagnon-Grenier’s answer to this very same question already makes my point for me!
TL;DR; We should still have more close votes than reviews.
Josh Caswell had a slightly different solution, but the spirit was the same:
Votes of any kind in each queue should be limited only by the queue's daily task limit; they should not be deducted from one's pool of that kind of vote.
Yes, this is something of a different conclusion from my post a year ago, and yes, this is a bit of a cynical view of the problem. However, I think the stats don’t lie—no matter how the queue is doing now, those questions clearly aren’t so awful that they are truly in need of closing, given that the votes just age away, anyway.
I think it is worth acknowledging that this is not really a technical problem but a social and psychological one, and understanding our (sometimes irrational!) frustrations is important to understanding what exactly the solution needs to be.
Thank you for taking the time to consider these issues, thank you for being willing to hear our feedback, and a big thank you for reading all this and taking it into consideration. It is well-appreciated. And who knows? Maybe it’s time for me to get a little more involved again.