36

Over the last couple weeks, I've noticed a single user (a moderator) closing an inordinate amount of questions through the CV Queue every couple of days or so. This is quite obvious because not only are normal users limited to the number of close votes they have per day (50), they are even more limited in the number of close vote reviews they can perform in a day (40).

Now, moderators don't have this limit, because as part of their volunteer role as an exception handler they may need to close more than 40 questions in a single 24-hour period. I think as a mere byproduct of how the system is designed (I could be wrong), this means they aren't limited to 40 reviews per day, either.

While it was a little troubling to see prior days where 1000 or so reviews were done in a single 24 hour period (for multiple days in a row), I am now more concerned than usual about seeing 1500 reviews in a single day (NB - it's true that I've personally petitioned for higher thresholds, but the number I always put forward was merely 80 to 100, never anything as extreme as a thousand or more).

There are historical examples of users unilaterally doing an action en masse that was viewed as problematic, to the point where either the Meta community creates a formal procedure or where the system is altered to alert someone...

Just to be clear, I don't have specific examples of problematic reviews here, nor am I necessarily unhappy that questions, in general, might be getting closed at a slightly more frequent rate recently; I simply have the following concerns:

  1. The sheer amount of reviews makes it more than a little daunting to go through and audit the reviews if the concern ever rises to that level. Combine that with the fact that this is a moderator and folks who are concerned may be more reticent to speak up if they notice an issue.
  2. Likewise, the likelihood of missing something, or just getting fatigued in general while reviewing nonstop for an entire day[1], and then having their overall performance decline, is heightened.
  3. This is a user who is unilaterally closing questions as a moderator in many cases, I have to assume. On occasion, that's not an issue, but for 1500 items in a single day, well... that's a lot of action happening without input from other users.
  4. After so many reviews so many days, the ever-present concern put forth by CMs on why they never raise the threshold (burnout) makes me concerned that the moderator might get burned out and not want to do this (or other) moderation task anymore.
  5. Moderators are exception-handlers. Reviewing questions for closure is not an exceptional task. I'm not sure if it's appropriate for a moderator to (ab)use their exceptional powers for a task not intended to be performed en masse.

For what it's worth, before these ~1000-review-days started occurring, this user had relatively very few close vote reviews under their belt; I might have been less concerned if this user had already performed thousands of CV reviews before starting to do several hundred a day... I'm not sure.

What does the community think about this? Is 1000/1500 unilateral closures/reviews in a day something that should be discouraged?


[1]: Some break-out numbers of how much reviewing this is at different speeds:

  • 1 review every 15 seconds for 6 hours and 15 minutes, or

  • 1 review every 30 seconds for 12 hours and 30 minutes.

    While close vote reviewing can be pretty easy to do reliably at a pretty fast rate in certain situations (e.g. a review every 6-10 seconds), that's only ever been done (and only ever been possible) for 40 reviews at a time... which means you're only reviewing for about 10 to 15 minutes. I think it's much more likely that errors will occur when working at that rate for many hours at a time.

35
  • 44
    I think it is an overall positive for the site.
    – Kevin B
    Commented Oct 2, 2020 at 20:33
  • 29
    I think it's bad for the site. It removes the consensus users must have to close a post.
    – Scratte
    Commented Oct 2, 2020 at 20:47
  • 40
    @Scratte: But we give moderators this ability to overrule consensus knowingly. If you don't trust diamond moderators, you can't trust anyone closing questions.
    – Makoto
    Commented Oct 2, 2020 at 20:47
  • 14
    In all of these cases, something has occurred on that post to get it to the review queue. there's atleast one other person in addition to the mod that felt it should be closed.
    – Kevin B
    Commented Oct 2, 2020 at 20:48
  • 10
    @Makoto Yes, we did. But did we know they would be using that power to close vote 1500 post on a day spending 10 seconds on each all day long?
    – Scratte
    Commented Oct 2, 2020 at 20:50
  • 24
    @Scratte: Is that any different than unilaterally deleting comments, closing questions not in the queue, deleting answers, handling spam, destroying accounts, or anything else? It's all in the name of janitorial work. If you can't trust them with one facet of that, you shouldn't trust them with any facet.
    – Makoto
    Commented Oct 2, 2020 at 20:51
  • 3
    @Makoto It is quite different from deleting comments, which are an entirely different class of entity on the site. It's a little different from the other stuff, for reasons ranging from "it takes a lot longer to go through that many items organically" to "those are inherently black & white actions with little-to-no judgment needed".
    – TylerH
    Commented Oct 2, 2020 at 20:53
  • 12
    @Scratte unless you see specific harm, debating if should be done is rather pointless. They are given unlimited reviews, so it’s expected they can exercise that power. 100 reviews would be fine? 200? 300 hundred? What’s the precise limit? Again, if the net effect is good, what’s there to complain about?
    – yivi
    Commented Oct 2, 2020 at 20:54
  • 5
    @Scratte You take a sample, like 100 posts, and extrapolate Commented Oct 2, 2020 at 21:00
  • 6
    I was also about writing a meta about this but didn't find how to tackle it .. I also found a lot of wrongly closed quesiton by that moderator .. Commented Oct 2, 2020 at 21:12
  • 35
    Thanks @TylerH, your concerns are duly noted and appreciated. I simply just have too much time on my hands at the moment due to being semi-retired (read: unemployed). I didn't rush through the reviews either - you can take the timestamp of the first and last review and calculate an average time taken (but this excludes breaks taken in-between).
    – Samuel Liew Mod
    Commented Oct 3, 2020 at 1:12
  • 5
    wonder what's the point of breaking out speeds at 6 and 12 hours when seeing reviewer history one can easily tell it took them 15 hours, ie 100 reviews per hour. (FWIW I personally would probably do 100 reviews in 20-30 minutes or in 10-15 minutes if I am lucky and there is not too much skip-worthy questions in the queue.) Another thing I don't quite understand is that estimate of prior experience ("very few close vote reviews") seems to be based solely on close queue while triage also provides relevant experience and we know that this user has been very actively involved in triage reviews
    – gnat
    Commented Oct 3, 2020 at 20:34
  • 15
    Thanks for your contributions to the site, Sam!! Commented Oct 4, 2020 at 15:11
  • 4
    @Scratte I think you are wrong about consensus of regular users being necessary for closure. I saw it at some smaller sites how diamond mods actively close and delete inappropriate questions - I even saw cases when this was officially encouraged. I think the only reason why this isn't a norm here is scale: amount of questions at SO is just too high to make it feasible for few moderators to participate in close reviews on a regular basis (and just in case if you get the idea that way out is to get more moderators, ask yourself how would you feel about having to elect 100... 200... 300 of them)
    – gnat
    Commented Oct 5, 2020 at 6:52
  • 4
    Personally I noticed this too and I think it's totally awesome. About time someone took a strong hand around here.
    – matt
    Commented Oct 5, 2020 at 14:35

9 Answers 9

45

Given who it is, I have to assume this is scripted to some degree. Also because, y'know, that's a LOT of reviews to do entirely by hand.

Sam's got a ReviewQueueHelper script, so let's assume that's what's being used here:

  • Keyboard shortcuts for review action buttons
  • Keyboard shortcuts for popup action dialogs (close, flag, delete, reject edit)
  • Auto-focus submit button when option selected in popup action dialogs (e.g.: so you can go "2 > 2 > space" to submit)
  • Automatically skips review audits
  • Esc goes back a previous dialog pane
  • More options available when SOMU Options userscript is installed

Keyboard shortcuts (and Esc, auto-focus) are good - that avoids a lot of potential mistakes and even hand/eye fatigue hunting for options to click. 👍

Auto-skip audits is... Not so great. 😬 Not that the audits in the close queue are particularly good either, but with that many reviews the opportunity for mental fatigue is extremely high.

Some effects I've observed in the past in folks doing this volume of question evaluation:

  • over-reliance on heuristics (grammar, spelling, keywords, the visual "shape" of a question)
  • decision hysteresis (once one action is taken, it stands a higher chance of being applied to the next task, requiring a much better / much worse question to appear in order to trigger a change)
  • plain ol' sloppiness (hitting the wrong key, double-tapping, etc)

Audits aren't the only way to avoid these issues, or even the best ones (time-delays go far). But I'd be concerned about anyone attempting this volume without some sort of fatigue-check in place.

Sam, if you read this: I strongly recommend slowing down and taking a bit of time to implement some fatigue checks in your script.

For others: if you have 10K, you should be able to review reviews. Don't try to review 1500 of them, take samples (10 from one place maybe 900 or so reviews in, 10 more distributed randomly) and check those. That's why the functionality to view the complete review history exists after all!

One thing's for certain: SO is in absolutely no danger of running out of questions to review. Realistically, the answer to this question must come down to whether the effects are acceptable or not - which is gonna require manually auditing the reviews that are being done.

14
  • 12
    (in a monotone voice) You mean to tell me that the CMs didn't talk about what acceptable usage of bots should be from a moderator account? One is stunned!
    – Makoto
    Commented Oct 2, 2020 at 21:11
  • 1
    I've no reason to believe this is a bot; plenty of folks have done variations on review scripts before, going all the way back to Oli's cleanup tool on Ask Ubuntu (which itself inspired aspects of the current review system). There've been loads of discussions on effective / dangerous usage of such things in the past, but... Can't speak to what's been talked about lately.
    – Shog9
    Commented Oct 2, 2020 at 21:13
  • Quality of Life improvements via user script are good; like Shog I don't think this is a fully automated process at all... if it were, then yeah, that'd be a big alarm bell.
    – TylerH
    Commented Oct 2, 2020 at 21:15
  • 1
    You brought up an interesting heuristic in 'the shape of a question'. Sometimes I find myself over-utilizing this as well (Some questions just look well/poorly put together). Do you know of previous discussion on this? It's fascinating from a UX/UR viewpoint.
    – code11
    Commented Oct 2, 2020 at 21:26
  • 4
    Nothing I can point to off hand, @code11 - but it's been known and referenced for years. Shape is a fine heuristic, but like all heuristics it isn't a complete substitute for reading and understanding.
    – Shog9
    Commented Oct 2, 2020 at 21:38
  • 2
    Auto-skip audits is... Not so great. But it's necessary, though. Audits don't scale to this number of reviews, since afaik they don't account for the number of reviews/percentage wrong, and audits can be misleading at times. If you review 1500 posts a day, then I assume even if your error rate is far below average, you'd still fail a considerable amount of audits. I think I'm fairly knowledgeable on the rules, but I'd be surprised if I failed less than 1 audit per 1000 reviews.
    – Erik A
    Commented Oct 2, 2020 at 21:55
  • 1
    Oh I agree @erik. When Ive done large volumes of closing or deleting, my preference is sequence-detection: if I take the same action on 3 or more consecutive posts, it's time for a short break.
    – Shog9
    Commented Oct 2, 2020 at 21:58
  • 3
    That highlights an interesting question... do moderators get penalized/auto-suspended for failed audits like normal users do? Assuming they do, I also assume they could just lift their own suspension...
    – TylerH
    Commented Oct 2, 2020 at 22:08
  • 14
    Sure do, @tyler. Granted, they can also unban themselves... But I've certainly avoided that temptation when I found myself banned - it is kind of a good signal that you should change how you review!
    – Shog9
    Commented Oct 2, 2020 at 22:10
  • have you considered possibility that this user scripted skips (so that complicated cases automatically go under their radar and they manually review only exclusively triage like stuff)
    – gnat
    Commented Oct 3, 2020 at 15:04
  • @gnat How does one programmatically account for 'complicated'? 'If >2 LOC exists, skip'?
    – TylerH
    Commented Oct 5, 2020 at 13:25
  • @TylerH hard to tell, I would expect some heuristics similar to one used by the system to select questions for triage. I don't know its details (and frankly not even interested in knowing) but what I know for sure based on what I observed studying questions in the triage queue, this thing works really well
    – gnat
    Commented Oct 5, 2020 at 13:42
  • 1
    ...great thing about approaching automation from this angle is, one can tune and tweak heuristics to their liking anyway they want because cost of mistake is negligible: "incorrect automatic skip" would be not a problem at all (as opposed to incorrect automatic close which would be rather serious issue)
    – gnat
    Commented Oct 5, 2020 at 13:57
  • That's a very good point, @gnat. Reminds me a bit of when automatic vacuum brakes were first installed on trains, and their selling point was "they work, even when they don't".
    – TylerH
    Commented Oct 5, 2020 at 14:02
66

I sampled and checked 100 of these 1500 reviews (boring sampling details are at the bottom of this answer for those interested).

To me, all of the sampled closures looked deserved (at the time of closing - that is, ignoring edits made to questions after closure).

The vast majority of close reasons looked accurate, except for 4 cases where I could imagine an alternative close option but wasn't sure if it would really be a better match, and 1 case where it felt really off and I would certainly pick a different close reason (given accuracy of reasons in other questions in my sample selection, this one looked more like an occasional misclick).

If the rest of the reviews outside of my sample size are like that, then the overall closing accuracy looks fairly acceptable to me.

As a side note, prior to starting my study I was going to support Shog and second the recommendation to turn off suppressing the audits, because–to me, personally–audits make an invaluable tool to monitor and control focus and quality of my reviews. However, observing the results of this study made me change my mind and abstain from recommending this because, as I wrote above, the quality of reviews looked acceptable as is.


Studying these reviews helped me address another interesting concern raised here, about an insufficient amount of prior close reviews.

What was special about the questions I checked is that all of them looked really clear-cut as close-worthy; very easy to decide. For the sake of completeness, one (only one!) didn't look like that to me, but when I took a closer look into it, I noticed that the reviewer has a gold badge in the question tag, meaning that–to them–it was just as easy and clear as other questions were to me.

Overall, it looked like the reviewer thoroughly skipped all questions that could be a gray area where one could have even a hint of doubt about whether it is close-worthy or not. Given the amount of reviews, I wouldn't be surprised if part of these skips were automated (say by a bot that iterates over available reviews and automatically clicks skip based on some reverse-triage heuristics, so that when one reviews manually, they have most of complicated stuff already filtered out by skips made by this bot).

Okay, now this observation made me recall that there is another review queue where one can gain relevant experience reviewing stuff like that - Triage. The Triage queue focuses on questions of exactly the kind I just checked here.

And then, it just dawned on me that this user has more than enough prior experience in triage reviews, which is really most relevant to the kind of questions I checked. This kind of completed the puzzle by making it totally understandable how they could review these 100 questions with such speed and accuracy.


Sort of a follow up, further discussion in the comments under the question made me wonder if we can somehow utilise experience of this extraordinary review marathon in more routine, typical workings of the review system. And one thing that caught my attention here is how easy it turned out to correctly perform Triage-like reviews—it felt like a miracle when I saw that the accuracy of reviews at hour 15 looked about the same as hour 1.

This naturally brings a thought: what if regular reviewers had an option to choose some sort of "easy mode" in the Close Vote Review queue where the system would only feed them reviews for questions scoring the lowest 10-20% by some quality metric? (The metric(s) currently used in selecting questions for Triage would probably work well enough)

Probably folks using such a mode would be inclined to keep reviewing over a longer term, which in turn could help us solve the old, painful problem of terrible attrition of Close Vote Reviewers,

[...] many users find it difficult to work in review queue. Drop after 250 (silver badge) suggests that even after substantial amount of reviews, many users still fail to discover a way to work productively


Finally, as promised above, here is an explanation of how I sampled reviews for this check.

I went to the Reviews tab in the user's profile (it's public by design; here is an example of how it looks in my profile). I observed that every page in this tab lists 20 reviews, so 1500 reviews would occupy 75 pages or a bit less if some questions were deleted.

I decided that I want to check 100 reviews at 5 more or less uniformly distributed pages. For that, I picked pages 1, 16, 32, 48, and 64 (page 68 and higher turned out listing yesterday's and older reviews, i.e. out of the scope of my check). I opened all links to questions on these pages (total 100) and studied these.

9
  • 17
    Yes, this is the outcome I had expected, glad to see someone did it :) Commented Oct 2, 2020 at 23:06
  • 4
    Nice work, gnat! FWIW, I don't think audits are appropriate for this kind of volume - they're tuned for 1/day at 20 reviews after all; realistically you'd want a bathtub distribution for audits across a large number of reviews. As I said somewhere else, when doing this sort of work myself I went with sequence detection as a cheap defense against hysteresis.
    – Shog9
    Commented Oct 3, 2020 at 1:56
  • 8
    The reviewer in question commented in the chat he went for the "clear cut" cases, which makes sense since it's better to close in quantity rather than waste time on one particular post and let a dozen others "age away".
    – bad_coder
    Commented Oct 3, 2020 at 23:46
  • 1
    I also sampled a few reviews (~30) for manual checking and while I agreed in at least 95% of the cases, I had a few deviations from the moderators reviewing practice. Relatively often I would have chosen "needs clarity" instead of "needs focus" and instead of closing as "asking for recommendations" I would instead simply remove the recommendation from the question if the recommendation was only part of it. Now, this may rather hint at a problem on my side. Anyway, it looks as if the reviewing work was rather high quality. Commented Oct 5, 2020 at 21:57
  • 2
    What seems "clear-cut" to you might not be clear cut to everyone else. To me, "clear-cut" is simply spam. The rest is plain subjective. "Needs more focus"- How much more is more? "Needs details" How much details? Should I explain I clicked This menu button and started typing in this code editor. Where is the line? That's why It's important to maintain consensus. 3 members opinions, who are actually interested in reviewing the question is the minimum I'd consider acceptable.
    – TheMaster
    Commented Oct 6, 2020 at 5:44
  • 1
    Every time new I'm a bit surprised how few people actually edit their questions. Often there are like half baked questions that could be on topic if only some information would be added or clarified but the overwhelming majority of askers simply don't edit their own question. I would guess that they lack the ability to improve their own question, but maybe it's also that they simply don't want to. It seems like a bit of waste of energy, because I guess, they would still like to get an answer to their problem. Commented Oct 6, 2020 at 8:52
  • @TheMaster there's not much subjective in close reasons really. These are indeed determined by consensus. And consensus is in turn decided on meta, in discussions like this one. Moderators are elected to implement this consensus so moderator closure is expected to be per meta consensus (and if one feels it isn't they can challenge it on meta - in discussion like this one). Three votes required when non-moderator users close are merely means to minimize risk that they deviate from meta consensus
    – gnat
    Commented Oct 6, 2020 at 9:15
  • Presenting this thing as my opinion vs your opinion won't accomplish anything because what looks like "my" oipinion to you is in fact primarily based on what I learned from meta. If you really want things to change, you need to either convince people that my understanding of meta consensus is incorrect, or - if you disagree with current consensus - convince people to change it. If you make it I'll adapt - or even if I disagree this won't bother you anyway, because the very nature of meta consensus is that it prevails irrespectively of individual disagreement
    – gnat
    Commented Oct 6, 2020 at 9:15
  • 1
    Well, to add more anecdotal evidence to your hard data: I came across a few more of this mod's closures in today's Reopen Reviews. As before, those in tags I know something about were, IMHO, valid closures. But then I came across one in an obscure, low-traffic tag ... Haskell! After overcoming my initial shock, I decided to skip; but my "awe-level" for the moderator (which was already substantial) went up a few notches. (The question had two reopen votes by the time I came across it, BTW.) Commented Oct 6, 2020 at 15:12
31

If it's discouraged, then we have 1k-1.5k questions which are just left open to languish and may not be of high quality.

I think there would be an actual problem if we had a pattern of demonstrably bad reviews or demonstrably bad choices made during these closures.

Until then...I'll let the process be.

10
  • 5
    It would take longer to review the closures than it takes to close them. So how can anyone find a pattern?
    – Scratte
    Commented Oct 2, 2020 at 20:39
  • 8
    It isn't like there's not an audit trail...if you're concerned about it, then one can look at the audit trail to see if there really are anomalies.
    – Makoto
    Commented Oct 2, 2020 at 20:40
  • 1
    I see. But doesn't one need to demonstrate a failure rate? Meaning one would need to meticulously go through all of them.
    – Scratte
    Commented Oct 2, 2020 at 20:41
  • 8
    Traditionally mods have been discouraged from burning down the close vote queue. That's not a sustainable model of moderation. Commented Oct 2, 2020 at 20:43
  • 7
    @AndrasDeak: Maybe they're bored? If they do this and they're doing a good job even with the volume that they're doing it at, then I really don't see the harm. They're the ones that make the conscious choice to close all of those questions.
    – Makoto
    Commented Oct 2, 2020 at 20:48
  • 1
    @AndrasDeak Do you have a link to a Meta discussion or chat message re: that discouragement where we can read about it?
    – TylerH
    Commented Oct 2, 2020 at 20:53
  • @AndrasDeak [citation needed] we've had a campaign exactly to do this.
    – Braiam
    Commented Oct 2, 2020 at 20:59
  • 2
    That’s it’s not sustainable doesn’t mean it’s bad on itself. Sure, we can’t and shouldn’t depend on mods for this, but if they can and want to help? Great.
    – yivi
    Commented Oct 2, 2020 at 21:00
  • @TylerH I've got two references in chat. Too bad both of them come from me :D Turns out I remembered a specific event on another network. Unfortunately I was hazy on the details 2 years ago.... So yeah, I can see no sign of such a policy on SO. Commented Oct 2, 2020 at 21:12
  • 3
    This, for me, is the most compelling reason. The potential "bad" from a few mistakes (if even worse than the community at large) is not as bad as letting questions age out of the queue that should have been closed. Commented Oct 3, 2020 at 16:54
25

I personally wouldn't want to review for 12 hours straight, but if someone else wants to that's fine. I'm sure that there are much more than 1500 close-worthy questions on the site.

The queue currently sits at 4.3k, so they did the site a favor as far as I'm concerned.

Since this was done in a review queue, you can presumably audit them to see if you think that they were incorrect. Unless there's proof that they were making a large number of mistakes, I don't see what the problem is.

8
  • 3
    I agree that they did a good thing. But consider the fact that a moderator can be wrong since they are human. I feel maybe having another human just making sure that the moderator chose the correct thing.
    – 10 Rep
    Commented Oct 2, 2020 at 20:58
  • 6
    @10Rep True - you can always look at their recent reviews tab and pick out a random sampling to see if you agree with how they're reviewing, though. Unless a large number of their recent reviews were wrong, though, I really don't see what the problem is. Commented Oct 2, 2020 at 21:22
  • @10Rep Also, I think that mods are still subject to review audits (correct me if I'm wrong), which would tend to discourage robo-reviewing. Commented Oct 2, 2020 at 21:23
  • Sure, mods are subjected to review audits, (source). And I amn't accusing moderators for robo-reviewing. But I'm saying they might make a mistake that could get buried somewhere. It's the same reason why mods rarely ever delete posts without a full-fledged reason.
    – 10 Rep
    Commented Oct 2, 2020 at 21:26
  • 6
    @10Rep A moderator deleting a post is a rather different ball game than a moderator closing a post. A mod-closed post can be reopened by normal users but a mod-deleted post cannot be undeleted by community votes (as a delete-voted posts can). Commented Oct 2, 2020 at 21:34
  • "The queue currently sits at 4.3k" - how can I see the numbers? Mind to edit proof link for us?
    – Sinatr
    Commented Oct 5, 2020 at 14:07
  • 2
    @Sinatr If you click on the Review icon on the top toolbar and then click on "All queues", it'll show you how many items each queue has in it. Commented Oct 5, 2020 at 14:08
  • @EJoshuaS-ReinstateMonica, thanks, didn't know that.
    – Sinatr
    Commented Oct 5, 2020 at 14:19
23

Considering the amount of pure unmitigated crap that gets posted to Stack Overflow nowadays, it's not just okay, it's necessary. Curators are expected to manage that Sisyphean workload with the equivalent of sticks and stones; getting upset because they build a tool to help them, and that tool is imperfect, is unhelpful at best and discouraging at worst.

Until or unless we get many more curators or far fewer questions, we can only do what we can do. And that will, sadly, result in false positives. Don't blame the curators for that - blame the company that has forced the necessity of inventing such tools by refusing to provide better tools.

3
  • 12
    Couldn't be more true. After reviewing 1k+ reviews and unilaterally closing the questions myself, there is still the same number of pending close reviews in the queue the next day.
    – Samuel Liew Mod
    Commented Oct 5, 2020 at 13:08
  • 5
    @SamuelLiew, you are one of the undeserved heroes keeping this site alive. Please keep it up, and don't let anyone bring you down over it.
    – Ian Kemp
    Commented Oct 5, 2020 at 13:14
  • 1
    To be clear, I'm not upset that this happened (in fact I agree it's good that, all else being equal, more questions than normal got closed on the site those days), I'm simply concerned from a fatigue/accuracy point of view and curious where the community's thoughts lie on this.
    – TylerH
    Commented Oct 5, 2020 at 13:27
16

For any of these closed questions, the OP has the option to edit the post. This will submit the question to the Reopen Review Queue, where community members can (if they so desire) judge the validity of the original closure (edit history and close reasons are always available, even if not fully shown in the review itself), and cast reopen votes if they feel that either the original question, or the edited/improved version is worthy of such action.

Users with 3K+ reputation can also cast reopen votes on any such closed posts, if they so desire - so there is, IMHO, plenty of scope for potential corrective action, if the said Moderator has been lax or sloppy (but vide infra).

For what it's worth, I do a lot of Reopen reviews (typically, 40 per day, each day, for the past few months). Since this recent spell of "heavy-duty" closures, I have certainly noticed many posts appearing therein that were close by this Moderator: I have yet to come across one that I have felt to have been a poor decision (although I have skipped a number, as I regularly do for questions outside of my knowledge-belt).

So, in summary, I don't think there are any issues in this specific case, but I do (partly) share your concerns over possible "burn-out" for the reviewer.

6
  • 6
    By your reasoning in the first paragraph we don't mind crap robo reviews and audit hackers either, because at worst the questions can be reopened. Commented Oct 2, 2020 at 20:51
  • 2
    @AndrasDeak I certainly didn't mean to imply anything of the sort! Of course there is corrective action available for almost any action taken by anyone. I just wanted to lead into the point(s) I made further into my post. Commented Oct 2, 2020 at 20:53
  • 1
    It may be useful/helpful to get some statistics on how many of these mod-closed questions have received reopen votes, though. Just from an anecdotal/memory point-of-view, it seems to me that almost all those I have come across in the Reopen reviews were sent there by edits (OP or others). Commented Oct 2, 2020 at 20:55
  • 1
    @AdrianMole Any reopen votes due to edits probably would not count as 'problematic closures', at least not reliably. Such statistics would need to focus on reopens without any intermediate action between closure vote and reopen vote.
    – TylerH
    Commented Oct 2, 2020 at 21:04
  • @TylerH Fair comment. Know anybody with SEDE skills? Commented Oct 2, 2020 at 21:05
  • 9
    @AdrianMole I hear if you say SEDE 3 times in front of a mirror rene appears Commented Oct 2, 2020 at 21:07
12

There are historical examples of users unilaterally doing an action en masse that was viewed as problematic, to the point where either the Meta community creates a formal procedure or where the system is altered to alert someone...

Yet there are other examples of the exact opposite:

  • animuson
  • Bill the Lizard
  • Shog9
  • Robert Harvey
  • and I'm sure I'm missing some

But in concrete terms, if the user has more or less the same accuracy as the community at large at closing questions (closed questions that were reopened without edits/closed questions), I don't see any problem with this. The moderator could just as well go through the close queue on the moderators tools and would achieve the same thing... just without filters and other nice quality of life improvements that the review queue includes.

8
  • 3
    I'm guessing you have specific events in mind for each user you listed? Right now I don't see how a list of users is a 1-to-1 response for examples of user events where large quantities of items were handled all at once. Also, Shog was an employee, presumably, for whatever event you're thinking (and animuson might have been, too)... so that's a little different.
    – TylerH
    Commented Oct 2, 2020 at 21:00
  • @TylerH I'm mostly going by memory on this. Sadly the examples I remember can only be identified by screenshots (which doesn't have easy keywords to search for), since the postwithdelete doesn't include any user data that would link the close vote to the user.
    – Braiam
    Commented Oct 2, 2020 at 21:02
  • bummer; can you recall if they were all CV Queue burn-downs? Or various site actions?
    – TylerH
    Commented Oct 2, 2020 at 21:03
  • @TylerH the only one I can pinpoint to a specific campaign was Blil with the burn down the queue. But I've seen Bill going +100 reviews at other times. You may ask it yourself.
    – Braiam
    Commented Oct 2, 2020 at 21:05
  • Sure; and 100 reviews would not be such an odd thing; 1000 is an extra order of magnitude :-)
    – TylerH
    Commented Oct 2, 2020 at 21:09
  • 8
    A note on Bill (and other moderators from that time period such as CasperOne): they had close flags in their flag queue as well, so a massive number of questions got closed just clearing the queue. They used this experience to good effect in review, but their review counts could only hint at the insane volume of questions closed in those early days.
    – Shog9
    Commented Oct 2, 2020 at 21:36
  • 6
    I'm disappointed that you don't have me down in your little compilation of button-clickers extraordinaire. :-)
    – Cody Gray Mod
    Commented Oct 3, 2020 at 10:42
  • 4
    @CodyGray: in contrast, I'm happy to be on the list Commented Oct 5, 2020 at 16:13
10

Participating in a relatively small tag, I'd have to record my opinions here. In tags I'm participating in, during these review queue close spurts, a close vote by one of the users in the tag is usually the final judgment - i.e., the question is closed by a moderator during review. This gives excessive power to a single 3k user's decision and maintains that user's close decision on the question as final.

This creates a unbalanced pressure to close vote queue. If there is an equal pressure in the reopen queue as well, I think everything will be balanced. The questions closed are, on first look, without code. Although questions without code, are sometimes give me teh codez type questions (even that isn't a valid close reason, unless it's too broad as well), sometimes, they are not and are well researched questions. It requires subject matter experts to differentiate and cast votes accordingly.

A single 3k user's decision + a moderator's (who probably knows nothing about the tag) decision based on the "looks" of the question shouldn't count so much. If the same power is provided to a single 3k user's reopen vote, I'd be glad. But, I'd have to cast a vote, convince and gather others in my tag to reopen the question, if it is even possible.

I agree with moderators being exception handlers and better suited to handle exceptions, given their power level.

3
  • Most of your post seems to show concern about this procedure, but then your final sentence indicates you're OK with it. Can you provide a little bit of clarity there?
    – TylerH
    Commented Oct 5, 2020 at 13:56
  • 2
    @TylerH Not sure why you think my final sentence says "I'm OK". I agree with moderators being exception handler = your 5th point: Moderators are exception-handlers. Reviewing questions for closure is not an exceptional task. I'm not sure if it's appropriate for a moderator to (ab)use their exceptional powers for a task not intended to be performed en masse
    – TheMaster
    Commented Oct 5, 2020 at 13:59
  • 1
    Ah, thanks, so there's an implied "rather than turbo reviewers" statement at the end of that sentence that I was not inferring, initially. That clarifies things :-)
    – TylerH
    Commented Oct 5, 2020 at 14:40
9

My first reaction, mostly based on the title of this question, was that there is no way anyone would even be able to do 1500 reviews in a single day without severe fatigue and subsequently lots of errors (or simply doing robo reviewing). But that was coming from a 40 reviews a day limit perspective.

Thinking more about it I changed my mind and think this is actually an eye opener. Doing lots of reviews at once with high quality is possible. I'm very thankful to this moderator for the reviewing work and hope that in the future more reviews can be done by others as well (if they want to).

Here are the details: Additionally to gnat's review I reviewed a few (~50) of the reviews of that moderator manually and I found that they are all high quality. A couple of times I would choose rather "needs clarity" instead of "needs focus" but that might be personal preference and once I could salvage a closed "recommendation" question by removing the recommendation part and leaving something answerable. With an average speed of 20 reviews per ~10 minutes for many hours (!) this person shows that one can deliver high quality reviews in the order of hundreds per day if one is experienced and abstains from robo reviewing.

I'm a bit concerned about the reduced consensus and missing audits. The moderator is basically doing regular curation work here, not real exception handling. The diamond does shortcut the usual 3 close votes threshold. I would feel a bit safer if we could still have the consensus feature also in these cases, where moderators aren't really in their moderating role somehow. It's not urgent but we might not always get such high quality work.

But the main point is that this proves in my eyes that hundreds of accurate reviews per day are possible and that there are people actually willing to do them. In order to accommodate for these people we should increase the limit for trustworthy users (large rep, experienced reviewers, good audit history). A limit of 40 reviews per day is too small, even if not all of the trustworthy users will exceed the limit, some might and therefore it's worth to change it. Take home message: We should trust experienced users (users like this moderator only without a diamond) more. They know what they are doing.

As a small thing to accommodate for different amount of reviewing time during the week, maybe the limit should be weekly, not daily. That would allow for more flexibility.

Last, but not least, the question of long term fatigue and impact on motivation is still unclear. The moderator is just now doing a couple of hundreds of reviews more but we don't know what the long term impact on this moderator is. I hope he will be fine. Part of the 40 reviews a day limit is surely also protection of the reviewers themselves. I'm a bit worried there it might lead to burnout soon and I hope the moderator takes care of himself there.

1
  • 1
    I did waffle a bit on whether to say "a user" vs "a moderator" in the question's title to avoid that confusion, but I erred on the side of "a user", as I really wanted to avoid anyone ganging up on the person due to who they were or the position they were in. The user themself has changed the title to clarify that point (likely because of this answer), so I'm happy to let that new title stay.
    – TylerH
    Commented Oct 6, 2020 at 13:17

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .