44

So I've got myself a ban from review for reviewing incorrectly. Okay, no worries, I did review incorrectly.

The fact that one of my reviews was incorrect is not in dispute; it was. Nor am I complaining about the mods; that's their job. Nor yet am I asking directly about my own ban; there's only a day left on it.

However, I'm left wondering how, in general, the SO mods handle incorrect reviews when they're brought up by flags or meta posts. Is it common to issue a review ban for the first infraction? Is there a policy of "three strikes and you're out"? Or is there no policy at all?

What's the general policy the mods use to handle incorrect reviews?

6
  • 6
    It could be that the mod involved had the user script which allows all approvers or all rejecters to be be banned in one hit rather than having to ban them all individually and you just caught in the cross fire. Not that this is an excuse - just a reason.
    – ChrisF Mod
    Commented Jun 22, 2016 at 16:19
  • That... would make a lot of sense, @ChrisF. Still, a wake-up call can't hurt.
    – ArtOfCode
    Commented Jun 22, 2016 at 16:21
  • 1
    @ChrisF He's not disputing his ban. He specifically said, "I did review incorrectly" He just asked what the policy is in the abstract.
    – Servy
    Commented Jun 22, 2016 at 16:22
  • 3
    @Servy - I know. It could be that independently of the policy he just caught up in a general ban where a more specific ban (or no ban) might have been more appropriate and I was suggesting a reason why that might be. It's why I posted it as a comment rather than an answer ;)
    – ChrisF Mod
    Commented Jun 22, 2016 at 16:26
  • 2
    You were handed a ban manually by a moderator? I have never heard of them banning someone for one wrong review; usually it's the system that does it automatically after a couple failed audits.
    – TylerH
    Commented Jun 22, 2016 at 23:00
  • 3
    @TylerH Happens a fair bit actually, just the scale of SO makes it seem rare.
    – ArtOfCode
    Commented Jun 22, 2016 at 23:25

1 Answer 1

64

I am increasingly likely to hand out short review bans for blatantly wrong reviews that I come across.

This is primarily motivated by my observation that if I see something bad and do nothing about it then the odds of the reviewer noticing or learning from it are so vanishingly slim that I might as well have not bothered to view the review to begin with.

There are two cases that I'm mostly motivated about right now:

  1. Approving/allowing spam to exist for longer than necessary.
  2. Mis-using "requires editing" on questions that are fundamentally unsalvageable.

It's very easy to find a bad editor or reviewer and then spend all my time for that morning chasing up one single issue.

The scale of the bad reviews problem is large enough that investigating every single incorrect reviewer individually and figuring out if it's a one-off blip or a long term trend really isn't viable. It's simplest and scalable to hand a short review ban to all the incorrect reviews regardless of history. The harm of a 3 day review ban for a misclick vs not banning roboreviewers is strongly skewed towards needing to act on the roboreviewers.

In terms of lengths of ban I typically use 3 days for "I think it was a good faith, but incorrect review" as a minimum to make sure the user has a good chance of seeing the message. (Quite a large percentage of users aren't active over weekends it seems). For the case where there's no possible way that outcome could be reasonable I tend to pick 7 days instead, but that's a crude rule of thumb.

(As a feature request I'd love a mechanism that let me tip bad reviews into the normal automatic audit banning mechanism, counting as just 1 failure but that's lots of dev work from where we are right now. That would be a scalable single click which did the right thing for the circumstances of each reviewer).

14
  • 18
    I agree with this 100%. Not going to post another answer, as it would just say the same thing. Also, +100 on that feature request.
    – elixenide Mod
    Commented Jun 22, 2016 at 16:35
  • Just as a for my information can you guys see if someone is close to a ban? Would such information be helpful if you do not have it already? Commented Jun 22, 2016 at 16:37
  • 2
    @NathanOliver not trivially. We can see if they have previously been banned reasonably easily (but not from /review, requires more clicks although I could probably automate that with userscripts). Figuring out how close they are to the line can only be done by reading through review history and counting. Not my idea of fun. Commented Jun 22, 2016 at 16:39
  • 1
    "Mis-using "requires editing" on questions that are fundamentally unsalvageable" pretty please
    – Braiam
    Commented Jun 22, 2016 at 16:42
  • 1
    @Flexo Thanks for the info. Manual counting is bad. We aren't bean counters we are programmers :) Commented Jun 22, 2016 at 16:43
  • @Braiam Handled. :)
    – elixenide Mod
    Commented Jun 22, 2016 at 16:44
  • @Braiam done - that's exactly the kind of misunderstanding about triage I've been working at. Commented Jun 22, 2016 at 16:48
  • 2
    Maybe new reviewers should have to take a test first to learn the difference. They are all new reviewers. Commented Jun 22, 2016 at 16:54
  • 17
    As the moderator that imposed the ban in this situation, I felt obliged to post an answer; but I'd only be repeating what Flexo has said. But, I would like to stress that the current moderator tooling makes it very hard (read: time consuming and manual) for a moderator to determine how bad-a-reviewer a particular user is. I'd love to see a tool which allows us to easily see the reviews which a user reviewed, where they voted against the final consensus.
    – Matt Mod
    Commented Jun 22, 2016 at 17:26
  • @Matt cheers. If SO has the same review ban tools as other sites, I can sympathise with the lack of tooling :)
    – ArtOfCode
    Commented Jun 22, 2016 at 23:25
  • 2
    Ban shortly, ban often. I couldn't agree more. We see so many blatantly crap edits make their way to approval (sometimes unanimously!); making swarms of robo-reviewers bump into hard and pointy brick walls often could change this at least a bit. Of course this would imply more proper reviewers getting banned due to isolated mistakes (such as that of @ArtOfCode right now), but as the bans are short, this could still be worth while. Commented Jun 23, 2016 at 22:56
  • 4
    "I typically use 3 days [...] as a minimum to make sure the user has a good chance of seeing the message": does it mean that a user is not informed of a ban in any way if he doesn't try to do that? Wouldn't it be much better if those who have got a ban, irregardless of how short it is, were notified? This site does have notifications, and emails can also be sent. Otherwise you can frustrate me with longer-than-needed bans, and I might still entirely miss it if I am not active for some days. This is bad UX. If I am banned from something, I want to know it clearly! Commented Jun 24, 2016 at 17:21
  • "The harm of a 3 day review ban for a misclick vs not banning roboreviewers is strongly skewed towards needing to act on the roboreviewers." Sounds like review is flat-out broken, then, if the impact of bad reviewers is so large. You definitely shouldn't have to spend your entire mod day sussing out the history of a reviewer, and neither should you be backed into counting mistakes as malice because there's too much malice. And if there is that much malice, then this queue isn't helping the site.
    – jscs
    Commented Aug 16, 2016 at 16:45
  • @Matt seeing reviews where somebody voted against the consensus won't help with some cases, especially roboreviewers. Seeing people with ridiculous streaks or ratios (900 approved, 5 rejected) would probably be useful if you don't have it already; I know I've flagged a few posts because I noticed a weird review, checked the user, and saw 99.9% approvals.
    – ssube
    Commented Aug 18, 2016 at 15:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .