17

I have passed multiple review tests because I actually pay attention, try to be objective, help new users and so on. Sure I am new to reviews and have to learn to make less mistakes. However there is another mechanism that suspends your review privileges for several days if you take a wrong action on normal post. I have not failed a single of quite many audits, but still got suspended my privileges after doing a single mistake (at least there was only 1 post in explanation why I was suspended). Of course I understand that suspending the privileges is here to help new reviewers like me to better understand what action I (we) took wrong and why (and I did), but my question is following:

Why the mechanism doesn't count "positive points" for successfully done audits and doesn't take it in consideration when suspending review privileges, i.e. shorter time of suspension?

Something similar was proposed in this post with failed/passed audits, however the only reply was kind of weird...

Thanks for any thoughts!

3
  • 6
    Hmm, no, users do not get suspended for a single review audit failure. It takes many, with an appropriate warning before the hammer comes down. I'd have to guess it was actually this one, participants are supposed to learn what "requires editing" means. Did it work? Commented Oct 2, 2020 at 22:05
  • @HansPassant "Many" being 3? I do not think any warning it given other than the message one gets when failing an audit and knowing that one or two more within the next 30 days is going to make the hammer fall. Btw: You didn't link to an audit. But if I remember correctly having been review suspended within the past 30 days or so may make the hammer fall on just one failed audit.
    – Scratte
    Commented Oct 2, 2020 at 22:10
  • 2
    @HansPassant I do not talk about audit failure but regular review, I have not failed audit at all, yes you are right this was the one I have failed and of course I have understood why it was incorrect, hopefully the rest understood it too. However I got no warning, just got suspended and my question was why for same period as maybe some robo-reviewers or people who just want their badges and do not care. Well it might me more failures from me, the system should point it out then, shouldn't it? How am I supposed to learn if not? :)
    – Ruli
    Commented Oct 3, 2020 at 5:35

3 Answers 3

13

The feedback given by the audits, particularly their failures is presumably meant not to be solely punitive. Instead, its meant to provide feedback as to whether you understanding of the review being done matches that expected.

The boundary between good and bad is nontrivial in many cases and audits provide feedback at scale not available if you don't engage in alternate moderation guidance (Meta, chats etc.)

The review ban part of it is meant, I believe to reinforce that lesson, especially as its easy to get defensive about your wrong review (And I would know. I just got back from a review vacation). Having a period of time to review your decision allows a less emotional evaluation of the question.

So with all that being said, if you were to counter balance the audits and not give out review suspensions, I believe one might be more encouraged to not correctly learn and to ignore wrongly marked reviews.

EDIT: Looking back at the original question, I see one aspect I haven't addressed, namely, that the failed audit vacation duration be connected in some manner to the number of correct audits.

I believe that this could be effective. Consider the following situation: a reviewer has a 99.5% correct review rate, which corresponds to a 1/200 fail rate or, if they are doing 40 reviews a day, one per work week.

If you assume the absolute worst case scenario, that the one failure is always an audit, under the current system, after a few weeks (I forget the exact escalation algorithm), he will have racked up suspensions on the length of months and will no longer be able to contribute.

I think this is wrong because the ratio of bad questions to good ones is very much against us. We have currently set the sliding scale to the literal absolute extreme in favor of allowing hundreds of bad questions at the cost of one good one (and probably just a grudgingly acceptable one, if there's enough ambiguity to trip up a knowledgeable reviewer)

TLDR: Audits exist for a reason and are a fundamentally important tool in keeping moderation quality high, but maybe the suspension duration should be based somehow on number of correct reviews as OP suggested.

4
  • 6
    This is accurate; in fact, bans were not initially coupled to audits at all.
    – Shog9
    Commented Oct 2, 2020 at 22:00
  • 1
    Users habits (perhaps even account ownership) can change over time! There should also be some window of consideration.
    – ti7
    Commented Oct 5, 2020 at 16:13
  • 1
    @ti7 Thats a good point. I'd arbitrarily suggest a trailing year for window size. Account ownership is relevant, but seems somewhat out of context. I don't know that SO could track that effectively.
    – code11
    Commented Oct 5, 2020 at 16:35
  • @ti7 Good point. You gotta believe reviewers get better overtime with the review process. Those few initial bad choices keep coming back to haunt you. (on a looong review vaca) Commented Oct 20, 2020 at 14:55
5

Audits are not a "just try to keep your numbers positive" metric; they're a 'stop the line' policy.

Liken it to healthcare and a surgical environment: in the first case, it doesn't matter how many mistakes the surgical team makes so long as they do more good/correct steps than wrong steps. Who cares if you cut two major arteries if you sewed three flawless stitches?! You made more good actions than bad! Well... your patient ends up dead that way.

Instead, such teams usually have a 'stop the line' policy where the team (anyone in the team) "stops" the process at any point when they see any dangerous/bad thing happening. It's critically important that the surgeon not nick/cut a single artery by mistake even once.

So, likewise, it's most important that you not review an item incorrectly and allow bad signal to propagate (e.g. "x post or x kind of post is good/fine!"), than it is for you to simply "get more right than you get wrong". Every reviewer should get way more right than they get wrong... because it shouldn't be that hard, and one should be paying enough attention to each review to ensure that, too.

7
  • 2
    I like your analogy - Surgery is sorta similar to reviewing - one mistake can be a huge thing.
    – 10 Rep
    Commented Oct 2, 2020 at 23:18
  • 5
    And isn't that my case? I have failed one review (according to system report), got tens correctly, my failure rate is maybe 2%, will decrease I hope, what is "too much" mistakes then? one out of 100? I am just trying to say that maybe the punishment might not be the same for someone who does not pay attention at all than somebody who makes a mistake, its hard to compare with somebody who has your life in his hands, I won't kill anyone :)
    – Ruli
    Commented Oct 3, 2020 at 5:45
  • 1
    @Ruli You don't get suspended for one failure. There are other failures at play here or you were manually suspended by a moderator due to that one failure being bad enough to warrant it. I don't think even two recently consecutive failures is enough, if you follow it with enough successes and normal reviews. Review bans are also not permanent (at least not initially). Reviewing is an important task and a privilege; getting suspended for failing an audit is not the site saying "you're fired", it's saying "go take a break. Come back in a while when you've got fresh eyes/perspective".
    – TylerH
    Commented Oct 5, 2020 at 14:32
  • 2
    @TylerH I think your answer and comment phrase the intention of the feature very well. The only question I'd ask is, given a person has a proven track record of good reviews, and that they've already entered break-mode for at least a day or two to think things over, is there really benefit gained from lengthening that out to a month? Or by doing so have we swung the pendulum too hard in the other direction and reduced the number of active reviewers helping curate the site?
    – code11
    Commented Oct 5, 2020 at 19:38
  • @code11 There is an issue with your premise: "given a person has a proven track record of good reviews"; A user's review history is not checked for accuracy before a ban is doled out. In some cases, specific problematic reviews may have been what initially alerted a moderator to the situation (e.g. in the bad reviews chatroom). I don't know exactly what is checked by the system before instituting a review ban, but I believe it is just "how many audits have been failed out of the last n audits" or something like that.
    – TylerH
    Commented Oct 5, 2020 at 19:47
  • @code11 (continued) Regarding the duration of the bans, that's also above my pay-grade. I assume the intention of a long(er) ban is to impress upon a user the importance of paying closer attention. I do recall that the bans increase in length as they get applied. E.g. you get banned for a day, then a month, then a couple months, then a year, etc.
    – TylerH
    Commented Oct 5, 2020 at 19:48
  • 1
    @TylerH I believe it's 4 days and doubling every time if it's moderator imposed. If one has only been suspended by the system for failing audits, I believe the duration resets after 30 days, but I'm not sure. I also think it's 3 audits in a 30 day period for the first suspension, but coming out of a suspension, it's just one in the following 30 days period.
    – Scratte
    Commented Oct 5, 2020 at 21:18
4

In theory, you shouldn't fail any audits. At least in theory, they're designed in such a way as to be easy to review. (In practice, of course, that isn't always the case).

One thing I'll sometimes do is, if something looks like it might be an audit, you can open it up in another tab to see if it is. If it is an audit, it'll either have a lot of upvotes (if you're supposed to review "looks OK") or it'll have lots of downvotes (and likely also be closed and/or deleted) if you're supposed to review that it should be closed.

1
  • 4
    I know you also qualify this quote, but I'll add that the "you shouldn't fail any audits" is a draconian and detrimental way of operating the system. The reality is that everyone makes mistakes. If we optimize for finding that one somewhat mediocre but allowable question, we have let in thousands of really poor ones.
    – code11
    Commented Oct 3, 2020 at 16:39

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .