6

ok, so I have been reviewing posts on stack overflow for the better part of a year or more. Lately, I have noticed the audit posts have become much more arbitrary. For example, I saw this post and after carefully reading the question, and considering the comments, and considering my extensive knowledge of C# and .net, voted to close because it was unclear what that poster was asking. Several people posted answers, to no avail! This is exactly why we vote to close questions that are unclear!

Now, I am review banned for 7 days. After one failed audit.

Voting to close is a matter of opinion. The fact that I disagreed with the rest of the people who originally left this open eons ago does not mean I should be banned from reviewing. If audits are going to be conducted in such an arbitrary manner and with such severe consequences, why waste time reviewing posts?

What can be done to change this? Particularly, how can we ensure audit posts are not subject to opinion but are black and white?

8
  • 1
    I'm quite surprised by your getting review banned (at least automatically) for one failed audit. That seems … implausible, but it's not particularly easy to skim through and find any previous audits. (At least not for me.) And a manual ban in 5-10 minutes is still less likely. Commented Jan 24, 2015 at 4:51
  • 4
    The review ban takes into account all your review audits from the past 30 days. You just came off a 2-day review ban a day ago, and your very first action back in review was to fail another audit, so the system banned you again.
    – animuson StaffMod
    Commented Jan 24, 2015 at 4:53
  • 3
    I only failed one audit there too, and it was likewise a matter of opinion. Audits also should not be difficult nor designed to trip you up. I have been involved in numerous audits of aerospace procedures which has potential life or death consequences. An audit should be designed to tell if you are following the procedure, not if you arrive at the right answer. People are going to get it wrong every now and again.
    – theMayer
    Commented Jan 24, 2015 at 4:55
  • 2
    If you've been review banned, the ban is lifted, and you fail the very next review, then yes - you should be review banned again. Imagine this discussion with your child: Daddy, I bit my sister, and you scolded me. I bit her again, and you put me in timeout. When I got out of timeout, I bit her again. Why did you put me back in timeout? You can fail a single audit if you're coming off a ban and somehow fail the subsequent audit. Every question on any Stack Exchange site is somewhat "subject to opinion*, and if the site were otherwise the entire architecture would be invalid. Close, up, down: opinion.
    – Ken White
    Commented Jan 24, 2015 at 6:51
  • 1
    @animuson I had exactly one review ban in all my SO time, very recently. And it came after I failed one review test,. I had failed one other, weeks ago. So that makes 2, of many many reviews within several months. Not that I'd say the review process is overly strict - actually the opposite, when I see what kinds of edits and votes go through... the worst review area: questionable but HELPFUL and correct answers to questionable questions, like those asking for a link. Why punish someone who refuses to delete a HELPFUL reply? Delete the Q by all means, but don't punish the answer!
    – Mörre
    Commented Jan 24, 2015 at 8:56
  • 2
    @KenWhite - Nobody is biting someone's sister! Good grief. We're talking thousands of posts up for review, each getting a minimum of 5 votes. There is room for variation, and if the system doesn't allow for it, it is broken. And my point is that we can create an objective audit system that evaluates process, not outcome.
    – theMayer
    Commented Jan 24, 2015 at 12:52
  • Certainly there's room for variation. You missed the point, which is that if someone habitually fails audits, they're doing something wrong. If you failed enough times to be banned, the ban is lifted, and you immediately fail another review, changes are quite high that you are repeating the same behavior that caused the previous ban and that the lesson didn't take. There are a lot of users and a lot of reviews, and sometimes you have to go with the percentages. Banned-released-failed next review is a high percentage of bad reviews, IMO.
    – Ken White
    Commented Jan 24, 2015 at 18:29
  • 1
    @KenWhite have you even bothered to look at the audit I supposedly "failed"?
    – theMayer
    Commented Jan 25, 2015 at 14:12

2 Answers 2

12

I've talked about this before, but I don't believe the solution to debatable audits is for them to be handpicked by moderators or staff. That simply would not scale to a site of this size, particularly as the number of users and posts increases over time. We would not have enough time in the day to pick out the number of audit posts required for this. Also, the number of audits that are controversial account for a small fraction of the total number of audits applied.

I'm sounding like a broken record, but the way to combat questionable audits is to put in place a system for disputing them. This would let us focus on the relatively rare bad audits and remove them from the system, rather than take all our time hand-picking good audits.

3
  • Your previous post on the subject is spot on. That way the more experienced community members can decide what is too subjective and if it should result in a review ban.
    – theMayer
    Commented Jan 25, 2015 at 14:18
  • I've already harped on this a bit myself. Far too many of these 'relatively rare' audits have no business being in the audit queue. I don't believe that the current system has any effect whatsoever on the quality of the reviewers especially as the system is so easy to game anyway. Disputable audits seem to be the only way for the audit implementation to actually honor its basic charter: To improve the quality of the review system overall.
    – ouflak
    Commented Aug 5, 2015 at 8:18
  • @ouflak - On the contrary, I've seen the audits have a huge impact on abusive reviewers. There are currently 185 people banned from review on Stack Overflow due to audits, and every one I checked had earned that ban by spamming approvals on spam, vandalism, or otherwise terrible posts. That's just today, with most of the people who are banned correcting their behavior after they get a ban. Thousands of abusive reviewers have been stopped by audits, which dwarfs the number of people who have complained about bad audits on Meta. Providing a mechanism to remove bad audits will resolve the latter.
    – Brad Larson Mod
    Commented Aug 5, 2015 at 14:19
4

We cannot do anything to remove opinion. Moderators also have opinions, even. And even if no one would ever disagree with us, we don't have time to do that.

Also, though; this is the second time you've been review banned in just a few days. The last one you failed is iffy, ill agree... But the one that triggered your previous ban wasn't remotely questionable. I'm on mobile, so it's hard for me to look further, but there's that...

6
  • I couldn't see the one I previously reviewed, it showed up as blank for me. Perhaps it was a bug? But I do remember seeing a difficult post and thinking it could go either way. But really, that is not the point here.
    – theMayer
    Commented Jan 24, 2015 at 4:58
  • i do appreciate the effort to look, but I think we may want to rethink the audits to make them more effective. It does seem like something has changed in the last week though I have no idea what.
    – theMayer
    Commented Jan 24, 2015 at 4:59
  • It is the point, though. Your first review on returning from a ban was failed, too. It strikes me that is a pretty reasonable indicator. And you couldn't see that other post you failed a review on because it had been deleted...
    – Andrew Barber Mod
    Commented Jan 24, 2015 at 5:00
  • Well if that is how they want the system to work I think it is poor strategy.
    – theMayer
    Commented Jan 24, 2015 at 5:03
  • I think you do seem to have fallen into an edge case here. It does seem like you review well, from what I've seen. Probably just some bad luck. :( @themayer
    – Andrew Barber Mod
    Commented Jan 24, 2015 at 5:07
  • I mean, I think I'm a good reviewer, but not perfect. Oh well. I'm just a little frosted.
    – theMayer
    Commented Jan 24, 2015 at 5:07

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .