51

Every now and then someone on Meta complains about a failed audit, this is the latest one, and most of them have the same answer : you did the right thing, the question is indeed bad but it looked good to the algorithm because of upvotes/views/other criteria I don't know about.

Now, most of us know that the audit system isn't perfect and sometimes audit questions/answers are badly selected, maybe we should include that info in the failed audit messages, something that tells the user that failing an audit isn't a big deal if their overall review stats are correct, so that they just let it go instead of coming to Meta and posting Yet Another "why did I fail this audit?" Question.

15
  • 12
    Or even better, put a human review on the audits so it doesn't clog up Meta. Commented Oct 23, 2014 at 19:07
  • 1
    @BradleyDotNET in the long run that's a better solution but that requires a bit of development (a separate review queue, etc), in the meantime changing the failed audit message will take only a few minutes and is still better than nothing.
    – user2629998
    Commented Oct 23, 2014 at 19:10
  • 4
    @BradleyDotNET And then audit the audit reviews? Commented Oct 23, 2014 at 19:11
  • 9
    @ThisSuitIsBlackNot Put it in a high-rep review queue. At some point, you have to trust the users reviewing the audits. Or perhaps, if the audit gets enough "crap" votes, raise a custom mod flag so a really trusted user gets the final say. Anyways, this feature request is already on Meta, it should be discussed there. Commented Oct 23, 2014 at 19:14
  • 3
    @BradleyDotNET I'm pretty sure there are high-rep robo reviewers.
    – user2629998
    Commented Oct 23, 2014 at 19:15
  • @AndréDaniel I'm sure there are. Hence the "if we are really concerned, give a moderator the final say. I won't pretend I have the perfect system, but there needs to be a way to challenge audits that doesn't clog up Meta, or take too much time out of the moderator's busy days. Commented Oct 23, 2014 at 19:17
  • 5
    @BradleyDotNET: What do you say to that: meta.stackexchange.com/questions/188780/… Commented Oct 23, 2014 at 19:21
  • @Deduplicator That would be the original feature request I was referring to. I wasn't aware that it was migrated (or old enough to be when the split happened). I'm still hoping it happens :) Commented Oct 23, 2014 at 19:22
  • 4
    A friendlier message does not solve the problem. If you do a lot of reviewing, and happen to run into a couple of bad audits, you get review banned. So failing audits is considered a big deal. I stopped reviewing entirely (at least for now) after my second ban, and IMHO I failed only one audit where I probably should have voted differently. Commented Oct 24, 2014 at 5:21
  • 1
    If you have a feature request, write it up properly. "Do something" is just a rant, inviting more rants, under the guise of discussion.
    – Air
    Commented Oct 24, 2014 at 18:21
  • 2
    @AirThomas besides the title (even though I don't see it as a rant either), does my post look like a rant ? I don't think so. And it's not a feature request since I prefer to let the community discuss and propose other solutions rather than "impose" my solution and just get "I agree/disagree" responses.
    – user2629998
    Commented Oct 24, 2014 at 18:24
  • @AndréDaniel The top-voted answer begins with, "I hate the current audit logic" and goes on to suggest a change that the author claims, absurdly, "would require no additional work." The proof is in the pudding, in my opinion. Whether you want to call it a rant or not. (And, no offense intended, by the way.)
    – Air
    Commented Oct 24, 2014 at 18:37
  • Bad "reopen audit review" question. So "Nothing" is wrong with a question that is clearly a beginner javascript dupe. Upvoted because, yeah, it would be cool to use sleep() in JS. Demonstrates lack of googling for dups and broad ignorance of event driven model. To STOP! LOOK! and LISTEN! after hundreds of expert human reviews when SO programmatically chooses the audits -- inviting bugs -- is laughable and hurts morale. Audit questions should require strong human review to become audits, then lecture us.
    – Paul
    Commented Oct 24, 2014 at 19:18
  • "maybe we should include that info in the failed audit messages, something that tells the user ..." - that might work until you get banned for too many failed audits.
    – jww
    Commented Oct 25, 2014 at 12:52
  • I was about to ask such a question until I met this one :) What worries us is that we might get punished by a bad algorithm.
    – Adelin
    Commented Jun 15, 2018 at 10:37

3 Answers 3

54

Before I get into my proposed solution for bad review audits, I should state that bad audits are like plane crashes: they are very rare compared to the number of audits used, yet we think they're common because a big deal is made when they happen. I have looked over many, many audit cases since these were put into place, and only a tiny fraction of audits are bad cases.

Therefore, I don't believe the best solution would be to completely rework the way that audits are defined, but to instead handle the rare bad cases themselves. We really don't want humans hand-picking audits, since that won't scale (if done by moderators or staff), could be prone to abuse (if done by the community), and could lead to a whole new class of arguments on Meta if questionable cases were chosen for audits by people.

I still think the best way to deal with bad audit cases is what I describe in this answer. Give reviewers a means of contesting an audit via a button on the audit failure interface. This would then go into a queue for moderators (and the SE team) to review and give a yes / no as to whether an audit was proper or was a bad case. If marked as a bad case, it would be removed from circulation and the reviewer would have that failure be removed from their record (as well as any ban it triggered).

This would allow the community to identify the rare bad cases in a way that doesn't require a Meta protest, and it should be easy for moderators to tell at a glance if something was a good or bad audit case. Anything that takes any amount of thought for a moderator to tell if it was a good audit should be removed as an audit as a default position.

Someone abusing these disputes should be subject to a longer review ban, to avoid bad reviewers trying to annoy us by disputing all legitimate audits they get.

No disputing should be allowed for system-generated suggested edit audits. These are all gibberish, and if you approve them, you have no argument for doing so.

If such a system was implemented, it would eliminate almost all Meta audit protests overnight, give good reviewers a means of identifying and countering bad audits, all while making sure audit generation scaled with the site. It may even give SE information about audit generation techniques with high percentages of failure, and allow them to tweak audit algorithms based on that.

10
  • 6
    "No disputing should be allowed for system-generated suggested edit audits. These are all gibberish, and if you approve them, you have no argument for doing so." Is it really impossible for the system to accidentally generate an edit audit that looks legitimate? Thousand monkeys, thousand typewriters scenario?
    – Air
    Commented Oct 24, 2014 at 18:54
  • 9
    Allow disputing gibberish edit audits, as a honey pot for really disingenuous or incompetent reviewers. Then if they are arguing their review of gibberish or spam was OK when it clearly wasn't, put them out of their misery for good. Let people with gold audit badges review the failed and challenged audits somehow, I'd gladly do it.
    – Paul
    Commented Oct 24, 2014 at 19:36
  • 1
    @Paul If as Brad says bad audits are very rare, I'd argue against bringing them down to gold badge holders. It's too easy for robo reviewers to get the badge without having a good understanding of what is/n't acceptable on the site. At a minimum I'd suggest limiting it to 10k users and above. Commented Oct 24, 2014 at 21:00
  • @DanNeely That also suggests an issue with the badges to make them more useful rather than simply decorative. You can get to 10k without ever reviewing anything, and that is fine... but doesn't expose you to the realities of the janitorial side.
    – Paul
    Commented Oct 24, 2014 at 21:11
  • @Paul It takes a lot more effort to get to 10k, so there should be a lot fewer idiots; and I have trouble believing someone could earn at least 8k on questions or answers without figuring out what is generally considered acceptable practice. Commented Oct 24, 2014 at 21:19
  • 2
    @DanNeely "I have trouble believing someone could earn at least 8k..." It's well within the realm of possibility. Here's an extreme example.
    – Air
    Commented Oct 24, 2014 at 22:24
  • 1
    @DanNeely Not that I necessarily disagree with your conclusion. There are very few users with >=8k rep and <20 questions and answers on SO, for example.
    – Air
    Commented Oct 24, 2014 at 22:37
  • 1
    This makes so much sense that I wonder why it hasn't been adopted. Is there any objection? I don't participate on MSE enough to see the split votes on the original answer, but it's at +215, which looks like strong support. Is it just a questions of priority? Is there anything more we can do to support the proposal, beyond upvoting your answers? Commented Oct 25, 2014 at 16:15
  • 1
    Is there a concrete feature-request to further this excellent suggestion?
    – tripleee
    Commented Jun 10, 2015 at 9:58
  • 1
    @tripleee - meta.stackexchange.com/questions/188780/…
    – Brad Larson Mod
    Commented Jun 10, 2015 at 13:32
4

Suppose each item chosen for an audit review were always sent to 200(*) people including a few with high reputation. The review process works as at present but the 200 results are collected and if over 67% pass no further action is taken, it is deemed to be good review. If under 33% pass then it is a bad review item, so remove the failure from peoples review history, or at least exclude it from review banning decisions. If the passes and fails are approximately equal then get the moderators to examine the item and choose which of the previous choices is correct, but exclude that item from future audits.

(*) The numbers 200, 33% and 67% are just suggestion.

5
  • In a way this is already the case with the current reviews. I wonder why reviews audits are not chosen from the review results as to keep people on the mainstream track. I always thought it already works like that. Commented Oct 24, 2014 at 13:54
  • @Trilarion probably because one of the problems the review system was intended to fix was the number of idiots approving crap they shouldn't've been. Commented Oct 24, 2014 at 23:14
  • 2
    This proves that people agree with each other, not that the audit is bad.
    – Ben
    Commented Oct 25, 2014 at 13:05
  • @DanNeely Then you could use this idea the other way around. Manually chosen audits that are often done wrong (say we test them on the first 200) people are either manually revisited and deleted or a better indication what went wrong is given after the audit failed. E.g. "This was an audit. You failed. This audit is often failed. The most probable reason is...." Commented Oct 26, 2014 at 12:11
  • How about instead of removing it, the system brings it to the attention of moderators/employees to review the audit in question and remove it if necessary? Commented Oct 26, 2014 at 13:52
-7

A very partial solution addressing only a tiny part of the problem:

Users that have a gold badge for a given review queue should be exempt from audits for that review queue. They've proven their worth and have zero incentive to do crappy reviews.

If they did robo-review their way to 1000 in order to get a gold badge, then they have no motivation to continue robo-reviewing beyond 1000. Either they will stop (why spend more time robo-reviewing for no gain) or they will continue reviewing out of interest for SO, meaning they weren't robo-reviewing in the first place.

Sure, there are edge cases, but by and large these are the overwhelmingly likely patterns.

4
  • 5
    All that means is that they did 1000 reviews. The reason audits exist is because of robo-reviewers trying to get that badge. How have they proven their worth? Commented Oct 24, 2014 at 20:42
  • 1
    While they may not intentionally engage in crappy reviews, they may still unintentionally make mistakes. For instance, they may start unconsciously drifting in their evaluation of posts (e.g. the line between spam and not spam may change so that they start letting too many things through or marking as spam too much stuff). Also, the rules established by the community as to what should be deleted and for what reason changes over time. The audits help keep people in line. And it also seems to me that experienced folks would be the one most likely to correctly identify Brad's problematic cases.
    – Louis
    Commented Oct 24, 2014 at 20:47
  • @BradleyDotNET: If they did robo-review their way to 1000 in order to get a gold badge, then they have no motivation to continue robo-reviewing beyond 1000. Either they will stop (why spend more time robo-reviewing for no gain) or they will continue reviewing out of interest for SO, meaning they weren't robo-reviewing in the first place. Sure, there are edge cases, but by and large these are the overwhelmingly likely patterns. Commented Oct 25, 2014 at 15:35
  • 1
    @Jean-FrançoisCorbett: there are also those who genuinely believe they are helping, but their opinions on what is suitable and what is not significantly deviate from the community norm (e.g. everything except spam is an answer). Commented Oct 26, 2014 at 13:54

You must log in to answer this question.