2

Based on past experiences with review audits, specifically in the close votes review queue and the VLQ posts queue, I am proposing a thorough change to the audit system. I know Shog9 has alluded to being open to part of this here already, and we know that people intent on bending audits can cheat the system currently already. Still, I recognize that some sort of audit system is necessary for the review queues to work at all. So I am proposing the following changes to the review queue audit system. These would apply to all review queues.

Proposed changes:

  • A button after a failed audit that is labeled "I disagree with this audit." that requires you to type out a explanation as to why you think your choice was in fact correct. Doing so raises a moderator flag, and site moderators can then review the audit and decide if it still counts against you (It will count against you until a moderator has reviewed the flag) (Max 1 disputed audit per day). When an audit has been confirmed as false by a moderator, it is automatically disqualified from being used as an audit again and will not count against you from that point on. (This specifically would not be possible in the suggested edits queue due to the nature of audits there)
  • A reduced frequency of audits for reviewers who have passed many (and hold steward badges in said queue)
  • A heuristic determining whether or not too many of a user's reviews disagree with the eventual consensus. (At first a warning is displayed, then a moderator flag is raised, if the rate still remains too high a automatic review ban can be triggered)

Arguments for:

  • Robo reviewers are still quickly blocked from doing more harm, and can no longer avoid being blocked by simply cheating audits
  • Moderators get more automatic flags indicating behavior that they may deem worthy of a manual review ban
  • Not every disputed audit has to be addressed in a custom post flag or on meta
  • Legitimate reviewers who get caught by false audits have a well defined and standardized way of recourse (provided they did get caught by a false audit, a moderator will make that decision)

Addressing potential concerns:

  • Possibility to abuse the "Dispute audit feature".

    Due to the nature of the feature (the audits have to be verified false before they no longer count against you) and the rate limit (1 per day) I believe that the possibilities for abuse of this would be quite remote

  • Getting review banned for disagreeing with consensus because there are too many robo reviewers.

    This could definitely be an issue, but I believe that can be solved by sensibly setting the rate and ratio limits of this heuristic.

Final thoughts:

I would like some feedback on the form and merits of this proposed change (preferably backed up with statistics). I genuinely hope we can improve the current audit system (which serves several good purposes but is also flawed in my opinion).

11
  • I like the general approach. Can you please expand on why the button would be labelled "I understand, but disagree.". What is being understood? //// This relates to the current failed audit response that has an "I understand" button, often I could not click that button because I did not understand why my response to that audit was wrong, it was bad audit. Perhaps the new button should just be "I disagree".
    – AdrianHHH
    Commented Feb 19, 2016 at 13:33
  • @AdrianHHH "I understand you select these via a algo so they are bound to not always be correct but I want to have a human make that decision instead". But yes, I can see how that'd be construed wrong, so I edited it.
    – Magisch
    Commented Feb 19, 2016 at 13:35
  • The audit dispute process is poor at the present. I applaud you for trying to make it better.
    – AdrianHHH
    Commented Feb 19, 2016 at 13:43
  • 2
    "•A reduced frequency of audits for reviewers who have passed many" AFAIK, that's already implemented. Commented Feb 19, 2016 at 14:19
  • 3
    "•A heuristic determining wether or not too many of a user's reviews disagree with the eventual consensus. (At first a warning is displayed, then a moderator flag is raised, if the rate still remains too high a automatic review ban can be triggered)" That one is somewhat dangerous, because it penalizes deviation from the mob indiscriminately, whether good or bad. Though I guess it's not nearly as bad as in the suggested edit queue. Before trying to add automatic enforcement, I want a thorough evaluation there. Commented Feb 19, 2016 at 14:21
  • I also suggest that disputed audits count at least twice until and unless a moderator concurrs with the reviewer. Commented Feb 19, 2016 at 14:25
  • @Deduplicator Mind you, that you can only raise one a day and that mods see what you're up to raising them (and your history in that) means that mods may decide to put some more hefty review bans on people who abuse the system.
    – Magisch
    Commented Feb 19, 2016 at 14:44
  • Before I can say whether or not I agree, I think some data is important. 1.) How many failed review audits are their daily? This would be the maximum number of flags that the moderators can get from this system. 2.) How many users disagree with the review outcome on a daily basis? I'm not sure how to properly define this, but simply disagreeing with reviews doesn't mean you are doing it wrong. It may mean that others are doing it wrong, or a combination of both. But, if this is raising mod flags, we need to know how much work it is adding to the already burdened mods.
    – Andy Mod
    Commented Feb 19, 2016 at 14:48
  • 3
    I've posted stats on audits before: meta.stackoverflow.com/questions/292174/… - if you care to earn 10k, you can collect your own. Moderators also have access to fairly detailed stats on reviewer behavior and audits, so I'd recommend paying attention to Brad's thoughts on this. You can skip your usual conspiracy theories; there are no ads shown on audits.
    – Shog9
    Commented Feb 19, 2016 at 15:26
  • @CodyGray See Gimby's comment ;-) Commented Feb 19, 2016 at 15:53
  • 1

1 Answer 1

16

I'm obviously in favor of some kind of a system for disputing audits. I strongly believe that audits are necessary, but it's clear that the system occasionally picks a bad case to use as an audit. These bad audit cases are relatively rare, so it makes sense to identify and deal with them individually rather than scrap the whole system.

In my original proposal, I had suggested something like a moderator-only review queue for contested audit cases. Such a queue and the logic around it might not be the quickest thing to develop, so let me simplify that into something that could be implemented in a shorter span of time.

I propose that when a user fails an audit, they be able to select a "This is an incorrect audit" button and be required to enter a description as to why this is a bad audit. This would then fire off a moderator flag of a new "disputed audit" type, which would appear in our normal moderator flag list.

These flags would use the same flag-handling mechanism we have right now, would be sortable like any other flag type, and could use much of the same logic. When viewed by a moderator, they would show the audit case, the action taken by the reviewer, and give us two options: Remove Audit or Decline.

Selecting the first would remove the audit failure from the user's record, recalculating any review bans that had been applied, and would mark the post to not be used as an audit in the future. Selecting the "Decline" option would leave things as they are and count against the user as a declined flag. If abused, this would lead to a temporary flag ban using the mechanisms we already have in place, and we'd be able to see this in their flag history.

We recently received the ability to redact revisions via a flag-queue-based review process, and that seemed to be implemented quickly by leveraging the existing moderator flag queue. This would work in the same way, with the only new logic being that around marking specific posts as "not to be used for audits" and automatic recalculations of review bans.

I do have some concerns about your other two suggestions: the reduction in audit frequency with number of passed audits and the automatic banning of people who review against the crowd. For the first, careless reviewers who spam reviews will be exposed to many more audits just by pure numbers. We've seen people pass 10-20 audits in a row who still approve obvious spam and vandalism. There's already some weighting in the system, but we have to be careful that we don't make it easier for truly careless reviewers.

To the second point: the problem you're trying to solve here is the case where people somehow game audits and pass them while still reviewing terribly. Based on my manual examination of the review queues over the years, I think this is a very minor issue. Only a tiny number of reviewers ever attempt to circumvent audits in any way, and most of those are people who were paying attention anyways. Moderators can spot truly anomalous reviewers who cheat audits, and we impose severe penalties when this happens. This is not a problem I worry too much about.

There are cases, though, where inattentive reviewing is not being caught properly. I often have to issue manual review bans for reviewers who approve spam when I find instances of it live on the site and see that multiple reviewers approved it. The workflow to manually ban these users is a pain.

In those cases, I would love to have retroactive audit failures where posts that are destroyed as spam yet were approved in the last month by someone would trigger an automatic and immediate audit failure for that user. This would catch many bad reviewers without requiring moderator intervention.

I could imagine a similar capability so that moderators could overturn specific suggested edits and have them count as audits to anyone who approved them. Spam and terrible edits are the most pressing cases where moderators catch bad reviewers, and these would handle most of these incidents. System-imposed penalties for going against the crowd would have too many false positives for me to be comfortable with.

3
  • This answer seems to suffer a little from scope creep :) I see a lot on the wish list for a redesigned proposal with the purpose to "simplify that into something that could be implemented in a shorter span of time" :) All really useful things to have, I'm sure.
    – Gimby
    Commented Feb 22, 2016 at 16:12
  • @Gimby - In the original example, I was talking about adding some kind of special review queue and adding special handling mechanics around removing audits from circulation pending review. This is a more refined proposal that would, from what I've seen, be easier to implement by only requiring extensions to existing capabilities. I've just provided more detail to narrow the specifications for such a feature. The latter half of the answer addresses separate feature requests in the question, and would be independent of the first. I'm merely presenting alternatives there.
    – Brad Larson Mod
    Commented Feb 22, 2016 at 16:26
  • The first proposal-half should maybe be its own feature request? The original on stackexchange is also only an answer with a ton of upvotes, but I guess as an answer it isn't going to attract attention.
    – Gimby
    Commented Feb 22, 2016 at 16:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .