Skip to main content
3 of 3
replaced http://meta.stackexchange.com/ with https://meta.stackexchange.com/

I'm obviously in favor of some kind of a system for disputing audits. I strongly believe that audits are necessary, but it's clear that the system occasionally picks a bad case to use as an audit. These bad audit cases are relatively rare, so it makes sense to identify and deal with them individually rather than scrap the whole system.

In my original proposal, I had suggested something like a moderator-only review queue for contested audit cases. Such a queue and the logic around it might not be the quickest thing to develop, so let me simplify that into something that could be implemented in a shorter span of time.

I propose that when a user fails an audit, they be able to select a "This is an incorrect audit" button and be required to enter a description as to why this is a bad audit. This would then fire off a moderator flag of a new "disputed audit" type, which would appear in our normal moderator flag list.

These flags would use the same flag-handling mechanism we have right now, would be sortable like any other flag type, and could use much of the same logic. When viewed by a moderator, they would show the audit case, the action taken by the reviewer, and give us two options: Remove Audit or Decline.

Selecting the first would remove the audit failure from the user's record, recalculating any review bans that had been applied, and would mark the post to not be used as an audit in the future. Selecting the "Decline" option would leave things as they are and count against the user as a declined flag. If abused, this would lead to a temporary flag ban using the mechanisms we already have in place, and we'd be able to see this in their flag history.

We recently received the ability to redact revisions via a flag-queue-based review process, and that seemed to be implemented quickly by leveraging the existing moderator flag queue. This would work in the same way, with the only new logic being that around marking specific posts as "not to be used for audits" and automatic recalculations of review bans.

I do have some concerns about your other two suggestions: the reduction in audit frequency with number of passed audits and the automatic banning of people who review against the crowd. For the first, careless reviewers who spam reviews will be exposed to many more audits just by pure numbers. We've seen people pass 10-20 audits in a row who still approve obvious spam and vandalism. There's already some weighting in the system, but we have to be careful that we don't make it easier for truly careless reviewers.

To the second point: the problem you're trying to solve here is the case where people somehow game audits and pass them while still reviewing terribly. Based on my manual examination of the review queues over the years, I think this is a very minor issue. Only a tiny number of reviewers ever attempt to circumvent audits in any way, and most of those are people who were paying attention anyways. Moderators can spot truly anomalous reviewers who cheat audits, and we impose severe penalties when this happens. This is not a problem I worry too much about.

There are cases, though, where inattentive reviewing is not being caught properly. I often have to issue manual review bans for reviewers who approve spam when I find instances of it live on the site and see that multiple reviewers approved it. The workflow to manually ban these users is a pain.

In those cases, I would love to have retroactive audit failures where posts that are destroyed as spam yet were approved in the last month by someone would trigger an automatic and immediate audit failure for that user. This would catch many bad reviewers without requiring moderator intervention.

I could imagine a similar capability so that moderators could overturn specific suggested edits and have them count as audits to anyone who approved them. Spam and terrible edits are the most pressing cases where moderators catch bad reviewers, and these would handle most of these incidents. System-imposed penalties for going against the crowd would have too many false positives for me to be comfortable with.

Brad Larson Mod
  • 170.3k
  • 114
  • 634
  • 562