Based on past experiences with review audits, specifically in the close votes review queue and the VLQ posts queue, I am proposing a thorough change to the audit system. I know Shog9 has alluded to being open to part of this here already, and we know that people intent on bending audits can cheat the system currently already. Still, I recognize that some sort of audit system is necessary for the review queues to work at all. So I am proposing the following changes to the review queue audit system. These would apply to all review queues.
Proposed changes:
- A button after a failed audit that is labeled "I disagree with this audit." that requires you to type out a explanation as to why you think your choice was in fact correct. Doing so raises a moderator flag, and site moderators can then review the audit and decide if it still counts against you (It will count against you until a moderator has reviewed the flag) (Max 1 disputed audit per day). When an audit has been confirmed as false by a moderator, it is automatically disqualified from being used as an audit again and will not count against you from that point on. (This specifically would not be possible in the suggested edits queue due to the nature of audits there)
- A reduced frequency of audits for reviewers who have passed many (and hold steward badges in said queue)
- A heuristic determining whether or not too many of a user's reviews disagree with the eventual consensus. (At first a warning is displayed, then a moderator flag is raised, if the rate still remains too high a automatic review ban can be triggered)
Arguments for:
- Robo reviewers are still quickly blocked from doing more harm, and can no longer avoid being blocked by simply cheating audits
- Moderators get more automatic flags indicating behavior that they may deem worthy of a manual review ban
- Not every disputed audit has to be addressed in a custom post flag or on meta
- Legitimate reviewers who get caught by false audits have a well defined and standardized way of recourse (provided they did get caught by a false audit, a moderator will make that decision)
Addressing potential concerns:
Possibility to abuse the "Dispute audit feature".
Due to the nature of the feature (the audits have to be verified false before they no longer count against you) and the rate limit (1 per day) I believe that the possibilities for abuse of this would be quite remote
Getting review banned for disagreeing with consensus because there are too many robo reviewers.
This could definitely be an issue, but I believe that can be solved by sensibly setting the rate and ratio limits of this heuristic.
Final thoughts:
I would like some feedback on the form and merits of this proposed change (preferably backed up with statistics). I genuinely hope we can improve the current audit system (which serves several good purposes but is also flawed in my opinion).