TL;DR: We need measures against mass-robo-reviewing since the Stack Overflow Documentation Review queue™ has been introduced.
This is a follow-up to What can we do about robo-reviewing on documentation?
General issues
Introduction
I hadn't really been able to track this on a global scale (i.e. not only in a specific tag), but yesterday the review queues have been introduced, causing a huge surge in reviewers and probably also robo-reviewers. It also provides (for 10k+ users currently; users with less reputation will only see their own reviews there) a history which easily shows many reviews by a single user in a very short time frame, making it easy to spot trivial click-through robo-reviewing.
There have been hundreds of reviews alone yesterday which were blatantly robo-reviewing.
I flagged nine people yesterday, just looking at the review history from 19:00 to 24:00 UTC. They all showed only one to six seconds delay between their reviews - I do look at these changes, they're not only typo fixes where I could believe a six second delay, but full-blown edits.
There also have been people with longer delays, but approving very large code additions within 20 seconds, e.g. [just a random example spotted in recent review history - I do not know whether Approve would have been the correct choice or not - point is, the review must have been superficial]. Could you really review, check for correctness and plagiarism in these 20 seconds? I notice these reviewers sometimes rejecting the change proposals then usually very obviously bad/defacing the example or such.
Extrapolating this to the full review history, they probably make up 25%-40% of all the recent reviews since review queue addition, invariably leading to many change proposals being only reviewed by robo-reviewers(!!!). That's totally defeating the whole purpose of a review queue, which is preventing bad content from entering and then going unnoticed.
Example: This change has been reviewed by three people who were all doing their reviews in something less than eight seconds between their reviews (as obvious from their Documentation tab) - the lone rejecting person was quickly overruled. Good that the regex tag is more active and there are people looking at the approved and rejected history and it could be quickly rolled back but many tags are less active and robo-reviewed changes will definitely slip through, unnoticed!
User-moderating issues
Flagging people which we notice robo-reviewing is quite clumsy to have to first open a question/answer of them, flag it, copying a boilerplate like:
Using {question/answer} to flag irregular behavior of user: https://stackoverflow.com/users/{id}/?tab=documentation shows the user being serially reviewing larger change requests in documentation, with sometimes only {X} seconds between each review. (e.g. {Y} sec for stackoverflow.com/documentation/proposed/changes/{proposal-id}) Could you please remind him that this isn't acceptable and defeating the purpose of review?
This is really making it relatively time-consuming to report people - each offender takes me two to three minutes from noticing to submitting the flag.
Additionally, even if they are caught, I cannot, due to lack of domain-specific knowledge, re-review all their reviewed submissions and assess the actual quality of the submissions (except for blatant issues). These robo-reviewed change proposals need to be re-evaluated for their quality after it's obvious that the reviewers haven't done their job correctly.
Banning issues
For repeated offenders, apart from developers (?), moderators do not have the ability to give doc-review bans yet, only full-site suspensions. While this may be appropriate in some cases, it probably usually isn't.
Seems to be already possible as per a comment from Brad Larson. [Another moderator I've asked yesterday seemed not to be aware of it being possible.]
Proposals
Automated review audits
Use automated audits like other review queues do, which can be seeded from unilaterally approved or rejected change proposals by at least two people [gold-badge holders also may be sometimes wrong and as they only require a single review, better don't use singlehandedly reviewed proposals as audit source], and will cover a large range of rejection reasons - from blatant mistakes and plagiarism to domain-specific issues. Just skip those which have been rejected as duplicates since the duplicate may subsequently have been removed.
Yes, domain-specific issues also - it is not shameful to skip and that shall be encouraged!
Due to Documentation reviewers not being perfect and it being sometimes possible, I suggest a relatively large margin of failure [like allowing one failure per three to four audits or so, in general] (but not too infrequent audits, like on average every few submissions; for people shown usually to be correct, audit count can be reduced) before banning people automatically.
Give moderators the ability to doc-review ban people
In case the audits don't catch everyone or people circumventing them with directly going to the tag review queue, they shall be able to issue temporary bans for documentation reviewing.
Also, in case some people have very bad luck with their audits and just getting bad audits which really should have received the opposite handling, there shall be a possibility for them to reset audit bans.
Putting change proposals reviewed by banned people back to queue
Now, if a person has been Doc-review banned, there now probably are a bunch of change proposals which never should have been approved or rejected.
Put them back into the queue.
For rejected changes (e.g. people spamming "too specific" - the most trivial rejection reason not needing further explanation) just re-propose the change automatically, if there would be no conflict.
For approved changes, in case there is a conflict-free rollback (ideally after applying a three-way merge - if it's not possible to be that fine grained, perhaps resort to using improvement requests by the Community user? - for example, have the Community user propose that rollback). If the approval was fine, people will just reject that rollback.
Perhaps, to avoid unnecessary spam in the review queue, only put the changes back to review if there was only zero or one further reviewer with the same action, who has not been Doc-banned sometime after the approval and the change proposal ended up with the opposite handling than the now banned reviewer did. [E.g. request rejected by the robo-reviewer and approved by a gold-badge holder - no action. Request approved by the robo-reviewer and one person with 10k rep - put back into queue.]
Flagging reviews directly (by user)
Users should be able more easily to report irregular reviewing behavior by just having e.g. a flag symbol inside the change proposal views next to the usernames of the reviewers - moderators then shall be able to mark the flags as helpful, ask for further clarification or just decline them.
Conclusion
Status quo is not acceptable and absolutely needs to be acted on.
I have a few proposals above; I do not see it as critical to include them the exact way I proposed, but some variation of what I proposed would be appreciated in order to ensure a high quality standard for Documentation - it is annoying when answers are incorrect, but one should be largely able to trust correctness of Documentation; sure not everything can be correct, but we shall strive to ensuring as much correctness as possible.